Video retrieval method and apparatus

Abstract

The present disclosure relates to the technical field of computer, and particularly relates to a video retrieval method and device. The method comprises the following steps: acquiring multi-modal input information; performing weighted fusion on the multi-modal input information through an attention mechanism to obtain a retrieval vector; performing retrieval in a vector database by using the retrieval vector and user preference information corresponding to the multi-modal input information to obtain a first video segment set corresponding to the multi-modal input information, wherein the vector database comprises multi-dimensional video semantic vectors of at least one video segment. The present disclosure can improve video retrieval accuracy, comprehensiveness and retrieval efficiency, and improve the video retrieval experience.

Description

Technical Field

[0001] This disclosure relates to the field of computer technology, and in particular to a video retrieval method and apparatus. Background Technology

[0002] With the development of science and technology and the explosive growth of video data, some video retrieval technologies, such as those based solely on text keyword matching, face numerous challenges. They fail to fully utilize the multimodal information in videos (such as visual, auditory, and scene information), resulting in inaccurate and incomplete search results, making it difficult to meet users' increasingly diverse and precise search needs. Furthermore, when faced with massive amounts of video files, traditional retrieval algorithms are inefficient and lack in-depth understanding of user search habits and semantics, failing to achieve an intelligent search experience. Summary of the Invention

[0003] This disclosure provides a video retrieval method and apparatus, which can improve the accuracy, comprehensiveness, and efficiency of video retrieval, and enhance the video retrieval experience. The technical solution of this disclosure is as follows:

[0004] According to a first aspect of the present disclosure, a video retrieval method is provided, comprising: Acquire multimodal input information; The multimodal input information is weighted and fused using an attention mechanism to obtain a retrieval vector; Using the retrieval vector and the user preference information corresponding to the multimodal input information, a retrieval is performed in the vector database to obtain a first set of video segments corresponding to the multimodal input information, wherein the vector database includes multi-dimensional video semantic vectors of at least one video segment.

[0005] According to some embodiments, the method further includes: Frame analysis is performed on the keyframe set of each historical video using object detection algorithm and image classification algorithm to obtain the tag information and keywords corresponding to each historical video. A scene detection algorithm is used to detect each historical video to obtain color change information and object movement information corresponding to each historical video, and the shot switching point is obtained based on the color change information and the object movement information. Based on the tag information, the keywords, and the shot switching points, each historical video is segmented at the frame level to obtain a second video segment set corresponding to each historical video. Visual features are extracted from each first video segment in the second video segment set using a feature extraction model to obtain the visual vector corresponding to each first video segment. Obtain the text vector corresponding to each of the first video segments; The visual vectors and text vectors corresponding to each first video segment are fused to obtain the multi-dimensional video semantic vectors corresponding to each first video segment, and the multi-dimensional video semantic vectors corresponding to each first video segment are added to the vector database.

[0006] According to some embodiments, the method further includes: Frames are extracted from each historical video according to the frame extraction rate corresponding to each historical video to obtain a set of keyframes for each historical video. The frame extraction rate is determined based on at least one of the video content change information of each historical video and the decoding efficiency corresponding to each historical video. The video content change information includes at least one of the following: image change information of adjacent image frames, pixel difference information, and structural similarity information.

[0007] According to some embodiments, obtaining the text vectors corresponding to each of the first video segments includes: Based on the decoding information of each historical video, the tag information, and the keywords, each first video segment is input into a multimodal model for recognition, the semantic vector of each first video segment is obtained, and the descriptive text of each first video segment is obtained based on the semantic vector; The descriptive text is input into a large model and a Transformer-based Bidirectional Encoder Representations from Transformers (BERT) model for processing to obtain the text vectors corresponding to each first video segment.

[0008] According to some embodiments, the step of weighted fusion of the multimodal input information through an attention mechanism to obtain a retrieval vector includes: Based on the information type corresponding to the multimodal input information, obtain the weight of each input information in the multimodal input information; Based on the weights of each input information, the multimodal input information is weighted and fused to obtain a retrieval vector.

[0009] According to some embodiments, the method further includes at least one of the following: If the number of searches reaches a threshold, the weights of each input information are adjusted to obtain the adjusted weights. If the changes in user preference information meet the information requirements, the weights of each input information are adjusted to obtain the adjusted weights. If the search vector contains keywords not stored in the vector database, the weights of each input information are adjusted to obtain the adjusted weights.

[0010] According to some embodiments, the step of using the retrieval vector and the user preference information corresponding to the multimodal input information to perform a retrieval in a vector database to obtain a first set of video segments corresponding to the multimodal input information includes: Obtain the text vector and semantic vector from the retrieval vector; The text matching degree is obtained based on the text vector and the text description vector of each second video segment in the sub-vector database, wherein the sub-vector database is the database in the vector database that corresponds to the multimodal input information; The scene matching degree is obtained based on the semantic vector and the visual vector of each second video segment; The matching score of each second video segment is obtained based on the text matching degree, the first weight of the text matching degree, the scene matching degree, and the second weight of the scene matching degree. Based on the user preference information corresponding to the multimodal input information, obtain the number of video segments corresponding to the first video segment set; Based on the matching scores of each second video segment and the number of video segments, a first set of video segments corresponding to the multimodal input information is obtained.

[0011] According to some embodiments, the method further includes: Based on the association information between each third video segment in the first video segment set and the multimodal input information, and the user preference information corresponding to the multimodal input information, the first video segment set is sorted using a Transformer sorting model to obtain the third video segment set. The association information includes semantic relevance information, scene consistency information, and temporal information.

[0012] According to some embodiments, the method further includes: Obtain user preference information corresponding to the multimodal input information, wherein the user preference information includes at least one of the following: search keyword information, modal usage preference information, video viewing information, and interaction behavior information corresponding to the user identifier of the multimodal input information; Based on the user preference information, the Transformer-based ranking model is adjusted to obtain the adjusted Transformer-based ranking model.

[0013] According to a second aspect of the present disclosure, a video retrieval device is provided, comprising: The information acquisition unit is used to acquire multimodal input information; The vector acquisition unit is used to perform weighted fusion of the multimodal input information through an attention mechanism to obtain a retrieval vector; A video retrieval unit is used to perform a retrieval in a vector database using the retrieval vector and the user preference information corresponding to the multimodal input information, to obtain a first set of video segments corresponding to the multimodal input information, wherein the vector database includes multi-dimensional video semantic vectors of at least one video segment.

[0014] According to a third aspect of the present disclosure, an electronic device is provided, comprising: processor; Memory used to store the processor's executable instructions; The processor is configured to execute the instructions to implement the video retrieval method described in any one of the preceding aspects.

[0015] According to a fourth aspect of the present disclosure, a storage medium is provided that, when instructions in the storage medium are executed by a processor of an electronic device, enables the electronic device to perform the video retrieval method described in any of the preceding aspects.

[0016] According to a fifth aspect of the present disclosure, a computer program product is provided, including a computer program that, when executed by a processor, implements the method described in any one of the preceding aspects.

[0017] The technical solutions provided by the embodiments of this disclosure bring at least the following beneficial effects: In some or related embodiments, multimodal input information is acquired; the multimodal input information is weighted and fused using an attention mechanism to obtain a retrieval vector; the retrieval vector and user preference information corresponding to the multimodal input information are used to perform a retrieval in a vector database to obtain a first set of video segments corresponding to the multimodal input information, wherein the vector database includes multi-dimensional video semantic vectors of at least one video segment. Therefore, video retrieval can be performed based on multimodal input information, reducing the low accuracy of single-modal retrieval; the fusion of multimodal input information can solve the problem of cross-modal semantic gap; and video retrieval based on retrieval vectors and user preferences allows for in-depth mining of user retrieval habits and semantic understanding, improving the accuracy, comprehensiveness, and efficiency of video retrieval, and enhancing the video retrieval experience.

[0018] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this disclosure. Attached Figure Description

[0019] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this disclosure and, together with the description, serve to explain the principles of this disclosure, and are not intended to unduly limit this disclosure.

[0020] Figure 1 This is a flowchart of the first video retrieval method provided in this embodiment; Figure 2 This is a flowchart of the second video retrieval method provided in this disclosure embodiment; Figure 3 This is a block diagram illustrating a video retrieval device according to an exemplary embodiment; Figure 4 This is an example schematic diagram of an electronic device according to an exemplary embodiment. Detailed Implementation

[0021] To enable those skilled in the art to better understand the technical solutions of this disclosure, the technical solutions in the embodiments of this disclosure will be clearly and completely described below with reference to the accompanying drawings.

[0022] This disclosure provides a video retrieval method and apparatus. In some embodiments, the terms "video retrieval method" and "information processing method," "communication method," etc., can be used interchangeably; the terms "video retrieval apparatus" and "information processing apparatus," "communication apparatus," etc., can be used interchangeably; and the terms "information processing system," "communication system," etc., can be used interchangeably.

[0023] This disclosure is not exhaustive, but merely illustrative of some embodiments, and is not intended to limit the scope of protection of this disclosure. Unless otherwise specified, each step in a particular embodiment can be implemented as an independent embodiment, and the steps can be arbitrarily combined. For example, a solution after removing some steps in a particular embodiment can also be implemented as an independent embodiment, and the order of the steps in a particular embodiment can be arbitrarily interchanged. Furthermore, the optional implementation methods in a particular embodiment can be arbitrarily combined; moreover, the embodiments can be arbitrarily combined, for example, some or all steps of different embodiments can be arbitrarily combined, and a particular embodiment can be arbitrarily combined with the optional implementation methods of other embodiments.

[0024] In each of the disclosed embodiments, unless otherwise specified or in case of logical conflict, the terminology and / or descriptions of the embodiments are consistent and can be referenced by each other. Technical features in different embodiments can be combined to form new embodiments based on their inherent logical relationships.

[0025] The terminology used in the embodiments of this disclosure is for the purpose of describing particular embodiments only and is not intended to limit the scope of this disclosure.

[0026] In this disclosure, unless otherwise stated, elements expressed in the singular form, such as "a," "an," "the," "the," "the," "the," "the," "the," "this," etc., can mean "one and only one," or "one or more," "at least one," etc. For example, when using articles such as "a," "an," "the," etc. in translation, the noun following the article can be understood as either a singular or a plural expression.

[0027] In the embodiments disclosed herein, "multiple" refers to two or more.

[0028] In some embodiments, the terms “at least one of,” “one or more,” “a plurality of,” and “multiple” may be used interchangeably.

[0029] The prefixes "first," "second," etc., used in the embodiments of this disclosure are merely for distinguishing different descriptive objects and do not impose restrictions on the position, order, priority, quantity, or content of the descriptive objects. The description of the descriptive objects is found in the claims or the context of the embodiments, and the use of prefixes should not constitute unnecessary restrictions. For example, if the descriptive object is a "field," the ordinal numbers preceding "field" in "first field" and "second field" do not restrict the position or order of the "fields." "First" and "second" do not restrict whether the "fields" they modify are in the same message, nor do they restrict the order of "first field" and "second field." Similarly, if the descriptive object is a "level," the ordinal numbers preceding "level" in "first level" and "second level" do not restrict the priority between "levels." Furthermore, the number of descriptive objects is not limited by ordinal numbers and can be one or more. For example, in "first device," the number of "devices" can be one or more. Furthermore, the objects modified by different prefixes can be the same or different. For example, if the object being described is "device", then "first device" and "second device" can be the same device or different devices, and their types can be the same or different. Similarly, if the object being described is "information", then "first information" and "second information" can be the same information or different information, and their content can be the same or different.

[0030] In some embodiments, "electronic device" or "terminal device" may be referred to as "user equipment (UE)," "user electronic device," "mobile station (MS)," "mobile electronic device (MT)," "subscriber station," "mobile unit," "subscriber unit," "wireless unit," "remote unit," "mobile device," "wireless device," "wireless communication device," "remote device," "mobile subscriber station," "access terminal," "mobile electronic device," "wireless terminal," "remote terminal," "handset," "user agent," "mobile client," "client," etc.

[0031] In some embodiments, data, information, etc., may be obtained with the user's consent.

[0032] It should be noted that the terms "first," "second," etc., used in the specification, claims, and accompanying drawings of this disclosure are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this disclosure described herein can be implemented in orders other than those illustrated or described herein. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this disclosure. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this disclosure as detailed in the appended claims.

[0033] Figure 1 This is a flowchart of the first video retrieval method provided in the embodiments of this disclosure, such as... Figure 1 As shown, this video retrieval method can be used in scenarios where video retrieval or video search is performed based on multimodal input information and user preferences, and includes the following steps: In step S11, multimodal input information is acquired; In some embodiments, the implementing entity of this disclosure may be, for example, an electronic device. This electronic device does not specifically refer to a particular fixed electronic device. For example, when the identifier of the electronic device changes, the electronic device itself may also change accordingly. For example, when the structure of the electronic device changes, the electronic device may also change accordingly.

[0034] In some embodiments, the name of the multimodal input information is not limited. This multimodal input information can, for example, be input information obtained during video retrieval. This multimodal input information may include input information from multiple modalities. This multimodal input information does not specifically refer to any fixed information. For example, when the modal information corresponding to the multimodal input information changes, the multimodal input information may also change accordingly. This multimodal input information may include text input information, image input information, semantic tag input information, etc. For example, when the time point at which the multimodal input information is acquired changes, the multimodal input information may also change accordingly.

[0035] There are no restrictions on the method for acquiring multimodal input information. For example, different modal input information can be acquired using different methods.

[0036] According to some embodiments, multimodal input information can be obtained.

[0037] In step S12, the multimodal input information is weighted and fused using an attention mechanism to obtain the retrieval vector; According to some embodiments, the attention mechanism may be, for example, the ability to focus on key information while ignoring irrelevant information. This attention mechanism is not specifically defined by a single fixed mechanism. For example, the attention mechanism may change accordingly when the weights of different input information change.

[0038] In some embodiments, weighted fusion may be a process of fusing the input information according to the weights corresponding to each input information in the multimodal input information. This weighted fusion does not refer to a specific fixed processing method. For example, when the weights corresponding to each modal input information in the multimodal input information change, the weighted fusion method may also change accordingly.

[0039] According to some embodiments, the retrieval vector can be, for example, a vector used for video retrieval. This retrieval vector can be a vector obtained by weighted fusion of the corresponding vectors of each modal input information in the multimodal input information. The name of the retrieval vector is not limited. For example, the retrieval vector can also be called a target vector, input vector, etc. The retrieval vector does not specifically refer to a fixed vector. For example, when the multimodal input information changes, the retrieval vector can also change accordingly. For example, when the weighted fusion method changes, the retrieval vector can also change accordingly. The retrieval vector can also be called a search vector, that is, the video retrieval method can also be called a video search method.

[0040] In some embodiments, an attention mechanism can be used to weight and fuse multimodal input information to obtain a retrieval vector.

[0041] In step S13, the user preference information corresponding to the retrieval vector and the multimodal input information is used to search the vector database to obtain the first set of video segments corresponding to the multimodal input information. The vector database includes multi-dimensional video semantic vectors of at least one video segment.

[0042] In some embodiments, user preference information may be used to indicate the preferences of a user corresponding to multimodal input information. This user preference information may, for example, include preferred video information. The user preference information does not specifically refer to any fixed information. For example, the user preference information may change when the time point at which it is acquired changes. Similarly, the user preference information may change when the user corresponding to the multimodal input information changes.

[0043] According to some embodiments, the vector database may be, for example, a database to be retrieved. This vector database may include vectors corresponding to multiple video segments, and may also include at least one video segment. The vector database is not specifically defined as a fixed database. For example, when the amount of data corresponding to the vector database changes, the vector database may also change accordingly. For example, when a piece of data in the vector database changes, the vector database may also change accordingly.

[0044] In some embodiments, a multi-dimensional video semantic vector can be, for example, a semantic vector with multiple dimensions corresponding to a certain video segment. One video segment can correspond to one multi-dimensional video semantic vector. When a video segment changes, the multi-dimensional video semantic vector of that video segment can also change accordingly. For example, when the method of obtaining the multi-dimensional video semantic vector changes, the multi-dimensional video semantic vector can also change accordingly.

[0045] According to some embodiments, a video segment can be, for example, a video obtained by segmenting a video along a timeline. This video segment does not specifically refer to a fixed segment. For example, when the content of a video segment changes, the video segment may also change accordingly. For example, when the duration of a video segment changes, the video segment may also change accordingly.

[0046] In some embodiments, the video clip set may be a collection of at least one video clip. This first video clip set may include, for example, at least one retrieved video clip corresponding to the multimodal input information. The first video clip set does not specifically refer to a fixed set. For example, the first video clip set may change when the number of video clips included in it changes. Similarly, the first video clip set may change when the method of obtaining it changes. Changing the method of obtaining the first video clip set may include, for example, a change in the retrieval vector or a change in the vector database.

[0047] In some embodiments, user preference information corresponding to retrieval vectors and multimodal input information can be used to search a vector database to obtain a first set of video segments corresponding to the multimodal input information. The vector database includes multi-dimensional video semantic vectors of at least one video segment.

[0048] In some or related embodiments, multimodal input information is acquired; a retrieval vector is obtained by weighted fusion of the multimodal input information through an attention mechanism; and a retrieval is performed in a vector database using the retrieval vector and user preference information corresponding to the multimodal input information to obtain a first set of video segments corresponding to the multimodal input information. The vector database includes multi-dimensional video semantic vectors of at least one video segment. Therefore, video retrieval can be performed based on multimodal input information, reducing the low accuracy of single-modal retrieval. The fusion of multimodal input information can address the cross-modal semantic gap problem, and video retrieval based on retrieval vectors and user preferences improves the accuracy, comprehensiveness, and efficiency of video retrieval, thus enhancing the video retrieval experience. Furthermore, multimodal input including text, images, and semantic tags can be supported. A unified retrieval vector is generated through weighted fusion using an attention mechanism, achieving precise matching of "keywords + scenes" and enabling linked retrieval of keywords and key scenes through cross-modal vector fusion.

[0049] Figure 2 This is a flowchart of the second video retrieval method provided in the embodiments of this disclosure, such as... Figure 2 As shown, it includes the following steps: In step S21, multimodal input information is acquired; In some embodiments, the implementing entity of this disclosure may be, for example, an electronic device. This electronic device does not specifically refer to a particular fixed electronic device. For example, when the identifier of the electronic device changes, the electronic device itself may also change accordingly. For example, when the structure of the electronic device changes, the electronic device may also change accordingly.

[0050] The relevant processes can be as described above, and will not be repeated here.

[0051] In some embodiments, the multimodal input information may include, for example: [1] Text input: Supports natural language keywords (such as "eating") and topic elements (such as "breakfast scene"), which are converted into vectors by CLIP text encoder; [2] Image input: For example images uploaded by users (such as photos of "dining table"), image vectors are extracted using a pre-trained visual model, which can be matched with video visual vectors in the index library for cross-modal matching. [3] Semantic label input: Supports receiving annotation information, such as labels such as “indoor”, “close-up”, “candlelight dinner”, etc., which can be directly associated with scene label vectors in the index library.

[0052] In step S22, the weights of each input information in the multimodal input information are obtained according to the information type corresponding to the multimodal input information; The relevant processes can be as described above, and will not be repeated here.

[0053] According to some embodiments, the method further includes at least one of the following: When the number of searches reaches the threshold, the weights of each input information are adjusted to obtain the adjusted weights. If the changes in user preference information meet the information requirements, the weights of each input information are adjusted to obtain the adjusted weights. When the search vector contains keywords not stored in the vector database, the weights of each input information are adjusted to obtain the adjusted weights. Therefore, adjusting the weights can improve the accuracy of obtaining the weights corresponding to each input information, thereby improving the accuracy of the search vector acquisition and the overall accuracy of video retrieval.

[0054] In some embodiments, an attention mechanism is used to weightedly fuse input vectors from different modalities to generate a unified retrieval vector (e.g., "text keyword vector × 0.3 + image visual vector × 0.4 + semantic vector × 0.3"). Alternatively, only two of these vectors can be selected, such as "text keyword vector × 0.5 + image visual vector × 0.5," with different weighting and fusion processes applied based on user history profiles and search habits. This ensures a multi-dimensional expression of search intent and better aligns with diverse user search habits. Specific details are as follows: 1) Specific implementation of the attention mechanism: The core idea is to enable the execution entity to automatically determine the importance of different modal inputs, much like a human would. For example, when a user inputs both the text "sunset at the beach" and an image, the system can analyze whether the image contains a sunset scene and assign a higher weight to the image vector. If the user only inputs the text and the semantic tag "romantic scene," the weights of the text and semantic vectors will be higher (e.g., text 0.3, image 0.5, semantic 0.2). The formula simplifies to: Retrieval Vector = Each Modal Vector × Sum of Corresponding Weights.

[0055] 2) Dynamic weighting strategy: Triggering conditions for weight adjustment: "Update learning" will be performed when the following conditions are met: (1) Update after N consecutive searches: N=10 by default, that is, the weight will be adjusted according to the behavior of every 10 searches; (2) Modal preference mutation: For example, if a user suddenly changes from "pure text search" to "image + text" search more than 5 times, the weight will be adjusted immediately; (3) New keyword pattern: When a keyword combination that has never been searched before appears (such as "drone aerial photography + snow mountain"), the system will temporarily increase the relevant modal weight to match the new demand.

[0056] Based on some implementation examples, the following scenario illustrates the situation: Scenario 1: A user frequently searches for "food pictures": It is recorded that 80% of the user's searches involve uploading food pictures, and they frequently click on the "Chinese food" tag. When the user enters the text "braised pork" again and uploads a picture, the attention mechanism automatically assigns a higher weight to the image vector (e.g., text 0.3 → 0.2, image 0.4 → 0.5), prioritizing the return of videos containing braised pork pictures.

[0057] In step S23, the multimodal input information is weighted and fused according to the weights of each input information to obtain the retrieval vector; The relevant processes can be as described above, and will not be repeated here.

[0058] In step S24, the user preference information corresponding to the retrieval vector and multimodal input information is used to search the vector database to obtain the first set of video segments corresponding to the multimodal input information. The vector database includes multi-dimensional video semantic vectors of at least one video segment.

[0059] The relevant processes can be as described above, and will not be repeated here.

[0060] According to some embodiments, when a first set of video clips is obtained, the first set of video clips can be displayed, and the display method is not limited. For example, it can be displayed after being sent to a target terminal, or it can be displayed directly.

[0061] According to some embodiments, the method further includes: Frame analysis was performed on keyframe sets of each historical video using object detection and image classification algorithms to obtain the tag information and keywords corresponding to each historical video. A scene detection algorithm is used to detect each historical video, obtain the color change information and object movement information corresponding to each historical video, and obtain the shot switching point based on the color change information and object movement information. Based on tag information, keywords and shot transition points, each historical video is segmented at the frame level to obtain a set of second video segments corresponding to each historical video. Visual features are extracted from each first video segment in the second video segment set using a feature extraction model to obtain the visual vector corresponding to each first video segment. Obtain the text vector corresponding to each first video segment; The visual vectors and text vectors corresponding to each first video segment are fused to obtain a multi-dimensional video semantic vector for each first video segment. This multi-dimensional video semantic vector is then added to a vector database. Therefore, a vector database can be established, which can improve video retrieval efficiency.

[0062] According to some embodiments, the object detection algorithm may be, for example, an algorithm for detecting and locating objects in video frames. The name of the object detection algorithm is not limited, and it does not specifically refer to a particular fixed algorithm. For example, when the algorithm type of the object detection algorithm changes, the object detection algorithm may also change accordingly. When the algorithm parameters of the object detection algorithm change, the object detection algorithm may also change accordingly. The object detection algorithm may, for example, include Convolutional Neural Networks (CNNs) and Transformer end-to-end networks. The object detection algorithm may also be, for example, an AI vision algorithm.

[0063] In some embodiments, the image classification algorithm may be, for example, an algorithm that adds category labels to video frames of historical videos. This image classification algorithm is not specifically defined by any one fixed algorithm. For example, the image classification algorithm may change when the algorithm type changes. For instance, the image classification algorithm may be a CNN algorithm.

[0064] According to some embodiments, the historical video may be a video that occurred before the execution of this technical solution, and the historical video does not specifically refer to a particular fixed video. For example, when the content of the historical video changes, the historical video may also change accordingly. For example, when the acquisition time of the historical video changes, the historical video may also change accordingly.

[0065] In some embodiments, the second video clip set may be a collection of video clips corresponding to each historical video. This second video clip set does not specifically refer to a fixed set. For example, when the historical videos corresponding to the second video clip set change, the second video clip set may also change accordingly. For example, when the method of obtaining the second video clip set or the number of video clips changes, the second video clip set may also change accordingly.

[0066] According to some embodiments, the method further includes: Frames are extracted from each historical video according to the extraction frame rate corresponding to each historical video to obtain the key frame set of each historical video. The extraction frame rate is determined based on at least one of the video content change information of each historical video and the decoding efficiency of each historical video. The video content change information includes at least one of the following: image change information of adjacent image frames, pixel difference information, and structural similarity information.

[0067] According to some embodiments, the frame extraction rate may be, for example, the frame rate at which keyframes of each historical video are acquired. This frame extraction rate does not specifically refer to a fixed frame rate. For example, when the specific value corresponding to the frame extraction rate changes, the frame extraction rate may also change accordingly.

[0068] According to some embodiments, the text vector corresponding to each first video segment is obtained, including: Based on the decoding information, tag information, and keywords of each historical video, each first video segment is input into a multimodal model for recognition, the semantic vector of each first video segment is obtained, and the descriptive text of each first video segment is obtained based on the semantic vector; The descriptive text is input into the large model and the BERT model for processing to obtain the text vector corresponding to each first video segment.

[0069] In some embodiments, when constructing a vectorized index library, i.e., a vector database, a multi-dimensional video semantic representation can be built, which may specifically include: (1) During the video decoding and frame extraction stages: When decoding video, FFmpeg can be used to adaptively adjust the frame extraction rate according to the frequency of changes in the video content, just like a camera intelligently taking pictures. Specific details are as follows: 【1】Dynamically adjust the shooting frequency: When decoding video using FFmpeg, information about changes in adjacent frames can be obtained: if there is a lot of action in the scene (such as a racing scene), more frames are extracted (similar to rapid burst shooting); if the scene has little change (such as a static landscape), fewer frames are extracted (similar to interval shooting). Compared to the traditional method of extracting 25 frames per second, this technical solution has a maximum frame extraction interval of no more than 1 second to avoid missing key frames; the minimum frame extraction interval is 0.02 seconds (50fps) to ensure clear motion details.

[0070] [2] Methods for judging changes in the image: Calculate the pixel differences and structural similarity between adjacent frames. For example, if the car's position has moved in two frames, or the scene changes from indoors to outdoors, and the change is considered significant, more frames can be extracted to reduce the rate of missing key frames.

[0071] 【3】High-efficiency decoding: Leveraging FFmpeg's intelligent filtering capabilities, frames are automatically extracted based on visual changes. It also supports multi-threaded processing, allowing for simultaneous operation of an 8-core processor, resulting in faster processing of high-definition video. Object detection and image classification algorithms can be used to analyze keyframes: images are "tagged" to initially generate video tags and keywords. Detailed explanations are as follows: ① Recognizing objects and scenes in the image: Object recognition tools (similar to the eyesight of artificial intelligence (AI)) are used to identify objects in the scene (such as bicycles and people), and scene recognition tools are used to determine the environment (such as mountains and streets). For example, if a frame of "mountain bike riding" is extracted, it will be labeled with tags such as "bicycle", "mountain", and "riding".

[0072] ② Generate search keywords: The identified object and scene labels are merged, irrelevant words are removed, and concise search keywords (such as "mountain bike riding") are generated to facilitate matching during subsequent searches. The accuracy of merging labels is higher than that of single recognition tools.

[0073] (2) Intelligent segmentation: The PySceneDetect scene detection algorithm is then used to identify shot transition points. Combined with the video tags and keywords generated above, the video is segmented again at the frame level, dividing the original video into content-independent segments (such as "objects", "food", "actions" and similar scenes). Specific details are as follows: A. Find the camera transition point: First, observe color changes: compare the color distribution of adjacent scenes. For example, when switching from a blue sky during the day to a black scene at night, a significant color difference is marked as a transition point. Next, observe object movement (an auxiliary method): when the direction or speed of movement of a large number of objects in the scene suddenly changes (such as when the camera zooms out from a close-up to a distant view), this is also considered a scene transition.

[0074] B. The specific process of segmenting a video may include: 1) How much color difference should there be before cutting? Set a "sensitivity" value (also known as a preset value), with a default value of 30 (0-100). For example, if the color difference between two frames is calculated to be ≥30, it can be determined that a splitting operation can be performed (for example, when switching from a warm indoor color tone to a cool outdoor color tone, the difference may reach 40, which will trigger the splitting).

[0075] 2) Minimum length of each segment: Each segment must contain at least 10 frames (equivalent to 0.4 seconds based on the commonly used 25 frames per second) to avoid cutting into fragmented segments of less than a few seconds (such as an advertisement that flashes by and is then cut off).

[0076] C. How tags help with segmentation: After segmenting the scene, you can check whether it is reasonable by combining the previously generated tags (such as "food" and "action"): if the tags of the scene before and after are different (such as switching from "cooking in the kitchen" to "eating at the table"), and the color difference meets the standard, it will be segmented into two independent segments; if the tags are the same (such as different angles of the same kitchen), even if the colors are different, they will be merged into one segment to ensure the continuity of the content.

[0077] (3) Multimodal feature extraction: [1] Visual features: Visual vectors of video frames are extracted using the Contrastive Language-Image Pre-training (CLIP) and Shifted Window Transformer (SwinTransformer) models to represent the content of the scene (such as color, object category (type of object, moving object and stationary object, etc.), human outline, action, etc.). [2] Text tags: Combining the video tags, keywords, and scene-segmented video segments obtained from the above video decoding and frame extraction, the video content is transformed into semantic vectors through a pre-trained multimodal model, and the descriptive text of the video segments is initially generated (such as "Shot 0: A girl is eating", "Shot 1: What is she eating", "Shot 2: Where is she eating"). The video dialogue generation pre-trained model (Towards Detailed Video Understanding via LargeVision and Language Models, Video-ChatGPT) is used again to perform logical optimization on the descriptive text of the generated video segments, and the BERT model is used to transform it into more logical text vectors. [3] Semantic Vector Construction: Visual features, i.e., visual vectors, are fused with text vectors to form a multi-dimensional semantic vector containing "keywords (e.g., 'what food was eaten') + scene tags (e.g., 'where was eaten')," which is stored in the Faiss high-efficiency vector database. Therefore, based on FFmpeg adaptive frame extraction and PySceneDetect intelligent segmentation, combined with models such as CLIP and BERT, video visual and semantic information can be transformed into multi-dimensional semantic vectors. According to some embodiments, user preference information corresponding to retrieval vectors and multimodal input information is used to perform a retrieval in a vector database to obtain a first set of video segments corresponding to the multimodal input information, including: Retrieve the text vector and semantic vector from the search vector; The text matching degree is obtained based on the text vector and the text description vector of each second video segment in the sub-vector database. The sub-vector database is the database in the vector database that corresponds to the multimodal input information. The scene matching degree is obtained based on the semantic vector and the visual vector of each second video segment; The matching score for each second video segment is obtained based on the text matching degree, the first weight of the text matching degree, the scene matching degree, and the second weight of the scene matching degree. Based on the user preference information corresponding to the multimodal input information, obtain the number of video segments corresponding to the first video segment set; Based on the matching scores and number of video segments for each second video segment, a set of first video segments corresponding to the multimodal input information is obtained. Therefore, the visual content and semantic description of the video can be transformed into high-dimensional semantic vectors, enabling the index to more comprehensively represent the composite information of the video, such as "object-scene-action," significantly improving the accuracy of video retrieval. Furthermore, visual features of images can be extracted, combined with text semantic analysis and semantic tags, and an attention mechanism can be used to generate a unified retrieval vector, enabling search queries to accurately match the composite needs of "keywords + scenes," especially effective in cross-modal video retrieval scenarios. A vectorized index containing video tags, keywords, key scenes, and semantic tags can be constructed, integrating multimodal inputs of text / images / semantic tags. Combined with efficient vector retrieval algorithms and dynamic optimization rules, a technological upgrade from "keyword and key scene matching" to "semantic understanding" is achieved, enabling cross-modal vector fusion for the linked retrieval of keywords and key scenes.

[0078] According to some embodiments, semantic vectors may also be called scene label vectors, for example.

[0079] In some embodiments, scene recognition technology can, for example, use the YOLOv8 (You Only Look Once version 8) or Segment Anything Model to identify key objects (such as "bicycle") and scene types (such as "mountain biking") in a video, generating scene tag vectors. Linked retrieval can be employed, including, for example, allowing the user to simultaneously input the keyword "bicycle" and the tag "mountain scene." A double matching degree can be calculated using vector dot product, prioritizing the return of video clips that simultaneously satisfy both the text keyword and the visual scene. An example of the double matching degree calculation method is illustrated below: After the user inputs the keyword "bicycle" and the tag "mountain scene", the following can be included: (1) Convert the input into vectors: convert the text "bicycle" into a text vector T, the tag "mountain scene" into a semantic vector S, and extract the visual vector V of the video clip (including the objects and scene features in the picture). (2) Calculate the matching degree respectively: text matching degree: calculate the similarity between the text vector T and the video text description vector (similar to calculating the relevance of "bicycle" appearing in the video tag); scene matching degree: calculate the similarity between the semantic vector S and the video visual vector V (for example, to determine whether the video picture has mountain features). (3) Calculate the total score by vector dot product: calculate the total score of double matching using "text matching degree × 0.5 + scene matching degree × 0.5", and return the video with the higher score first to ensure that it meets the requirements of both the keyword "bicycle" and "mountain scene". The weight of each matching degree is not limited.

[0080] In some embodiments, a hierarchical retrieval strategy can be used to efficiently retrieve videos using vectors. In order to quickly respond to user retrieval, the Inverted File with Product Quantization (IVF-PQ) quantization algorithm is first used: (1) The video vectors are first divided into multiple "drawers" (cluster centers). When searching, only the drawers that may contain similar vectors are opened. For example, when looking for "mountain bike" videos, the "sports-bicycle" drawer is located first, instead of searching the entire network, so as to determine the sub-vector database corresponding to the multimodal input information. (2) The vectors are broken down into multiple "parts" for comparison. For example, the video features are broken down into "color", "object" and "scene" parts, and the similarity is calculated quickly, which is faster than directly comparing the entire vector.

[0081] In some embodiments, the top N video segments with the highest similarity to the input vector can be retrieved from the acquired vector database. N is related to the user input; if the number of user input modalities is large, different weighting weights are adjusted based on the user's search habits, automatically adjusting the value of N. If there are many input modalities, the videos to be retrieved can be quickly matched, so the value of N will decrease; if there are few input modalities, the videos to be retrieved cannot be obtained, so the value of N will increase. This improves the accuracy of the retrieval while responding quickly and accurately to user searches.

[0082] The adaptive adjustment of the N value is explained as follows: The N value is dynamically adjusted based on the number of user-input modalities: Initially, N=500. If the input modalities are ≥3 (e.g., text + image + tag), N=500-100×(number of modalities-3) (minimum 100); if the modalities are <3, N=500+100×(3-number of modalities) (maximum 1000). For example, when performing a pure text search, if the number of modalities is 1, N is adjusted to 700 to broaden the search scope; when three modalities are input, N=500 for precise matching, balancing search speed and accuracy.

[0083] According to some embodiments, the method further includes: Based on the association information between each third video segment in the first video segment set and the multimodal input information, and the user preference information corresponding to the multimodal input information, the first video segment set is sorted using a Transformer-based ranking model to obtain the third video segment set. The association information includes semantic relevance information, scene consistency information, and temporal information.

[0084] To better align retrieval with personalized user needs, the retrieved videos are further optimized based on the Transformer ranking model and the following dimensions. Therefore, the Transformer algorithm can be used to construct an efficient retrieval and ranking system, improving the efficiency of processing massive amounts of data. Specifically, a Hierarchical Navigable Small World (HNSW) can be used to quickly retrieve massive amounts of videos. Combined with the Transformer ranking model, which considers semantic relevance, visual similarity, video clarity, and user historical behavior, this video retrieval efficiency is significantly improved compared to keyword matching in some other implementations, and it more accurately matches user search needs. (1) Semantic relevance information: Calculate the semantic matching degree between the video description text and the input keywords; at the same time, use the large language model to expand the semantics of the keywords (such as "high tide" → "low tide" → "seaside" → "beach scene") to expand the search scope; and combine the time / location context (such as "2025 summer seaside video") to filter irrelevant results.

[0085] (2) Scene consistency information: detect the matching probability between the visual features of the video and the input image / scene label (such as the color distribution matching of the "seaside" scene); (3) Timing information: Sequence verification of continuous scene segments (such as the logical chain of "open the refrigerator → take out food → cook → start eating").

[0086] According to some embodiments, video logical sorting based on Transformer: using the Transformer encoder architecture, the above semantic, scene, visual, and temporal features are transformed into "comparable digital signals", similar to generating a "comprehensive scorecard" for each video segment.

[0087] According to some implementation examples, the feature fusion method is illustrated as follows: (1) Semantic relevance information: Generative pre-trained Transformer (GPT) model-like large models can be used to expand keywords (such as "high tide" associated with "seaside"), and calculate the semantic matching score (0-100 points) between the video text and the input; (2) Scene consistency information: compare the matching probability between the video visual features (such as the blue tone of the seaside) and the input image / label, and convert it into scene score; (3) Temporal information: verify the order of continuous segments according to the "event sequence knowledge base" (such as "open the door → enter the house → turn on the light"), and generate temporal score; (4) Fusion strategy: sum the three scores according to the weight (default semantic 40% + scene 35% + temporal 25%), and the video with the higher score is ranked first, to ensure that the result meets the user's needs. Therefore, the GPT series models can be used to semantically expand search terms (e.g., "car" → "engine + interior + driving scenario"), and combined with time and location association retrieval and user historical behavior analysis, the retrieval weight can be dynamically adjusted to better match users' personalized searches and effectively solve the problem of insufficient discovery of users' potential needs.

[0088] According to some embodiments, the method further includes: Obtain user preference information corresponding to multimodal input information, wherein the user preference information includes at least one of the following: search keyword information corresponding to the user identifier of the multimodal input information, modal usage preference information, video viewing information, and interaction behavior information; Based on user preference information, the Transformer-based ranking model is adjusted to obtain the adjusted Transformer-based ranking model. A dynamic optimization mechanism enables adaptive evolution of the retrieval strategy, continuously improving search performance based on real-time user feedback, further enhancing the robustness and applicability of the technical solution.

[0089] In some implementations, user preferences can be modeled: by recording user profiles, such as historical search behavior (e.g., frequent use of tags like "palace intrigue" and "Qing Dynasty drama"), browsing behavior, and collection records, user search preferences and habits can be learned. Specific profile feature dimensions are as follows: (1) Search keyword frequency: for example, if a user searches for "mountain biking" multiple times, "mountain" and "biking" can be recorded as high-frequency words; (2) Modal usage preference: record the user's commonly used input methods (e.g., 90% of searches include images); (3) Browsing / collection records: analyze the types of videos the user likes (e.g., frequently collect "seaside scenery" videos); (4) Interaction behavior: for example, the click position and dwell time on search results (to determine whether the user is more interested in a certain type of result). By recording user preference data, the weights and parameters of the search algorithm can be dynamically adjusted to achieve personalized search services and improve search performance. Custom rules (e.g., "prioritize returning the most recent trending short dramas") are also supported and integrated into the ranking algorithm through the rule engine.

[0090] In some embodiments, user preference modeling and retrieval rule optimization may include: (1) Browsing behavior quantification: converting the number of clicks, video viewing time (e.g., 20 points for a completion rate ≥ 80%), number of favorites / likes, number of skipped segments, etc. into numerical scores, for example, adding 10 points for each click on a related video, and adding 15 points for watching for more than 5 minutes. (2) Preference classification algorithm: using a frequency-based clustering method, automatically grouping according to historical search keywords (e.g., "palace intrigue" appears more than 10 times), tag usage habits, and browsing content categories (e.g., Qing Dynasty dramas account for 60%), generating a "user preference tag set", and dynamically adjusting semantic / scene weights (e.g., increasing the ranking weight of related videos for users with preference by 30%).

[0091] In one or related embodiments, the weights of each input information in the multimodal input information are obtained according to the information type corresponding to the multimodal input information; the multimodal input information is weighted and fused according to the weights of each input information to obtain a retrieval vector. Therefore, the matching between the weights and the current multimodal input information can be improved, the accuracy of retrieval vector acquisition can be improved, and the accuracy of video retrieval can be improved.

[0092] A block diagram of a video retrieval device is shown according to an exemplary embodiment. (Refer to...) Figure 3 The device 300 includes: Information acquisition unit 301 is used to acquire multimodal input information; The vector acquisition unit 302 is used to obtain the retrieval vector by weighted fusion of multimodal input information through an attention mechanism; The video retrieval unit 303 is used to retrieve the first set of video segments corresponding to the multimodal input information by using the retrieval vector and the user preference information corresponding to the multimodal input information in the vector database. The vector database includes multi-dimensional video semantic vectors of at least one video segment.

[0093] According to some embodiments, the information acquisition unit 301 is further configured to: Frame analysis was performed on keyframe sets of each historical video using object detection and image classification algorithms to obtain the tag information and keywords corresponding to each historical video. A scene detection algorithm is used to detect each historical video, obtain the color change information and object movement information corresponding to each historical video, and obtain the shot switching point based on the color change information and object movement information. Based on tag information, keywords and shot transition points, each historical video is segmented at the frame level to obtain a set of second video segments corresponding to each historical video. Visual features are extracted from each first video segment in the second video segment set using a feature extraction model to obtain the visual vector corresponding to each first video segment. Obtain the text vector corresponding to each first video segment; The visual vectors and text vectors corresponding to each first video segment are fused to obtain the multi-dimensional video semantic vectors corresponding to each first video segment, and then the multi-dimensional video semantic vectors corresponding to each first video segment are added to the vector database.

[0094] According to some embodiments, the information acquisition unit 301 is further configured to: Frames are extracted from each historical video according to the extraction frame rate corresponding to each historical video to obtain the key frame set of each historical video. The extraction frame rate is determined based on at least one of the video content change information of each historical video and the decoding efficiency of each historical video. The video content change information includes at least one of the following: image change information of adjacent image frames, pixel difference information, and structural similarity information.

[0095] According to some embodiments, when the information acquisition unit 301 acquires the text vectors corresponding to each first video segment, it is specifically used for: Based on the decoding information, tag information, and keywords of each historical video, each first video segment is input into a multimodal model for recognition, the semantic vector of each first video segment is obtained, and the descriptive text of each first video segment is obtained based on the semantic vector; The descriptive text is input into the large model and the BERT model for processing to obtain the text vector corresponding to each first video segment.

[0096] According to some embodiments, the vector acquisition unit 302 is used to perform weighted fusion of multimodal input information through an attention mechanism to acquire the retrieval vector, specifically for: Based on the information type corresponding to the multimodal input information, obtain the weight of each input information in the multimodal input information; Based on the weights of each input information, the multimodal input information is weighted and fused to obtain the retrieval vector.

[0097] According to some embodiments, the vector acquisition unit 302 is also specifically used for at least one of the following: When the number of searches reaches the threshold, the weights of each input information are adjusted to obtain the adjusted weights. If the changes in user preference information meet the information requirements, the weights of each input information are adjusted to obtain the adjusted weights. If the search vector contains keywords not stored in the vector database, the weights of each input information are adjusted to obtain the adjusted weights.

[0098] According to some embodiments, when the video retrieval unit 303 retrieves a first set of video segments corresponding to the multimodal input information by using retrieval vectors and user preference information corresponding to the multimodal input information to perform a retrieval in a vector database, it is specifically used for: Retrieve the text vector and semantic vector from the search vector; The text matching degree is obtained based on the text vector and the text description vector of each second video segment in the sub-vector database. The sub-vector database is the database in the vector database that corresponds to the multimodal input information. The scene matching degree is obtained based on the semantic vector and the visual vector of each second video segment; The matching score for each second video segment is obtained based on the text matching degree, the first weight of the text matching degree, the scene matching degree, and the second weight of the scene matching degree. Based on the user preference information corresponding to the multimodal input information, obtain the number of video segments corresponding to the first video segment set; Based on the matching score of each second video segment and the number of video segments, obtain the first video segment set corresponding to the multimodal input information.

[0099] According to some embodiments, the video retrieval unit 303 is also specifically used for: Based on the association information between each third video segment in the first video segment set and the multimodal input information, and the user preference information corresponding to the multimodal input information, the first video segment set is sorted using a Transformer-based ranking model to obtain the third video segment set. The association information includes semantic relevance information, scene consistency information, and temporal information.

[0100] According to some embodiments, the video retrieval unit 303 is also specifically used for: Obtain user preference information corresponding to multimodal input information, wherein the user preference information includes at least one of the following: search keyword information corresponding to the user identifier of the multimodal input information, modal usage preference information, video viewing information, and interaction behavior information; Based on user preference information, the Transformer-based ranking model is adjusted to obtain the adjusted Transformer-based ranking model.

[0101] Regarding the apparatus in the above embodiments, the specific manner in which each module performs its operation has been described in detail in the embodiments related to the method, and will not be elaborated upon here.

[0102] Figure 4 A schematic block diagram of an example electronic device 400 that can be used to implement embodiments of the present disclosure is shown. The electronic device 400 is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0103] like Figure 4 As shown, the electronic device 400 includes a computing unit 401, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 402 or a computer program loaded from a storage unit 408 into a random access memory (RAM) 403. The RAM 403 may also store various programs and data required for the operation of the electronic device 400. The computing unit 401, ROM 402, and RAM 403 are interconnected via a bus 404. An input / output (I / O) interface 405 is also connected to the bus 404.

[0104] Multiple components in electronic device 400 are connected to I / O interface 405, including: input unit 406, such as keyboard, mouse, etc.; output unit 407, such as various types of displays, speakers, etc.; storage unit 408, such as disk, optical disk, etc.; and communication unit 409, such as network card, modem, wireless transceiver, etc. Communication unit 409 allows electronic device 400 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0105] The computing unit 401 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the computing unit 401 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 401 performs the various methods and processes described above. For example, in some embodiments, the above methods can be implemented as a computer software program tangibly contained in a machine-readable medium, such as storage unit 408. In some embodiments, part or all of the computer program can be loaded and / or installed on the electronic device 400 via ROM 402 and / or communication unit 409. When the computer program is loaded into RAM 403 and executed by the computing unit 401, one or more steps of the methods described above can be performed. Alternatively, in other embodiments, the computing unit 401 can be configured to perform the above methods by any other suitable means (e.g., by means of firmware).

[0106] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), payload-programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0107] The program code used to implement the methods of this disclosure may be written in any combination of one or more programming languages. This program code may be provided to a processor or controller of a general-purpose computer, special-purpose computer, or other programmable data processing apparatus, such that when executed by the processor or controller, the program code causes the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The program code may be executed entirely on a machine, partially on a machine, as a standalone software package partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0108] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0109] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device for displaying information to the user (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor); and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0110] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as data servers), or middleware components (e.g., application servers), or frontend components (e.g., user computers with graphical user interfaces or web browsers through which users can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., communication networks). Examples of communication networks include local area networks (LANs), wide area networks (WANs), the Internet, and blockchain networks.

[0111] Computer systems can include clients and servers. Clients and servers are generally geographically separated and typically interact via communication networks. The client-server relationship is established by computer programs running on the respective computers and having a client-server relationship with each other. A server can be a cloud server, also known as a cloud computing server or cloud host, a hosting product within the cloud computing service system that addresses the shortcomings of traditional physical hosts and VPS (Virtual Private Server, or simply "VPS") services, such as high management difficulty and weak business scalability. Servers can also be servers for distributed systems or servers incorporating blockchain technology.

[0112] It should be understood that the various forms of processes shown above can be used to rearrange, add, or delete steps. For example, the steps described in this disclosure can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution disclosed in this disclosure can be achieved, and this is not limited herein.

[0113] The specific embodiments described above do not constitute a limitation on the scope of protection of this disclosure. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this disclosure should be included within the scope of protection of this disclosure.

Claims

1. A video retrieval method, characterized in that, include: Acquire multimodal input information; The multimodal input information is weighted and fused using an attention mechanism to obtain a retrieval vector; Using the retrieval vector and the user preference information corresponding to the multimodal input information, a retrieval is performed in the vector database to obtain a first set of video segments corresponding to the multimodal input information, wherein the vector database includes multi-dimensional video semantic vectors of at least one video segment.

2. The method according to claim 1, characterized in that, The method further includes: Frame analysis is performed on the keyframe set of each historical video using object detection algorithm and image classification algorithm to obtain the tag information and keywords corresponding to each historical video. A scene detection algorithm is used to detect each historical video to obtain color change information and object movement information corresponding to each historical video, and the shot switching point is obtained based on the color change information and the object movement information. Based on the tag information, the keywords, and the shot switching points, each historical video is segmented at the frame level to obtain a second video segment set corresponding to each historical video. Visual features are extracted from each first video segment in the second video segment set using a feature extraction model to obtain the visual vector corresponding to each first video segment. Obtain the text vector corresponding to each of the first video segments; The visual vectors and text vectors corresponding to each first video segment are fused to obtain the multi-dimensional video semantic vectors corresponding to each first video segment, and the multi-dimensional video semantic vectors corresponding to each first video segment are added to the vector database.

3. The method according to claim 2, characterized in that, The method further includes: Frames are extracted from each historical video according to the frame extraction rate corresponding to each historical video to obtain a set of keyframes for each historical video. The frame extraction rate is determined based on at least one of the video content change information of each historical video and the decoding efficiency corresponding to each historical video. The video content change information includes at least one of the following: image change information of adjacent image frames, pixel difference information, and structural similarity information.

4. The method according to claim 2, characterized in that, The step of obtaining the text vector corresponding to each of the first video segments includes: Based on the decoding information of each historical video, the tag information, and the keywords, each first video segment is input into a multimodal model for recognition, the semantic vector of each first video segment is obtained, and the descriptive text of each first video segment is obtained based on the semantic vector; The descriptive text is input into the large model and the BERT model for processing to obtain the text vectors corresponding to each first video segment.

5. The method according to claim 1, characterized in that, The step of weighted fusion of the multimodal input information through an attention mechanism to obtain a retrieval vector includes: Based on the information type corresponding to the multimodal input information, obtain the weight of each input information in the multimodal input information; Based on the weights of each input information, the multimodal input information is weighted and fused to obtain a retrieval vector.

6. The method according to claim 5, characterized in that, The method further includes at least one of the following: If the number of searches reaches a threshold, the weights of each input information are adjusted to obtain the adjusted weights. If the changes in user preference information meet the information requirements, the weights of each input information are adjusted to obtain the adjusted weights. If the search vector contains keywords not stored in the vector database, the weights of each input information are adjusted to obtain the adjusted weights.

7. The method according to claim 1, characterized in that, The step of using the retrieval vector and the user preference information corresponding to the multimodal input information to perform a retrieval in the vector database to obtain a first set of video segments corresponding to the multimodal input information includes: Obtain the text vector and semantic vector from the retrieval vector; The text matching degree is obtained based on the text vector and the text description vector of each second video segment in the sub-vector database, wherein the sub-vector database is the database in the vector database that corresponds to the multimodal input information; The scene matching degree is obtained based on the semantic vector and the visual vector of each second video segment; The matching score of each second video segment is obtained based on the text matching degree, the first weight of the text matching degree, the scene matching degree, and the second weight of the scene matching degree. Based on the user preference information corresponding to the multimodal input information, obtain the number of video segments corresponding to the first video segment set; Based on the matching scores of each second video segment and the number of video segments, a first set of video segments corresponding to the multimodal input information is obtained.

8. The method according to claim 1, characterized in that, The method further includes: Based on the association information between each third video segment in the first video segment set and the multimodal input information, and the user preference information corresponding to the multimodal input information, the first video segment set is sorted using a Transformer sorting model to obtain the third video segment set. The association information includes semantic relevance information, scene consistency information, and temporal information.

9. The method according to claim 8, characterized in that, The method further includes: Obtain user preference information corresponding to the multimodal input information, wherein the user preference information includes at least one of the following: search keyword information, modal usage preference information, video viewing information, and interaction behavior information corresponding to the user identifier of the multimodal input information; Based on the user preference information, the Transformer-based ranking model is adjusted to obtain the adjusted Transformer-based ranking model.

10. A video retrieval device, characterized in that, include: The information acquisition unit is used to acquire multimodal input information; The vector acquisition unit is used to perform weighted fusion of the multimodal input information through an attention mechanism to obtain a retrieval vector; A video retrieval unit is used to perform a retrieval in a vector database using the retrieval vector and the user preference information corresponding to the multimodal input information, to obtain a first set of video segments corresponding to the multimodal input information, wherein the vector database includes multi-dimensional video semantic vectors of at least one video segment.

Images