Data search method and device, electronic equipment and storage medium

Abstract

The invention is applicable to the technical field of data search, and provides a data search method, which comprises the following steps of: acquiring frame image data in an input video stream; performing image analysis on the frame image data through a preset visual large model, and generating text description corresponding to the frame image data and corresponding first environment element information; constructing a time sequence video knowledge base based on the text description and the first environment element information; and after input data of a target user is obtained, searching in the time sequence video knowledge base based on the input data to obtain a search result, and outputting the search result. The method solves the problem that the existing search method cannot understand complex natural language requirements and cannot dynamically adjust search results according to contexts or environment information at different time points, so that the working efficiency is low.

Description

Technical Field

[0001] This application belongs to the field of data search technology, and in particular relates to a data search method, apparatus, electronic device and storage medium. Background Technology

[0002] Currently, most existing video search methods rely on manually set tags, keyframe matching, or feature-based searches. These methods cannot understand complex natural language requirements, can only perform keyword or low-dimensional feature matching, and cannot dynamically adjust search results based on context or environmental information at different times. Furthermore, they are costly, have low coverage, and struggle to handle massive amounts of video data. Therefore, there is an urgent need for a data search method with high accuracy in video search to address the problems of existing methods' inability to understand complex natural language requirements and dynamically adjust search results based on context or environmental information at different times, resulting in low efficiency. Summary of the Invention

[0003] This application provides a data search method that addresses the problems of existing search methods, such as their inability to understand complex natural language requirements and their inability to dynamically adjust search results based on context or environmental information at different times, leading to low work efficiency. By using a pre-set visual large model to perform image analysis on frame image data in the input video stream, text descriptions and corresponding first environmental element information are generated for each frame image data. Based on the text descriptions and first environmental element information, a temporal video knowledge base is constructed. After obtaining input data from the target user, the method searches the temporal video knowledge base based on the input data to obtain search results, which are then output. This solves the problems of existing search methods, such as their inability to understand complex natural language requirements and their inability to dynamically adjust search results based on context or environmental information at different times, leading to low work efficiency.

[0004] In a first aspect, embodiments of this application provide a data search method, the method comprising the following steps:

[0005] Acquire frame image data from the input video stream;

[0006] The frame image data is analyzed by a preset visual big model to generate a text description and corresponding first environmental element information for the frame image data.

[0007] Based on the text description and the first environmental element information, a time-series video knowledge base is constructed.

[0008] After obtaining the input data of the target user, a search is performed on the time-series video knowledge base based on the input data to obtain the search results, and the search results are output.

[0009] Optionally, the frame image data includes keyframe image data and continuous frame image data. The step of performing image analysis on the frame image data using a preset visual large model to generate a text description corresponding to the frame image data includes:

[0010] The keyframe image data is analyzed using a pre-defined visual large model to generate a first text description corresponding to the keyframe image data.

[0011] Image analysis is performed on the continuous frame image data to generate a second text description corresponding to the continuous frame image data.

[0012] Optionally, the step of constructing a time-series video knowledge base based on the text description and the first environmental element information includes:

[0013] The text description is subjected to semantic extraction processing to obtain the first semantic vector corresponding to the text description;

[0014] The text description is tagged to obtain multiple text tags;

[0015] The text tags are subjected to semantic extraction processing to obtain the first semantic vector corresponding to the text tags;

[0016] Obtain the video time point corresponding to the first semantic vector;

[0017] By binding the first semantic vector, the first environmental element information, and the video time points, a time-series knowledge base is constructed.

[0018] Optionally, the step of searching the temporal video knowledge base based on the input data to obtain search results includes:

[0019] The input data is subjected to semantic extraction processing to obtain the second semantic vector and the corresponding second environmental element information.

[0020] Based on the second semantic vector and the second environmental element information, a search is performed in the time-series video knowledge base to obtain search results.

[0021] Optionally, the step of performing semantic extraction processing on the input data to obtain the semantic vector corresponding to the input data and the corresponding environmental element information includes:

[0022] The input data is semantically extracted using a pre-defined multimodal large model to obtain a second semantic vector corresponding to the input data.

[0023] The input data is subjected to environment extraction processing to obtain the second environmental element information corresponding to the input data. The preset multimodal large model is obtained by training the pre-trained multimodal large model with a training dataset. The training dataset includes multimodal sample data, semantic category annotation data corresponding to the sample data, and environmental element annotation data corresponding to the sample data. The pre-trained multimodal large model outputs a semantic prediction vector, semantic category data corresponding to the semantic prediction vector, and environmental element data corresponding to the semantic prediction vector.

[0024] Optionally, the search performed in the temporal video knowledge base based on the semantic vector and the environmental element information to obtain search results includes:

[0025] Calculate the semantic similarity between the second semantic vector and the first semantic vector of each frame image in the temporal video knowledge base;

[0026] In addition, the environmental matching degree between the second environmental element information and the first environmental element information of each frame image in the temporal video knowledge base is calculated.

[0027] Based on the semantic similarity and the environmental matching degree, target image data is selected from the temporal video knowledge base, and the target image data is used as the search result corresponding to the input data.

[0028] Optionally, the step of selecting target image data from the temporal video knowledge base based on the semantic similarity and the environmental matching degree, and using the target image data as the search result corresponding to the input data, includes:

[0029] Select target image data from the time-series video knowledge base whose semantic similarity is greater than a preset semantic similarity threshold and whose environment matching degree is greater than a preset environment matching threshold.

[0030] Alternatively, for each frame of image data, calculate the comprehensive similarity with the environment matching degree corresponding to all the first semantic vectors;

[0031] Select target image data from the time-series video knowledge base whose semantic similarity is greater than the preset semantic similarity threshold and whose overall similarity is greater than the preset overall similarity.

[0032] Secondly, embodiments of this application provide a data search device, the data search device comprising:

[0033] The acquisition module is used to acquire frame image data from the input video stream;

[0034] The generation module is used to perform image analysis on the frame image data using a preset visual large model, and generate a text description and corresponding first environmental element information for the frame image data.

[0035] The construction module is used to construct a time-series video knowledge base based on the text description and the first environmental element information;

[0036] The search module is used to obtain input data from the target user, perform a search on the time-series video knowledge base based on the input data, obtain search results, and output the search results.

[0037] Thirdly, embodiments of the present invention provide an electronic device, including: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps in the data search method provided in embodiments of the present invention.

[0038] Fourthly, embodiments of the present invention provide a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps in the data search method provided in the embodiments of the invention.

[0039] The above-mentioned solution of this application has the following beneficial effects: acquiring frame image data from the input video stream; performing image analysis on the frame image data using a preset visual large model to generate text descriptions and corresponding first environmental element information for the frame image data; constructing a temporal video knowledge base based on the text descriptions and first environmental element information; and after acquiring the input data of the target user, searching the temporal video knowledge base based on the input data to obtain search results and outputting the search results. This invention performs image analysis on frame image data from the input video stream using a preset visual large model to generate text descriptions and corresponding first environmental element information for the frame image data. Based on the text descriptions and first environmental element information, a temporal video knowledge base is constructed. After acquiring the input data of the target user, searching the temporal video knowledge base based on the input data to obtain search results and outputting the search results solves the problems of existing search methods, such as the inability to understand complex natural language requirements and the inability to dynamically adjust search results according to the context or environmental information at different time points, resulting in low work efficiency.

[0040] Other beneficial effects of this application will be described in detail in the following detailed description section. Attached Figure Description

[0041] To more clearly illustrate the technical solutions in the embodiments of this application, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0042] Figure 1 A flowchart illustrating a data search method provided in one embodiment of this application;

[0043] Figure 2 This is a schematic diagram of the structure of a data search device provided in an embodiment of this application;

[0044] Figure 3 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0045] In the following description, specific details such as particular system architectures and techniques are set forth for illustrative purposes and not for limitation, in order to provide a thorough understanding of the embodiments of this application. However, those skilled in the art will understand that this application may also be implemented in other embodiments without these specific details. In other instances, detailed descriptions of well-known systems, apparatuses, circuits, and methods have been omitted so as not to obscure the description of this application with unnecessary detail.

[0046] It should be understood that, when used in this application specification and the appended claims, the term "comprising" indicates the presence of the described features, integrals, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components and / or a collection thereof.

[0047] It should also be understood that the term “and / or” as used in this application specification and the appended claims means any combination of one or more of the associated listed items and all possible combinations, and includes such combinations.

[0048] As used in this application specification and the appended claims, the term "if" may be interpreted, depending on the context, as "when," "once," "in response to determination," or "in response to detection." Similarly, the phrase "if determined" or "if detected [the described condition or event]" may be interpreted, depending on the context, as meaning "once determined," "in response to determination," "once detected [the described condition or event]," or "in response to detection [the described condition or event]."

[0049] Furthermore, in the description of this application and the appended claims, the terms "first," "second," "third," etc., are used only to distinguish descriptions and should not be construed as indicating or implying relative importance.

[0050] References to "one embodiment" or "some embodiments" as described in this specification mean that one or more embodiments of this application include a specific feature, structure, or characteristic described in connection with that embodiment. Therefore, the phrases "in one embodiment," "in some embodiments," "in other embodiments," "in still other embodiments," etc., appearing in different parts of this specification do not necessarily refer to the same embodiment, but rather mean "one or more, but not all, embodiments," unless otherwise specifically emphasized. The terms "comprising," "including," "having," and variations thereof mean "including but not limited to," unless otherwise specifically emphasized.

[0051] like Figure 1 As shown, Figure 1 This is a flowchart of a data search method provided in an embodiment of the present invention. The data search method includes the following steps:

[0052] 101. Obtain frame image data from the input video stream.

[0053] In this embodiment of the invention, the data search method described above can be applied to a data search platform, which can be built on or distributed servers. The data search platform includes a data interface (for sensors or users to upload data), a knowledge database, and a knowledge database construction program. The data interface can be used to obtain a first feature vector to be searched, and the knowledge database construction program can be used to construct the knowledge database. The knowledge database is specifically used to provide additional relational information for the identified data entities, thereby improving the depth of the data recognition system's understanding of the content.

[0054] The input video stream can be a continuous sequence of video data, video data continuously generated by various cameras, or continuous video data arranged in chronological order.

[0055] The aforementioned frame image data can be understood as frame images in a video stream, including keyframe image data and continuous frame image data. Keyframe image data can be frames in a video sequence where the content changes significantly, or frame images representing a scene extracted at fixed intervals. Continuous frame image data can be a series of temporally consecutive frame images in a video sequence.

[0056] 102. Perform image analysis on the frame image data using a pre-set visual large model to generate text descriptions and corresponding first environmental element information for the frame image data.

[0057] In this embodiment of the invention, the aforementioned pre-defined large-scale visual model can be a large-scale visual model built based on deep learning or machine learning, such as a Transformer model or a CNN. The Transformer model is a sequence model based on an attention mechanism, processing the input sequence in parallel through a self-attention mechanism. The CNN (Convolutional Neural Network) automatically extracts local features from images and performs classification or detection by simulating the working principle of the human visual system. The aforementioned large-scale visual model has the ability to convert video images into semantic text descriptions.

[0058] The aforementioned pre-defined large-scale visual model can be obtained by training an untrained large-scale visual model using a visual training dataset. This visual training dataset includes sample visual image data, text descriptions and environmental element annotations for the sample visual image data, sample visual video data, and text descriptions and environmental element annotations for the sample visual video data. The annotation data involves adding structured labels to the raw data, enabling the machine learning model to recognize and process both the raw and labeled data. Through the annotation data, the model can establish a mapping relationship between input data and the correct output label. The untrained large-scale visual model can be a large-scale visual model built based on deep learning or machine learning, such as a Transformer model or a CNN. The training can be supervised training, which uses a set of data with known labels to train the model. By optimizing the model parameters, the model can predict the label of new data or make decisions based on the characteristics of existing data. During training, a minimum loss function can be used to adjust the model parameters to minimize the difference between the model's output label and the input data. This loss function can be a mean squared error loss function, a cross-entropy loss function, etc. The loss function mentioned above is used to measure the difference between the model's prediction and the actual result. Its purpose is to improve prediction accuracy by minimizing the loss function value by adjusting the model parameters.

[0059] The aforementioned pre-defined visual model can identify the corresponding text descriptions and environmental element information in the frame image data.

[0060] The image analysis described above can be understood as the process of extracting visual features from frame image data using a pre-defined large visual model, such as visual features of objects, scenes, actions, and attributes.

[0061] The above generation can be understood as the process of generating corresponding text descriptions and corresponding first environmental element information by performing image analysis on frame image data through a preset visual large model.

[0062] The above text description can be understood as using words to express and explain the content of the image, forming a form of written language. It can be a natural language description, such as a red car appearing at the school gate at night, or a woman wearing a red dress.

[0063] The aforementioned first environmental element information can be environmental element information corresponding to the frame image data. Environmental element information can be understood as contextual information related to the video data, such as time information, weather information, geographical location information, and social context information. The aforementioned time information can be the year, month, day, specific time of day (e.g., morning, night), and season (e.g., winter, spring) in which the video data occurred; the aforementioned weather information can be weather conditions, such as sunny, rainy, or snowy; the aforementioned geographical location information can be the geographical coordinates of the location where the video data occurred, the administrative division (e.g., Beijing), and the specific location type (e.g., commercial area, residential area, highway); the aforementioned social context information can be whether it is a holiday, a workday, or whether there is a specific large-scale event (e.g., during a sporting event, traffic control).

[0064] In one possible implementation, for example, if the video stream contains a scene of a person walking in a park, the visual big model can identify objects and features such as a person, park, trees, and grass, and convert these objects and features into text descriptions, such as "a person is walking in a park, surrounded by trees and grass." At the same time, the visual big model can identify corresponding information such as weather, time, and location.

[0065] 103. Construct a time-series video knowledge base based on text descriptions and information on the first environmental elements.

[0066] In this embodiment of the invention, the above-described construction can be understood as a process of organizing text descriptions and first environmental element information into a time-series video knowledge base through certain procedures and methods. This can be achieved by using knowledge graphs, vectorization, or other methods to structurally store the text descriptions and first environmental element information, thereby constructing the time-series video knowledge base.

[0067] The aforementioned time-series video knowledge base is used for storing, querying, and managing data. It supports data searching and reasoning based on multi-dimensional information such as time, events, and objects.

[0068] It should be noted that the time-series video knowledge base can support data search and reasoning based on multi-dimensional information such as time, event, and object, avoiding the reliance on keywords in traditional search.

[0069] 104. After obtaining the input data of the target user, search the time-series video knowledge base based on the input data, obtain the search results, and output the search results.

[0070] In this embodiment of the invention, the target user can be a user who needs to perform a data search using the data search platform, or a manager, etc.

[0071] The input data mentioned above can be text content entered by the target user during the data search process, such as a red car appearing at the school gate at night, or a woman wearing a red dress.

[0072] The above search can be understood as a process of searching a time-series video knowledge base based on input data.

[0073] The search results mentioned above can be obtained by searching the time-series video knowledge base based on the input data.

[0074] The above output can be understood as presenting search results to the user. Specifically, the search results can be arranged in chronological order and presented to the user for viewing and understanding.

[0075] In this embodiment of the invention, frame image data from an input video stream is acquired; image analysis of the frame image data is performed using a preset visual large-scale model to generate text descriptions and corresponding first environmental element information for the frame image data; a temporal video knowledge base is constructed based on the text descriptions and first environmental element information; after acquiring input data from a target user, a search is performed on the temporal video knowledge base based on the input data to obtain search results, and the search results are output. This invention solves the problems of existing search methods, such as the inability to understand complex natural language requirements and the inability to dynamically adjust search results based on context or environmental information at different times, leading to low work efficiency.

[0076] It is understood that in the specific implementation of this application, data such as image data, text data, knowledge data, and time series data are involved. When the embodiments in this application are applied to specific products or technologies, user permission or consent is required. Furthermore, the collection, use, and processing of related data, as well as the training, deployment, and invocation of algorithm models, must comply with the relevant laws, regulations, and standards of the relevant countries and regions.

[0077] Optionally, the frame image data includes keyframe image data and continuous frame image data. In the step of performing image analysis on the frame image data using a preset visual large model to generate text descriptions corresponding to the frame image data, the keyframe image data can be analyzed using a preset visual large model to generate a first text description corresponding to the keyframe image data; and the continuous frame image data can be analyzed to generate a second text description corresponding to the continuous frame image data.

[0078] In this embodiment of the invention, the aforementioned keyframe image data may be frames in a video sequence where the content changes significantly, or frame images representing a scene extracted at fixed intervals.

[0079] The aforementioned continuous frame image data can be a series of temporally consecutive frame images in a video sequence.

[0080] The aforementioned pre-defined large-scale visual model can be obtained by training an untrained large-scale visual model using a visual training dataset. This visual training dataset includes sample visual image data, text descriptions and environmental element annotations for the sample visual image data, sample visual video data, and text descriptions and environmental element annotations for the sample visual video data. The annotation data involves adding structured labels to the raw data, enabling the machine learning model to recognize and process both the raw and labeled data. Through the annotation data, the model can establish a mapping relationship between input data and the correct output label. The untrained large-scale visual model can be a large-scale visual model built based on deep learning or machine learning, such as a Transformer model or a CNN. The training can be supervised training, which uses a set of data with known labels to train the model. By optimizing the model parameters, the model can predict the label of new data or make decisions based on the characteristics of existing data. During training, a minimum loss function can be used to adjust the model parameters to minimize the difference between the model's output label and the input data. This loss function can be a mean squared error loss function, a cross-entropy loss function, etc. The aforementioned loss function measures the difference between the model's predictions and the actual results. Its purpose is to improve prediction accuracy by minimizing the loss function value through adjusting the model parameters. The pre-defined large-scale visual model can identify the corresponding text descriptions and environmental element information in the frame image data.

[0081] The image analysis described above can be a process of extracting visual features from frame image data through a pre-set large visual model, such as extracting visual features of objects, scenes, actions, attributes, etc.

[0082] The above generation can be a process of generating corresponding text descriptions by analyzing and understanding frame image data through a large visual model.

[0083] The above text description can be understood as using words to express and explain the content of the image, forming a form of written language. It can be a natural language description, such as a red car appearing at the school gate at night, or a woman wearing a red dress.

[0084] The first text description mentioned above can be a text description corresponding to the keyframe image data.

[0085] The second text description mentioned above can be a text description corresponding to consecutive frame image data.

[0086] In one possible embodiment, the video stream can be V={f 1， f 2……， f n}, where V represents the video stream, f i Let represent the image data of the i-th frame, and n represent the number of video frames.

[0087] The formula for generating the text description corresponding to the frame image data is: T i =M(f i ), where M represents the large visual model, T i A text description representing the image data of the i-th frame.

[0088] Optionally, in the step of constructing a time-series video knowledge base based on text description and first environmental element information, the text description can be tagged to obtain multiple text tags; the text tags can be semantically extracted to obtain the first semantic vector corresponding to the text tag; the video time point corresponding to the first semantic vector can be obtained; and the first semantic vector, the first environmental element information, and the video time point can be bound together to construct the time-series knowledge base.

[0089] In this embodiment of the invention, the above-mentioned tagging process can be understood as the process of classifying and marking keywords or phrases in the text description as text tags. For example, keywords such as people, time, events, and locations in the text description are marked as text tags.

[0090] The text tags mentioned above can be understood as HTML tags used to define the structure and display style of document content, such as headings, paragraphs, formatting, and quotations. HTML (Hypertext Markup Language) is a markup language used to create and design web pages, defining the structure and format of web page content through tags, such as text, images, and links.

[0091] The semantic extraction process described above can be understood as the process of extracting semantic features from text tags by analyzing the text tags.

[0092] The first semantic vector mentioned above can be the semantic vector corresponding to the text label, used to represent the semantic features in the text label.

[0093] The aforementioned first environmental element information can be environmental element information corresponding to the frame image data. Environmental element information can be understood as contextual information related to video data. Environmental element information can be time information, weather information, geographical location information, social context information, etc.

[0094] The aforementioned video time points can be understood as time points in the video stream used to mark events or actions.

[0095] The video time point corresponding to the first semantic vector mentioned above can be understood as the time point in the video stream when the first semantic vector occurs.

[0096] The above binding can be understood as a process of associating the first semantic vector, the first environmental element information, and the video time point so that they can recognize each other and work together.

[0097] Specifically, a semantic vector, first environmental element information, and video time points can be integrated together to form a record and create a time-series video knowledge base.

[0098] The aforementioned time-series video knowledge base is used for storing, querying, and managing data. It supports data searching and reasoning based on multi-dimensional information such as time, events, and objects.

[0099] In one possible embodiment, from the text description T i The formula for extracting text tags is:

[0100] L i ={l i1 , l i2 , ..., l im}

[0101] Among them, L i Let l represent the set of tags corresponding to the i-th frame image. i1 , l i2 , ..., l im This represents a text label, and m represents the number of text labels.

[0102] The formula for converting text tags into semantic vectors is:

[0103]

[0104] in, This represents the semantic vectorization function for text tags, where d represents the vector dimension.

[0105] Semantic vectorization is represented as:

[0106]

[0107] Among them, K i This represents the semantic vector of the i-th frame image.

[0108] It should be noted that the text description can be tagged to obtain multiple text tags. Semantic extraction processing can be performed on the text tags to obtain the first semantic vector corresponding to the text tag. The video time point corresponding to the first semantic vector can be obtained. The first semantic vector, the first environmental element information, and the video time point are bound together to build a time series knowledge base, which can improve the search efficiency and accuracy of video content.

[0109] Optionally, in the step of searching the temporal video knowledge base based on the input data to obtain the search results, the input data can be semantically extracted to obtain the second semantic vector corresponding to the input data and the corresponding second environmental element information; based on the second semantic vector and the second environmental element information, the search can be performed in the temporal video knowledge base to obtain the search results.

[0110] In this embodiment of the invention, the input data can be text content entered by the target user during the data search process, such as a red car appearing at the school gate at night, or a woman wearing a red dress.

[0111] The semantic extraction process described above can be understood as the process of extracting semantic features from the input data by analyzing the input data.

[0112] The second semantic vector mentioned above can be the semantic vector corresponding to the input data.

[0113] The aforementioned second environmental element information can be environmental element information corresponding to the input data, including weather, geographical location, scene context, etc.

[0114] The aforementioned time-series video knowledge base is used for storing, querying, and managing data. It supports data searching and reasoning based on multi-dimensional information such as time, events, and objects.

[0115] The above search can be a process of searching in a time-series video knowledge base based on the second semantic vector and the second environmental element information.

[0116] The search results mentioned above can be obtained by searching the temporal video knowledge base based on the second semantic vector and the second environmental element information, and the search results corresponding to the input data.

[0117] It should be noted that searching in the temporal video knowledge base based on the second semantic vector and the second environmental element information can improve the relevance of the search results.

[0118] Optionally, in the step of performing semantic extraction processing on the input data to obtain the second semantic vector and the corresponding second environmental element information, the input data can be semantically extracted using a preset multimodal large model to obtain the second semantic vector; and the input data can be environmentally extracted to obtain the second environmental element information.

[0119] In this embodiment of the invention, the aforementioned preset multimodal large model is obtained by training a pre-trained multimodal large model using a training dataset. The training dataset includes multimodal sample data, semantic category annotation data corresponding to the sample data, and environmental element annotation data corresponding to the sample data. The pre-trained multimodal large model outputs semantic prediction vectors, semantic category data corresponding to the semantic prediction vectors, and environmental element data corresponding to the semantic prediction vectors.

[0120] The aforementioned pre-trained multimodal large models can be multimodal large models built based on deep learning or machine learning, such as CLIP, MLLMs, VSM, etc. These multimodal large models possess cross-modal information fusion capabilities, enabling them to process data from different modalities simultaneously, such as text descriptions + image verification, capturing richer features and correlations, and improving the depth of understanding of complex scenes.

[0121] The labeled data mentioned above can be used to add structured labels to the raw data, enabling machine learning models to recognize and process both the raw and labeled data. Through labeled data, the model can establish a mapping relationship between input data and correct output labels.

[0122] The training described above can be supervised training. Supervised training uses a set of data with known labels to train the model. By optimizing the model parameters, the model can predict the labels of new data or make decisions based on the characteristics of existing data.

[0123] It's important to note that during training, the minimum loss function can be used to tune the model's parameters to minimize the difference between the model's output label and the input data. The loss function measures the difference between the model's prediction and the true result; its purpose is to improve prediction accuracy by minimizing the loss function value through adjusting the model parameters. The aforementioned loss function can be the mean squared error loss function, cross-entropy loss function, etc.

[0124] The aforementioned pre-defined multimodal large model can identify the second semantic vector corresponding to the input data and the corresponding second environmental element information.

[0125] The semantic extraction process described above can be a process of analyzing input data using a pre-defined multimodal large model to extract semantic features from the input data.

[0126] The second semantic vector mentioned above can be the semantic vector corresponding to the input data.

[0127] The above-mentioned environmental extraction process can be a process of extracting environmental elements from the input data by analyzing the input data through a preset multimodal large model.

[0128] The aforementioned second environmental element information can be the environmental element information corresponding to the input data.

[0129] In one possible implementation, when the input data is a red vehicle appearing at the school gate at night, the input data is represented as: Q = "Red vehicle appearing at the school gate at night".

[0130] The formula for converting input data into semantic vectors is:

[0131]

[0132] in, This represents the semantic vectorization function of the input data, where d represents the vector dimension.

[0133] Optionally, in the step of searching the temporal video knowledge base based on the second semantic vector and the second environmental element information to obtain search results, the semantic similarity between the second semantic vector and the first semantic vector of each frame image in the temporal video knowledge base can be calculated; and the environmental matching degree between the second environmental element information and the first environmental element information of each frame image in the temporal video knowledge base can be calculated; based on the semantic similarity and environmental matching degree, target image data is selected in the temporal video knowledge base, and the target image data is used as the search results corresponding to the input data.

[0134] In this embodiment of the invention, the first semantic vector may be the semantic vector corresponding to each frame image in the temporal video knowledge base.

[0135] The aforementioned semantic similarity can be the semantic similarity between the second semantic vector and the first semantic vectors of each frame image in the temporal video knowledge base. The degree of similarity between the second semantic vector and the first semantic vectors of each frame image in the temporal video knowledge base can be evaluated by calculating the cosine similarity. The cosine similarity is an indicator used to measure the similarity between the second semantic vector and the first semantic vectors of each frame image in the temporal video knowledge base. It can be determined by the cosine value of the angle between the second semantic vector and the first semantic vectors of each frame image in the temporal video knowledge base. The smaller the angle, the greater the similarity; conversely, the larger the angle, the smaller the similarity.

[0136] Specifically, the semantic similarity can be calculated by comparing the second semantic vector with the first semantic vector of each frame image in the temporal video knowledge base using cosine similarity. The formula for calculating cosine similarity is shown below:

[0137]

[0138] Where S(Q,K) i E(Q) represents the cosine similarity; a larger cosine value indicates higher semantic similarity. E(Q) represents the second semantic vector. i represents the first semantic vector; Q represents the input data.

[0139] The aforementioned first environmental element information can be the environmental element information corresponding to each frame image in the time-series video knowledge base.

[0140] The aforementioned environmental matching degree can be considered as the degree of environmental consistency between the second environmental element information and the first environmental element information of each frame image in the temporal video knowledge base. The environmental matching degree between the second environmental element information and the first environmental element information of each frame image in the temporal video knowledge base can be calculated using Euclidean distance. The smaller the Euclidean distance, the greater the environmental matching degree between the second environmental element information and the first environmental element information of each frame image in the temporal video knowledge base; conversely, the larger the Euclidean distance, the smaller the environmental matching degree between the second environmental element information and the first environmental element information of each frame image in the temporal video knowledge base. The aforementioned Euclidean distance is the straight-line distance between two points in Euclidean space.

[0141] The target image data mentioned above was selected from the temporal video knowledge base based on semantic similarity and environmental matching.

[0142] Specifically, the target image data can be selected from the time-series video knowledge base using the following formula:

[0143]

[0144] Where F(Q,K) i () indicates the final relevance score; , S(Q,K) represents the weight parameter, used to control the importance of the first semantic vector's matching degree with the environment; i S(C,K) represents the cosine similarity between the second semantic vector and the first semantic vector of each frame image in the temporal video knowledge base; i The cosine similarity is represented between the second environmental element information and the first environmental element information of each frame image in the time-series video knowledge base.

[0145] Furthermore, it can be based on F(Q,K) i Sort the search results by their values ​​from highest to lowest, and output the most relevant search results:

[0146]

[0147] It is understandable that by calculating the semantic similarity between the second semantic vector and the first semantic vector of each frame image in the temporal video knowledge base, and by calculating the environmental matching degree between the second environmental element information and the first environmental element information of each frame image in the temporal video knowledge base, and by selecting target image data in the temporal video knowledge base based on the semantic similarity and environmental matching degree, and using the target image data as the search result corresponding to the input data, the accuracy and relevance of the search results can be improved.

[0148] Optionally, in the step of selecting target image data from the temporal video knowledge base based on semantic similarity and environment matching degree, and using the target image data as the search result corresponding to the input data, target image data with semantic similarity greater than a preset semantic similarity threshold and environment matching degree greater than a preset environment matching threshold can be selected from the temporal video knowledge base; or, for each frame of image data, the comprehensive similarity with the environment matching degree corresponding to all first semantic vectors can be calculated; and target image data with semantic similarity greater than a preset semantic similarity threshold and comprehensive similarity greater than a preset comprehensive similarity can be selected from the temporal video knowledge base.

[0149] In this embodiment of the invention, the aforementioned preset semantic similarity threshold can be the semantic similarity threshold between the second semantic vector pre-set by the system and the first semantic vector of each frame image in the time-series video knowledge base.

[0150] The aforementioned preset environment matching threshold can be the environment matching degree threshold between the second environment element information pre-set by the system and the first environment element information of each frame image in the time-series video knowledge base.

[0151] The aforementioned comprehensive similarity can be the combined similarity of the environment matching scores corresponding to all first semantic vectors in each frame of the image. Specifically, methods such as cosine similarity can be used to calculate the comprehensive similarity by weighted averaging of the environment matching scores corresponding to all first semantic vectors in each frame of the image data. For example, given n first semantic vectors, the environment matching score corresponding to the i-th first semantic vector is S. i Then the overall similarity can be expressed as: M = Σ(S i * w i / Σw i ), where w i It is the weight of the i-th first semantic vector. The weighted average above is calculated by assigning different weights to each data point.

[0152] The aforementioned preset comprehensive similarity threshold can be a comprehensive similarity threshold pre-set by the system based on the matching degree of the environment corresponding to all first semantic vectors.

[0153] It should be noted that target image data with semantic similarity greater than a preset semantic similarity threshold and environmental matching degree greater than a preset environmental matching threshold can be selected from the temporal video knowledge base. Alternatively, for each frame of image data, the comprehensive similarity with the environmental matching degree corresponding to all first semantic vectors can be calculated. Target image data with semantic similarity greater than a preset semantic similarity threshold and comprehensive similarity greater than a preset comprehensive similarity can be selected from the temporal video knowledge base. This can improve the accuracy and efficiency of video retrieval and reduce manual intervention and error rate.

[0154] In this embodiment of the invention, a large visual model replaces manual annotation, significantly reducing labor costs. A temporal video knowledge base supports complex semantic searches based on dimensions such as time, event, and object. Users can directly input natural language for searching, avoiding the reliance on keywords in traditional searches and improving search accuracy and efficiency. During the search process, incorporating environmental factors enhances the contextual rationality of search results. This invention addresses the problems of existing search methods' inability to understand complex natural language requirements and their inability to dynamically adjust search results based on context or environmental information at different times, leading to low work efficiency.

[0155] like Figure 2 As shown, an embodiment of the present invention provides a data search device, which includes:

[0156] The acquisition module 201 is used to acquire frame image data from the input video stream;

[0157] The generation module 202 is used to perform image analysis on the frame image data through a preset visual large model, and generate a text description and corresponding first environmental element information corresponding to the frame image data.

[0158] Construction module 203 is used to construct a time-series video knowledge base based on the text description and the first environmental element information;

[0159] The search module 204 is used to obtain the input data of the target user, perform a search on the time-series video knowledge base based on the input data, obtain the search results, and output the search results.

[0160] Optionally, the generation module 202 is further configured to perform image analysis on the keyframe image data using a preset visual large model to generate a first text description corresponding to the keyframe image data; and to perform image analysis on the continuous frame image data to generate a second text description corresponding to the continuous frame image data.

[0161] Optionally, the construction module 203 is further configured to perform tagging processing on the text description to obtain multiple text tags; perform semantic extraction processing on the text tags to obtain a first semantic vector corresponding to the text tags; obtain the video time point corresponding to the first semantic vector; and bind the first semantic vector, the first environmental element information, and the video time point to construct a time series knowledge base.

[0162] Optionally, the search module 204 is further configured to perform semantic extraction processing on the input data to obtain a second semantic vector corresponding to the input data and a corresponding second environmental element information; and to perform a search in the temporal video knowledge base based on the second semantic vector and the second environmental element information to obtain search results.

[0163] Optionally, the search module 204 is further configured to perform semantic extraction processing on the input data using a preset multimodal large model to obtain a second semantic vector corresponding to the input data; and to perform environmental extraction processing on the input data to obtain second environmental element information corresponding to the input data. The preset multimodal large model is obtained by training a pre-trained multimodal large model using a training dataset. The training dataset includes multimodal sample data, semantic category annotation data corresponding to the sample data, and environmental element annotation data corresponding to the sample data. The pre-trained multimodal large model outputs a semantic prediction vector, semantic category data corresponding to the semantic prediction vector, and environmental element data corresponding to the semantic prediction vector.

[0164] Optionally, the search module 204 is further configured to calculate the semantic similarity between the second semantic vector and the first semantic vector of each frame image in the temporal video knowledge base; and to calculate the environmental matching degree between the second environmental element information and the first environmental element information of each frame image in the temporal video knowledge base; based on the semantic similarity and the environmental matching degree, to select target image data in the temporal video knowledge base, and to use the target image data as the search result corresponding to the input data.

[0165] Optionally, the search module 204 is further configured to select target image data in the temporal video knowledge base whose semantic similarity is greater than a preset semantic similarity threshold and whose environment matching degree is greater than a preset environment matching threshold; or, for each frame of image data, calculate the comprehensive similarity with the environment matching degree corresponding to all the first semantic vectors; and select target image data in the temporal video knowledge base whose semantic similarity is greater than the preset semantic similarity threshold and whose comprehensive similarity is greater than a preset comprehensive similarity.

[0166] like Figure 3 As shown, this embodiment of the invention also provides an electronic device, including a processor, which can execute any of the above-described data search methods.

[0167] Specifically, it includes a processor 301 and a memory 302, as well as a computer program stored in the memory 302 and capable of running on the processor 301 to execute a data search method, wherein:

[0168] The processor 301 executes the calculator program that uses the data search method stored in memory 302, performing the following steps:

[0169] Acquire frame image data from the input video stream;

[0170] The frame image data is analyzed by a preset visual big model to generate a text description and corresponding first environmental element information for the frame image data.

[0171] Based on the text description and the first environmental element information, a time-series video knowledge base is constructed.

[0172] After obtaining the input data of the target user, a search is performed on the time-series video knowledge base based on the input data to obtain the search results, and the search results are output.

[0173] Optionally, the frame image data includes keyframe image data and continuous frame image data. The step of processor 301 performing image analysis on the frame image data using a preset visual large model to generate a text description corresponding to the frame image data includes:

[0174] The keyframe image data is analyzed using a pre-defined visual large model to generate a first text description corresponding to the keyframe image data.

[0175] Image analysis is performed on the continuous frame image data to generate a second text description corresponding to the continuous frame image data.

[0176] Optionally, the process executed by processor 301 to construct a time-series video knowledge base based on the text description and the first environmental element information includes:

[0177] The text description is tagged to obtain multiple text tags;

[0178] The text tags are subjected to semantic extraction processing to obtain the first semantic vector corresponding to the text tags;

[0179] Obtain the video time point corresponding to the first semantic vector;

[0180] By binding the first semantic vector, the first environmental element information, and the video time points, a time-series knowledge base is constructed.

[0181] Optionally, the processor 301 performs a search on the temporal video knowledge base based on the input data to obtain search results, including:

[0182] The input data is subjected to semantic extraction processing to obtain the second semantic vector and the corresponding second environmental element information.

[0183] Based on the second semantic vector and the second environmental element information, a search is performed in the time-series video knowledge base to obtain search results.

[0184] Optionally, the semantic extraction processing performed by the processor 301 on the input data to obtain the semantic vector corresponding to the input data and the corresponding environmental element information includes:

[0185] The input data is semantically extracted using a pre-defined multimodal large model to obtain a second semantic vector corresponding to the input data.

[0186] The input data is subjected to environment extraction processing to obtain the second environmental element information corresponding to the input data. The preset multimodal large model is obtained by training the pre-trained multimodal large model with a training dataset. The training dataset includes multimodal sample data, semantic category annotation data corresponding to the sample data, and environmental element annotation data corresponding to the sample data. The pre-trained multimodal large model outputs a semantic prediction vector, semantic category data corresponding to the semantic prediction vector, and environmental element data corresponding to the semantic prediction vector.

[0187] Optionally, the processor 301 performs a search in the temporal video knowledge base based on the semantic vector and the environmental element information to obtain search results, including:

[0188] Calculate the semantic similarity between the second semantic vector and the first semantic vector of each frame image in the temporal video knowledge base;

[0189] In addition, the environmental matching degree between the second environmental element information and the first environmental element information of each frame image in the temporal video knowledge base is calculated.

[0190] Based on the semantic similarity and the environmental matching degree, target image data is selected from the temporal video knowledge base, and the target image data is used as the search result corresponding to the input data.

[0191] Optionally, the step of processor 301 selecting target image data from the temporal video knowledge base based on the semantic similarity and the environment matching degree, and using the target image data as the search result corresponding to the input data, includes:

[0192] Select target image data from the time-series video knowledge base whose semantic similarity is greater than a preset semantic similarity threshold and whose environment matching degree is greater than a preset environment matching threshold.

[0193] Alternatively, for each frame of image data, calculate the comprehensive similarity with the environment matching degree corresponding to all the first semantic vectors;

[0194] Select target image data from the time-series video knowledge base whose semantic similarity is greater than the preset semantic similarity threshold and whose overall similarity is greater than the preset overall similarity.

[0195] This invention also provides a computer-readable storage medium storing a computer program. When the computer program is executed by a processor, it implements the various processes of the data search method provided in this invention and achieves the same technical effect. To avoid repetition, it will not be described again here.

[0196] The above description is the preferred embodiment of this application. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principles described in this application, and these improvements and modifications should also be considered within the scope of protection of this application.

Claims

1. A data search method, characterized in that, The method includes the following steps: Acquire frame image data from the input video stream; The frame image data is analyzed by a preset visual big model to generate a text description and corresponding first environmental element information for the frame image data. Based on the text description and the first environmental element information, a time-series video knowledge base is constructed. After obtaining the input data of the target user, a search is performed on the time-series video knowledge base based on the input data to obtain the search results, and the search results are output.

2. The data search method as described in claim 1, characterized in that, The frame image data includes keyframe image data and continuous frame image data. The step of performing image analysis on the frame image data using a preset large-scale visual model to generate text descriptions corresponding to the frame image data includes: The keyframe image data is analyzed using a pre-defined visual large model to generate a first text description corresponding to the keyframe image data. Image analysis is performed on the continuous frame image data to generate a second text description corresponding to the continuous frame image data.

3. The data search method as described in claim 1, characterized in that, The step of constructing a time-series video knowledge base based on the text description and the first environmental element information includes: The text description is tagged to obtain multiple text tags; The text tags are subjected to semantic extraction processing to obtain the first semantic vector corresponding to the text tags; Obtain the video time point corresponding to the first semantic vector; By binding the first semantic vector, the first environmental element information, and the video time points, a time-series knowledge base is constructed.

4. The data search method as described in claim 1, characterized in that, The search performed on the temporal video knowledge base based on the input data to obtain search results includes: The input data is subjected to semantic extraction processing to obtain the second semantic vector and the corresponding second environmental element information. Based on the second semantic vector and the second environmental element information, a search is performed in the time-series video knowledge base to obtain search results.

5. The data search method as described in claim 4, characterized in that, The step of performing semantic extraction processing on the input data to obtain the semantic vector corresponding to the input data and the corresponding environmental element information includes: The input data is semantically extracted using a pre-defined multimodal large model to obtain a second semantic vector corresponding to the input data. The input data is subjected to environment extraction processing to obtain the second environmental element information corresponding to the input data. The preset multimodal large model is obtained by training the pre-trained multimodal large model with a training dataset. The training dataset includes multimodal sample data, semantic category annotation data corresponding to the sample data, and environmental element annotation data corresponding to the sample data. The pre-trained multimodal large model outputs a semantic prediction vector, semantic category data corresponding to the semantic prediction vector, and environmental element data corresponding to the semantic prediction vector.

6. The data search method as described in claim 4, characterized in that, The search is performed in the temporal video knowledge base based on the semantic vector and the environmental element information to obtain search results, including: Calculate the semantic similarity between the second semantic vector and the first semantic vector of each frame image in the temporal video knowledge base; In addition, the environmental matching degree between the second environmental element information and the first environmental element information of each frame image in the temporal video knowledge base is calculated. Based on the semantic similarity and the environmental matching degree, target image data is selected from the temporal video knowledge base, and the target image data is used as the search result corresponding to the input data.

7. The data search method as described in claim 6, characterized in that, The step of selecting target image data from the temporal video knowledge base based on the semantic similarity and the environmental matching degree, and using the target image data as the search result corresponding to the input data, includes: Select target image data from the time-series video knowledge base whose semantic similarity is greater than a preset semantic similarity threshold and whose environment matching degree is greater than a preset environment matching threshold. Alternatively, for each frame of image data, calculate the comprehensive similarity with the environment matching degree corresponding to all the first semantic vectors; Select target image data from the time-series video knowledge base whose semantic similarity is greater than the preset semantic similarity threshold and whose overall similarity is greater than the preset overall similarity.

8. A data search device, characterized in that, The data search device includes: The acquisition module is used to acquire frame image data from the input video stream; The generation module is used to perform image analysis on the frame image data using a preset visual large model, and generate a text description and corresponding first environmental element information for the frame image data. The construction module is used to construct a time-series video knowledge base based on the text description and the first environmental element information; The search module is used to obtain input data from the target user, perform a search on the time-series video knowledge base based on the input data, obtain search results, and output the search results.

9. An electronic device, characterized in that, include: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor, when executing the computer program, implements the steps of the data search method as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the steps of the data search method as described in any one of claims 1 to 7.

Images