Video query method, device, terminal and storage medium

Abstract

The present application relates to a video query method, device, terminal and storage medium, and relates to the field of video processing technology. The method comprises: obtaining query information and multiple candidate videos; splitting each of the multiple candidate videos to obtain multiple storyboard segments corresponding to the candidate videos, wherein the storyboard segment is a segment of video under the same scene lens in the candidate video; extracting the storyboard feature information of the storyboard segment, and constructing a storyboard feature library, wherein the storyboard feature library includes each storyboard feature information corresponding to the candidate video; querying the storyboard feature library according to the query information to determine the target video. Through the present application, the technical problem of low video query efficiency in the prior art is solved.

Description

Technical Field

[0001] The present application relates to the field of video processing technology, and in particular to a video query method, device, terminal and storage medium. Background Art

[0002] At present, the media industry is in a stage of rapid development in which information can be quickly gathered, disseminated in multiple ways, and accurately reached. Traditional queries based on metadata (title, type, introduction, actors, etc.) of converged media videos cannot structure and abstract the content features of converged media videos, making it difficult to query the corresponding original video based on the scene clips described by the user, so it can no longer meet the needs of users in the intelligent era. At present, video queries are mainly achieved by first building a video tag library, then matching the query information entered by the user with the video tags in the tag library, and then recommending videos with high matching degrees to users. Among them, the construction of a video tag library generally uses manual content segmentation summaries of each converged media video one by one as the video tag of the converged media video, or uses a neural network to perform feature recognition on each image frame in the converged media video, and uses the recognition results as the video tag of the converged media video.

[0003] As video content and video types continue to increase, manual review requires a lot of manpower, and the query accuracy depends on the subjective understanding of the content summarizer. The use of neural network recognition requires identifying and storing the feature information of each image frame, which not only requires a large amount of storage, but also consumes a lot of computing resources. Therefore, both methods have the problem of low efficiency.

[0004] In summary, there is currently no effective solution to the problem of low efficiency of video query in related technologies. Summary of the invention

[0005] The embodiments of the present application provide a video query method, device, terminal and storage medium to solve the technical problem of low video query efficiency in the prior art.

[0006] To achieve the above objectives, the embodiments of the present application adopt the following technical solutions:

[0007] In a first aspect, an embodiment of the present application provides a video query method, the method comprising:

[0008] Obtain query information and multiple candidate videos;

[0009] Splitting each candidate video among the multiple candidate videos to obtain multiple storyboard segments corresponding to the candidate video, wherein the storyboard segment is a segment of video under the same scene shot in the candidate video;

[0010] Extracting the storyboard feature information of the storyboard segments and constructing a storyboard feature library, wherein the storyboard feature library includes each storyboard feature information corresponding to the candidate video;

[0011] According to the query information, the storyboard feature library is queried to determine the target video.

[0012] In combination with the first aspect, in a possible design, splitting each of the multiple candidate videos to obtain multiple storyboard segments corresponding to the candidate video includes:

[0013] Based on the shot boundary time points in the candidate video, the candidate video is split to obtain a plurality of storyboard segments, wherein the shot boundary time points are the segmentation time points of the video shots of the candidate video.

[0014] In combination with the first aspect, in a possible design, the candidate video is split based on the shot boundary time points in the candidate video to obtain a plurality of storyboard segments, including:

[0015] Obtaining a pixel difference between two adjacent image frames in the candidate video, wherein the pixel difference includes a grayscale value;

[0016] If the pixel difference is greater than a first threshold, any time point between the timestamps of the two image frames is used as a shot boundary time point in the candidate video;

[0017] The candidate video is split based on the shot boundary time points to obtain a plurality of storyboard segments.

[0018] In combination with the first aspect, in a possible design manner, extracting the storyboard feature information of the storyboard segments and constructing a storyboard feature library includes:

[0019] Extracting image frames from the storyboard segment according to a sampling quantity that is positively correlated with the duration of the storyboard segment;

[0020] Performing content recognition on the image frame to obtain storyboard feature information of the storyboard segment;

[0021] A storyboard feature library is constructed, wherein the candidate video in the storyboard feature library corresponds to the storyboard feature information of the storyboard segments in the candidate video.

[0022] In combination with the first aspect, in a possible design, extracting an image frame from the storyboard segment according to a sampling quantity that is positively correlated with a duration of the storyboard segment includes:

[0023] Recording the starting time point and the ending time point of each storyboard segment in the candidate video;

[0024] The duration between the start time point and the end time point is used as the duration of the storyboard segment, and image frames are extracted from the storyboard segment according to a sampling quantity that is positively correlated with the duration.

[0025] In combination with the first aspect, in a possible design manner, performing content recognition on the image frame to obtain the storyboard feature information of the storyboard segment includes:

[0026] Using a preset multimodal training model of image-text semantics to perform content recognition on the image frame to obtain a representation vector of the image frame;

[0027] The representation vectors of each of the image frames are clustered to obtain a plurality of clusters, wherein a central vector in each of the clusters is determined as the storyboard feature information of the storyboard segment.

[0028] In combination with the first aspect, in a possible design, searching the storyboard feature library according to the query information to determine the target video includes:

[0029] The candidate video corresponding to the storyboard feature whose similarity with the query information is greater than a second threshold is searched from the storyboard feature library and determined as the target video.

[0030] In a second aspect, an embodiment of the present application provides a video query device, including:

[0031] An acquisition module, used to acquire query information and multiple candidate videos;

[0032] A splitting module is used to split each of the multiple candidate videos to obtain multiple storyboard segments corresponding to the candidate video, wherein the storyboard segment is a segment of video under the same scene shot in the candidate video;

[0033] A processing module, used to extract the storyboard feature information of the storyboard segments and construct a storyboard feature library, wherein the storyboard feature library includes the storyboard feature information corresponding to each of the candidate videos;

[0034] The query module is used to query the storyboard feature library according to the query information to determine the target video.

[0035] In a third aspect, an embodiment of the present application provides a terminal, comprising a memory and a processor, wherein the memory stores a computer program, and the processor is configured to run the computer program to execute the method of the first aspect and possible design methods thereof.

[0036] In a fourth aspect, an embodiment of the present application provides a storage medium, in which a computer program is stored, wherein the computer program is configured to execute the method of the first aspect and its possible design manner when running.

[0037] Compared with the prior art, a video query method, device, terminal and storage medium provided in the embodiment of the present application split the candidate video, extract the storyboard feature information of each storyboard segment of the candidate video, and store the storyboard feature information in a database. In this way, when querying the video, it is only necessary to compare whether the query information and the storyboard feature information match, and output the candidate video corresponding to the successfully matched storyboard feature information as the target video. This not only reduces the storage amount of the stored video features, occupies less space, but also reduces the amount of data for transmitting the video, so the network delay is low, the bandwidth occupancy is low, and the network transmission pressure is reduced. However, the transmission and storage occupy a large amount of resources, which will bring about the problem of low video query efficiency, so the problem of low video query efficiency in the prior art is solved, and the video query efficiency can be improved.

[0038] Details of one or more embodiments of the present application are set forth in the following drawings and description to make other features, objects, and advantages of the present application more readily apparent. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0040] Figure 1 A hardware structure block diagram of a terminal provided in an embodiment of the present application is shown;

[0041] Figure 2 A flow chart of a video query method provided by an embodiment of the present application is shown;

[0042] Figure 3 A system architecture diagram of a video query method provided by an embodiment of the present application is shown;

[0043] Figure 4 A schematic diagram of a video query method provided by an embodiment of the present application is shown;

[0044] Figure 5 A structural block diagram of a video query device provided in an embodiment of the present application is shown. DETAILED DESCRIPTION

[0045] In order to more clearly understand the purpose, technical solutions and advantages of the present application, the present application is described and illustrated below in conjunction with the accompanying drawings and embodiments.

[0046] Unless otherwise defined, the technical terms or scientific terms involved in this application shall have the general meaning understood by people with ordinary skills in the technical field to which this application belongs. The words "one", "a", "a", "the", "these" and the like in this application do not represent quantitative restrictions, and they can be singular or plural. The terms "include", "comprise", "have" and any variants thereof involved in this application are intended to cover non-exclusive inclusions; for example, a process, method and system, product or device comprising a series of steps or modules (units) is not limited to the listed steps or modules (units), but may include unlisted steps or modules (units), or may include other steps or modules (units) inherent to these processes, methods, products or devices. The words "connect", "connected", "coupled" and the like involved in this application are not limited to physical or mechanical connections, but may include electrical connections, whether directly or indirectly. The "multiple" involved in this application refers to two or more. "And / or" describes the association relationship of associated objects, indicating that there can be three relationships. For example, "A and / or B" can mean: A exists alone, A and B exist at the same time, and B exists alone. Usually, the character " / " indicates that the objects associated with each other are in an "or" relationship. The terms "first", "second", "third", etc. involved in this application are only used to distinguish similar objects and do not represent a specific ordering of the objects.

[0047] The video query method provided in the embodiment of the present application can be executed in a terminal or a computer device. Taking running on a terminal as an example, Figure 1 FIG. 1 shows a hardware structure block diagram of a terminal provided in an embodiment of the present application. Figure 1 As shown, the terminal may include one or more ( Figure 1 Only one is shown in the figure) processor 102, and memory 104 for storing data. Processor 102 may include but is not limited to a processing device such as a microprocessor MCU or a programmable logic device FPGA. Optionally, the terminal may also include a transmission device 106 and an input / output device 108 for communication functions. It can be understood by those skilled in the art that Figure 1 The structure shown is only for illustration and does not limit the structure of the above terminal. Figure 1 More or fewer components as shown, or with Figure 1 Different configurations shown.

[0048] The memory 104 can be used to store computer programs, for example, software programs and modules of application software. The processor 102 executes various functional applications and data processing by running the computer programs stored in the memory 104, that is, to implement the above method. The memory 104 may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some examples, the memory 104 may further include a memory remotely arranged relative to the processor 102, and these remote memories may be connected to the terminal via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0049] The transmission device 106 is used to receive or send data via a network. The specific example of the above network may include a wireless network provided by a communication provider of the terminal. In one example, the transmission device 106 includes a network adapter (Network Interface Controller, referred to as NIC), which can be connected to other network devices through a base station so as to communicate with the Internet. In one example, the transmission device 106 can be a radio frequency (Radio Frequency, referred to as RF) module, which is used to communicate with the Internet wirelessly.

[0050] The various techniques described in the embodiments of the present application can be used for video query, that is, the user inputs the query information, and the website or application will return the video with the highest relevance to the query information to the user. At present, this field mainly adopts the method of pre-labeling each video in the video library and matching the query information with the label to realize the query. At present, in the video query scene, the method of identifying each image frame in the video one by one will cause the calculation, transmission, and storage to occupy a lot of resources, so it is inefficient. In addition, the traditional recognition method adopts a neural network such as ResNet50, which mainly relies on manually designed features and rules, such as edge detection, color segmentation, feature extraction, etc., so it is only suitable for processing simple images, and the recognition accuracy for complex images is low. And training a specific model dedicated to identifying the labels of image frames requires a high cost, and the small or unbalanced training data set may cause the model to overfit or the model generalization ability is insufficient.

[0051] In view of this, the method provided in the embodiment of the present application splits the video according to different storyboards and extracts features from each storyboard segment. Compared with the traditional technology of performing feature recognition on each image frame in the integrated media video through a neural network, after the video is split into multiple storyboards, the feature extraction of each storyboard can be processed in parallel, thereby significantly improving the overall processing speed. Each storyboard usually only contains a specific scene or action, so unnecessary redundant information can be reduced during feature extraction, thereby improving processing efficiency. In addition, the storyboard feature information stored in the storyboard feature library corresponds to the number of storyboards in the video. Compared with the feature information of the image frame, the storage capacity of the storyboard feature library is small, so it takes up less space.

[0052] The present application embodiment provides a video query method. Figure 2 A flow chart of a video query method provided by an embodiment of the present application is shown. Figure 2 As shown, the method includes steps S201 to S204.

[0053] Step S201: Obtain query information and multiple candidate videos.

[0054] The user can input query information through the dialogue model. The query information is the description information of the video that the user needs to find. The query information includes a text description or a video description of the relevant content of the video that the user needs to find. For example, if the user enters "children playing with a ball in the stadium" in the input box, the query information is a text description; if the user enters a video, the query information is a video description.

[0055] In this step, the candidate video can be a video in the video library, that is, an offline video, or a real-time video stream. For example, a video stream can be obtained in real time by accessing a camera, and each video stream obtained by the camera is called a candidate video.

[0056] Step S202: split each of the multiple candidate videos to obtain multiple storyboard segments corresponding to the candidate video, wherein the storyboard segment is a segment of video captured under the same scene in the candidate video.

[0057] This step splits the candidate video into multiple storyboard segments. It can be understood that the image frames in the same storyboard segment belong to the same shot, while the image frames in different storyboard segments belong to different shots. Generally speaking, two image frames in the same shot are shot in the same shooting environment and conditions, such as the same lighting conditions, shooting angle, shooting distance, etc., and the objects, backgrounds and other features in the image frames are similar. Therefore, the candidate video can be split by comparing whether adjacent image frames are similar and splitting between two dissimilar image frames.

[0058] In some embodiments, the candidate videos can be split manually. Specifically, the adjacent image frames in the candidate videos are manually compared to see if they are similar, and splitting is performed between two dissimilar image frames. Different from the prior art of directly manually reviewing the candidate videos and defining labels for the content of each video segment, this step manually only splits the candidate videos, which has low technical difficulty, low error rate, and high speed, thus improving processing efficiency.

[0059] In other embodiments, a video segmentation device is used to split the candidate video. The video segmentation device has a built-in model for identifying each image frame in the candidate image, and the feature information of the image frame is automatically identified through the model, wherein the feature information of the image frame includes the pixel value of the pixel point. Specifically, when the lens is switched, the pixel values ​​of adjacent frames usually change significantly, and this change is used by this step to accurately identify the boundary of the lens. As an example, the video segmentation device obtains the pixel value of the pixel point of the same coordinate of two adjacent image frames, and when the pixel value is greater than the threshold, the two adjacent image frames are segmented, then the previous image frame of the two image frames belongs to the previous lens, and the next image frame belongs to the next lens. The use of the video segmentation device can accurately capture the slight changes between the image frames, ensure the accuracy of the segmentation, avoid misjudging the slight changes in the same lens as the lens switch, and does not require additional metadata or prior knowledge. It can be segmented only by relying on the pixel information of the image frame itself, and has better versatility.

[0060] Step S203: extracting the storyboard feature information of the storyboard segments and constructing a storyboard feature library, wherein the storyboard feature library includes each storyboard feature information corresponding to the candidate video.

[0061] This step is explained below through a specific example. There are three candidate videos, namely candidate video A, candidate video B, and candidate video C. Candidate video A is split to obtain two storyboard segments, namely storyboard segment A1 and storyboard segment A2 of candidate video A. Then, the storyboard feature information a1 of storyboard segment A1 and the storyboard feature information a2 of storyboard segment A2 are extracted respectively, and then the storyboard feature information a1 and the storyboard feature information a2 are stored in the library. Then, the storyboard feature library stores the storyboard feature information a1 and the storyboard feature information a2, and the storyboard feature information a1 has a corresponding relationship with candidate video A, and the storyboard feature information a2 has a corresponding relationship with candidate video A.

[0062] The video splitting in this step means splitting a long video into multiple shorter segments, achieving dimensional degradation, which helps to simplify the subsequent processing steps. Specifically, after the video is split into segments, the content of each video is identified and feature information is extracted and stored in the database. What is stored in the database is the storyboard feature information rather than the image feature information, so the storage volume is small and the space occupied is small. At the same time, compared with directly processing the entire video, the data volume of the split video is small, which also reduces network delay and bandwidth occupancy, thereby reducing network transmission pressure.

[0063] Step S204: query the storyboard feature library according to the query information to determine the target video.

[0064] The storyboard feature library stores storyboard feature information, so this step searches the storyboard feature library for storyboard feature information that matches the query information through query information, and then finds the candidate video through the correspondence between the storyboard feature information and the candidate video, and determines the candidate video as the target video. Exemplarily, the storyboard feature library includes storyboard feature information a1, storyboard feature information a2, storyboard feature information b1, storyboard feature information b2, storyboard feature information b3, and storyboard feature information b4. If the storyboard feature information b3 matches the query information, then the determined target video is candidate video B, and candidate video B can be pushed to the user later.

[0065] In some of the embodiments, step S204 may further include: searching the storyboard feature library for candidate videos corresponding to storyboard features having a similarity with the query information greater than a second threshold, and determining the candidate videos as the target videos.

[0066] Specifically, this embodiment determines whether the query information matches the storyboard feature information by the size of the similarity: a large similarity indicates a high degree of match, and a small similarity indicates a low degree of match. The size of the similarity is compared with a preset second threshold, and if the similarity is greater than the second threshold, the candidate video corresponding to the storyboard feature information is pushed to the user as the target video. It can be understood that the second threshold is a value preset by prior knowledge, and the present application does not provide a cumbersome description of the value of the second threshold.

[0067] The traditional video feature extraction and retrieval method is based on the image comparison of image frames. When the video data is large, the information of the image frames stored will lead to bottlenecks in storage and transmission, and the efficiency is low. In the embodiment recorded in the above steps S201 to S204, the candidate video will be split, the storyboard feature information of each storyboard segment of the candidate video will be extracted, and the storyboard feature information will be stored in the library. In this way, when querying the video, it is only necessary to compare whether the query information and the storyboard feature information match, and the candidate video corresponding to the storyboard feature information that successfully matches will be output as the target video. In this way, the storage capacity of the stored video features is reduced, the space occupancy is small, and the data volume of the transmitted video is also reduced, so the network delay is low, the bandwidth occupancy is low, and the network transmission pressure is reduced. However, the transmission and storage both occupy a large amount of resources, which will bring about the problem of low video query efficiency. Therefore, the problem of low video query efficiency in the prior art is solved by this embodiment, and the video query efficiency can be improved.

[0068] In some of the embodiments, the above-mentioned splitting of the candidate video to obtain multiple storyboard segments includes: splitting the candidate video based on the shot boundary time points in the candidate video to obtain multiple storyboard segments, wherein the shot boundary time points are the segmentation time points of the video shots of the candidate video.

[0069] Specifically, by performing shot boundary detection on the candidate video, the shot boundary time point of the candidate video is determined. The shot boundary time point is the segmentation time point, that is, the image frame with a timestamp before the segmentation time point is divided into the video segment (or called storyboard segment) of the previous shot, and the image frame with a timestamp after the segmentation time point is divided into the video segment of the next shot. The video content in each video segment has high similarity, and the content of the video segment can be comprehensively understood by browsing the key frame images, thereby avoiding repeated processing of a large number of similar or identical video images, simplifying the complexity of video content analysis, and improving video processing efficiency.

[0070] In some of the embodiments, the above-mentioned splitting of the candidate video based on the shot boundary time points in the candidate video to obtain multiple storyboard segments includes: obtaining a pixel difference between two adjacent image frames in the candidate video, wherein the pixel difference includes a grayscale value; if the pixel difference is greater than a first threshold, taking any time point between the timestamps of the two image frames as a shot boundary time point in the candidate video; splitting the candidate video based on the shot boundary time points to obtain multiple storyboard segments.

[0071] In addition, the pixel difference also includes color value, brightness value, etc., which is not limited in this application.

[0072] The following is an example of comparing the difference in grayscale values. The grayscale difference of pixels at the same coordinates in two adjacent image frames is shown in formula (1). In formula (1), fn (i, j), f n+1 (i, j) represent the grayscale values ​​of the pixel (i, j) in the nth frame and the n+1th frame respectively. The total frame difference between two adjacent image frames is shown in formula (2). In formula (2), M and N are the sizes of the image. If the total frame difference F d (f n , f n+1 ) is greater than the first threshold, it is determined that the lens has changed.

[0073] f d (i, j) = |f n+1 (i, j)-f n (i, j)| Formula (1)

[0074]

[0075] By comparing the grayscale values ​​of pixels, different areas or targets in the video can be accurately identified. Specifically, different objects or scenes usually show different grayscale features in grayscale images, so the video can be divided into different parts based on these features. This segmentation method based on pixel grayscale values ​​can not only accurately capture the slight changes between image frames and ensure the accuracy of segmentation, but also does not require content recognition of the image frames of the entire video, simplifying the processing flow of the early video segmentation. Therefore, while simplifying the operation, it improves the efficiency of segmentation.

[0076] In some of the embodiments, a sample of an image frame is obtained for each storyboard segment by sampling frames to reduce the number of samples. Specifically, the extraction of storyboard feature information of the storyboard segments and the construction of a storyboard feature library include: extracting image frames from the storyboard segments according to the number of samples that is positively correlated with the duration of the storyboard segments; performing content recognition on the image frames to obtain storyboard feature information of the storyboard segments; and constructing a storyboard feature library, wherein the candidate videos in the storyboard feature library correspond to the storyboard feature information of the storyboard segments in the candidate videos.

[0077] In this embodiment, the frame extraction operation reduces the total data volume of the video by reducing the number of frames in the video, thereby saving storage space. This is particularly important for scenes that need to store a large amount of video data, such as surveillance video storage, online video platforms, and the like. In addition, during video transmission, reducing the number of frames can reduce the amount of data required for transmission, thereby speeding up the transmission speed and reducing the transmission time. This is particularly beneficial for real-time video transmission or bandwidth-constrained environments such as mobile networks. The number of samples is positively correlated with the duration of the storyboard segment, which means that the longer the storyboard segment, the more samples are collected. This is to ensure that the number of samples extracted is reasonable, and to avoid extracting too few samples and losing key information in the storyboard segment, resulting in a low subsequent recognition rate. Therefore, through this embodiment, while ensuring the content integrity of the storyboard segment, the resources consumed by calculation, transmission, and storage are reduced by extracting frames.

[0078] In some of the embodiments, content recognition of the extracted image frames is achieved by training a multimodal training model of image-text semantics. Specifically, the above-mentioned content recognition of the image frames to obtain the storyboard feature information of the storyboard segment includes: using a preset multimodal training model of image-text semantics to perform content recognition on the image frames to obtain a representation vector of the image frames; clustering the representation vectors of each image frame to obtain multiple clusters, wherein the center vector in each cluster is determined as the storyboard feature information of the storyboard segment.

[0079] Specifically, a CNN neural network, such as CLIP4Clip, can be used to extract the representation vector of the image frame. Compared with conventional neural networks such as ResNet50, CLIP4Clip generates text and video embedding by jointly training a video encoder and a text encoder, that is, the query information input by the user can be embedded as text and input into the text encoder, and the candidate video can be embedded as a video and input into the video encoder. The encoder recognizes the query information to obtain the text feature vector, and semantically understands the candidate recognition to obtain the center vector of the video shot description. CLIP4Clip then uses the vector comparison algorithm of the Euclidean distance (i.e., L2) to calculate the similarity between the text feature vector and the center vector (i.e., the shot vector), and can use the topk parameter to retrieve several most similar shot vectors at a time, and use the specified vec_distance function to calculate the distance between the two vectors in the verification result. The verification result is used to evaluate the similarity (or degree of difference) between the predicted vector and the true vector, thereby evaluating the performance of CLIP4Clip.

[0080] Among them, Euclidean distance (L2) is a way to measure the distance between two points. It is also one of the most commonly used distance calculation methods and is widely used in the field of computer vision. The algorithm calculates the shortest straight-line distance between two points. If the Euclidean distance between two sample points is small, it means that the two sample points are relatively close and may belong to the same category. The Euclidean distance is shown below:

[0081]

[0082] Among them, a = (a1, a2, ..., an) and b = (b1, b2, ..., bn) are two points in n-dimensional Euclidean space.

[0083] The video query method provided in the embodiment of the present application is described and illustrated below through specific examples. Figure 3 A system architecture diagram of a video query method provided by an embodiment of the present application is shown. Figure 3 As shown, the application system of the method includes three modules: video clip feature extraction and storage 310, data storage 320, and retrieval processing 330.

[0084] The video clip features are extracted and stored in 310. The system gateway layer connects the user layer and the application layer and is responsible for receiving and parsing the converged media video. Unified formatting and integrity verification are performed at the system gateway layer to eliminate the impact of subsequent processing caused by differences in the bit rate, resolution, encoding format, etc. of the converged media video, and the converged media video is more compatible. After the processing is completed, the candidate video is obtained, and metadata is parsed for the candidate video, such as recording the timestamps, video names, shooters, and shooting locations of the candidate video. The candidate video is also subjected to a lens algorithm analysis. Specifically, the grayscale difference algorithm mentioned above can be used to perform lens cutting on the candidate video based on the lens algorithm analysis results. The multiple storyboard segments obtained by cutting are input into the CNN deep neural network for content recognition, and the obtained vector data is spliced ​​with the metadata to obtain the storyboard feature information.

[0085] The data storage 320 mainly adopts the object file storage method, creates a vector database through MySQL, and stores the storyboard feature information in it.

[0086] Retrieval processing 330 refers to calculating the similarity between description vectors in the CNN deep neural network, and outputting the results by controlling the quantity and distance, and outputting the candidate videos with similarity greater than the first threshold as target videos to the user.

[0087] Figure 4 A schematic diagram of a video query method provided by an embodiment of the present application is shown. Figure 4As shown, the method first inputs the converged media video, where the converged media video refers to the candidate video after the unified video formatting process. The lens features are extracted and converted into video feature vectors (equivalent to the above storyboard feature information), and then the video feature vectors are saved in the feature vector database. When the user enters the content scene description (equivalent to the above query information), the content scene description is converted into a semantic understanding vector. The topk retrieval method is used in the feature vector database. First, the Euclidean distance between the semantic understanding vector and the video feature vector is calculated, and then k video feature vectors with high similarity are output. The results are then filtered and returned through the vec_distance function, and finally the results are responded to the user.

[0088] In summary, the video query method provided in the embodiment of the present application extracts the video according to the frame pictures of the shot fragments, and the CNN deep neural network model understands the vector data of the video fragment feature description, and saves the video metadata and vector data to the database; finally, according to the scene fragment description input by the user, the fragment feature vector database is retrieved after semantic vectorization, and the retrieval results are subjected to Euclidean distance screening and normalization to obtain the final result. This query method splits the integrated media video from the whole piece to the scene fragments and then to the frame pictures of the fragments, realizing the dimensionality degradation, which saves storage and eliminates the possibility of network transmission becoming a bottleneck. This granular hierarchical splitting makes the entire feature extraction faster and more efficient than the traditional frame-by-frame image processing. In addition, this embodiment performs unified video formatting at the entry layer of video input, which is more compatible with integrated media videos and eliminates the subsequent processing effects caused by differences in bit rate, resolution, encoding format, etc. of integrated media diversified videos.

[0089] The embodiments of the present application also provide a video query device, which is used to implement the above embodiments and preferred implementation modes, and will not be repeated hereafter. The terms "module", "unit", "subunit", etc. used below may be a combination of software and / or hardware that implements a predetermined function. Although the devices described in the following embodiments are preferably implemented in software, implementation in hardware, or a combination of software and hardware, is also possible and contemplated.

[0090] Figure 5 A structural block diagram of a video query device provided by an embodiment of the present application is shown. Figure 5 As shown, the device comprises:

[0091] An acquisition module 51 is used to acquire query information and multiple candidate videos;

[0092] A splitting module 52 is used to split each of the multiple candidate videos to obtain multiple storyboard segments corresponding to the candidate video, wherein the storyboard segment is a segment of video under the same scene shot in the candidate video;

[0093] A processing module 53 is used to extract the storyboard feature information of the storyboard segments and construct a storyboard feature library, wherein the storyboard feature library includes the storyboard feature information corresponding to each candidate video;

[0094] The query module 54 is used to query the storyboard feature library according to the query information to determine the target video.

[0095] In some of the embodiments, the splitting module 52 is further used to split the candidate video based on the shot boundary time points in the candidate video to obtain multiple storyboard segments, wherein the shot boundary time points are the segmentation time points of the video shots of the candidate video.

[0096] In some of the embodiments, the splitting module 52 is further used to obtain a pixel difference between two adjacent image frames in the candidate video, wherein the pixel difference includes a grayscale value; if the pixel difference is greater than a first threshold, any time point between the timestamps of the two image frames is used as a shot boundary time point in the candidate video; and the candidate video is split based on the shot boundary time points to obtain a plurality of storyboard segments.

[0097] In some of the embodiments, the processing module 53 is further used to extract image frames from the storyboard segments according to a number of samples that is positively correlated with the duration of the storyboard segments; perform content recognition on the image frames to obtain storyboard feature information of the storyboard segments; and construct a storyboard feature library, wherein the candidate videos in the storyboard feature library correspond to the storyboard feature information of the storyboard segments in the candidate videos.

[0098] In some of the embodiments, the processing module 53 is further used to record the start time point and the end time point of each storyboard segment in the candidate video; the duration between the start time point and the end time point is used as the duration of the storyboard segment, and image frames are extracted from the storyboard segment according to a sampling number that is positively correlated with the duration.

[0099] In some of the embodiments, the processing module 53 is further used to perform content recognition on the image frame using a preset image-text semantic multimodal training model to obtain a representation vector of the image frame;

[0100] The representation vectors of each image frame are clustered to obtain a plurality of clusters, wherein the center vector in each cluster is determined as the storyboard feature information of the storyboard segment.

[0101] In some of the embodiments, the query module 54 is further used to query the storyboard feature library for candidate videos corresponding to storyboard features having a similarity with the query information greater than a second threshold, and determine them as target videos.

[0102] It should be noted that the above modules can be functional modules or program modules, and can be implemented by software or hardware. For modules implemented by hardware, the above modules can be located in the same processor; or the above modules can be located in different processors in any combination.

[0103] In this embodiment, an electronic device is further provided, including a memory and a processor, wherein a computer program is stored in the memory, and the processor is configured to run the computer program to execute the steps in any one of the above method embodiments.

[0104] Optionally, the electronic device may further include a transmission device and an input / output device, wherein the transmission device is connected to the processor, and the input / output device is connected to the processor.

[0105] Optionally, in this embodiment, the processor may be configured to execute the following steps S1 to S4 through a computer program.

[0106] S1. Obtain query information and multiple candidate videos.

[0107] S2. Split each of the multiple candidate videos to obtain multiple storyboard segments corresponding to the candidate video, wherein the storyboard segment is a segment of video captured by the same scene in the candidate video.

[0108] S3. Extracting storyboard feature information of the storyboard segments and constructing a storyboard feature library, wherein the storyboard feature library includes each storyboard feature information corresponding to the candidate video.

[0109] S4. According to the query information, query the storyboard feature library to determine the target video.

[0110] It should be noted that the specific examples in this embodiment can refer to the examples described in the above embodiments and optional implementation modes, and will not be repeated in this embodiment.

[0111] In addition, in combination with the method provided in the above embodiments, a storage medium may be provided in this embodiment to implement the method. The storage medium stores a computer program; when the computer program is executed by a processor, any one of the video query methods in the above embodiments is implemented.

[0112] The embodiment of the present application also provides a computer program product. When the computer program product is run on a computer, the computer executes each function or step executed by the processor in the above method embodiment.

[0113] It should be understood that the specific embodiments described herein are only used to explain the application, rather than to limit it. Based on the embodiments provided in this application, all other embodiments obtained by ordinary technicians in this field without creative work are within the protection scope of this application.

[0114] Obviously, the drawings are only some examples or embodiments of the present application. For ordinary technicians in the field, the present application can also be applied to other similar situations based on these drawings without creative work. In addition, it is understandable that although the work done in this development process may be complicated and lengthy, for ordinary technicians in the field, certain changes in design, manufacturing or production based on the technical content disclosed in this application are only conventional technical means and should not be regarded as insufficient content disclosed in this application.

[0115] The term "embodiment" in this application refers to a specific feature, structure or characteristic described in conjunction with the embodiment that can be included in at least one embodiment of the present application. The appearance of this phrase in various locations in the specification does not necessarily mean the same embodiment, nor does it mean that it is mutually exclusive with other embodiments and is independent or optional. It is clearly or implicitly understood by those of ordinary skill in the art that the embodiments described in this application can be combined with other embodiments without conflict.

[0116] The above-mentioned embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of patent protection. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the scope of protection of the present application. Therefore, the scope of protection of the present application shall be subject to the attached claims.

Claims

1. A video query method, characterized in that: The method comprises: Obtain query information and multiple candidate videos; Splitting each candidate video among the multiple candidate videos to obtain multiple storyboard segments corresponding to the candidate video, wherein the storyboard segment is a segment of video under the same scene shot in the candidate video; Extracting the storyboard feature information of the storyboard segments and constructing a storyboard feature library, wherein the storyboard feature library includes each storyboard feature information corresponding to the candidate video; According to the query information, the storyboard feature library is queried to determine the target video.

2. The video query method according to claim 1, characterized in that: The step of splitting each of the plurality of candidate videos to obtain a plurality of storyboard segments corresponding to the candidate video includes: Based on the shot boundary time points in the candidate video, the candidate video is split to obtain a plurality of storyboard segments, wherein the shot boundary time points are the segmentation time points of the video shots of the candidate video.

3. The video query method according to claim 2, characterized in that: The step of splitting the candidate video based on the shot boundary time points in the candidate video to obtain a plurality of storyboard segments includes: Obtaining a pixel difference between two adjacent image frames in the candidate video, wherein the pixel difference includes a grayscale value; If the pixel difference is greater than a first threshold, any time point between the timestamps of the two image frames is used as a shot boundary time point in the candidate video; Based on the shot boundary time points, the candidate video is split to obtain a plurality of storyboard segments.

4. The video query method according to claim 1, characterized in that: The step of extracting the storyboard feature information of the storyboard segments and constructing a storyboard feature library comprises: Extracting image frames from the storyboard segment according to a sampling quantity that is positively correlated with the duration of the storyboard segment; Performing content recognition on the image frame to obtain storyboard feature information of the storyboard segment; A storyboard feature library is constructed, wherein the candidate video in the storyboard feature library corresponds to the storyboard feature information of the storyboard segments in the candidate video.

5. The video query method according to claim 4, characterized in that: The step of extracting image frames from the storyboard segments according to the number of samples that is positively correlated with the duration of the storyboard segments comprises: Recording the starting time point and the ending time point of each storyboard segment in the candidate video; The duration between the start time point and the end time point is used as the duration of the storyboard segment, and image frames are extracted from the storyboard segment according to a sampling quantity that is positively correlated with the duration.

6. The video query method according to claim 4, characterized in that: The performing content recognition on the image frame to obtain the storyboard feature information of the storyboard segment includes: Using a preset multimodal training model of image-text semantics to perform content recognition on the image frame to obtain a representation vector of the image frame; The representation vectors of each of the image frames are clustered to obtain a plurality of clusters, wherein a central vector in each of the clusters is determined as the storyboard feature information of the storyboard segment.

7. The video query method according to any one of claims 1 to 6, characterized in that: The step of searching the storyboard feature library according to the query information to determine the target video includes: The candidate video corresponding to the storyboard feature whose similarity with the query information is greater than a second threshold is searched from the storyboard feature library and determined as the target video.

8. A video query device, characterized in that: include: An acquisition module, used to acquire query information and multiple candidate videos; A splitting module is used to split each of the multiple candidate videos to obtain multiple storyboard segments corresponding to the candidate video, wherein the storyboard segment is a segment of video under the same scene shot in the candidate video; A processing module, used to extract the storyboard feature information of the storyboard segments and construct a storyboard feature library, wherein the storyboard feature library includes the storyboard feature information corresponding to each of the candidate videos; The query module is used to query the storyboard feature library according to the query information to determine the target video.

9. A terminal comprising a memory and a processor, characterized in that: A computer program is stored in the memory, and the processor is configured to run the computer program to execute the video query method according to any one of claims 1 to 7.

10. A storage medium, characterized in that: The storage medium stores a computer program, wherein the computer program is configured to execute the video query method according to any one of claims 1 to 7 when running.

Images