Audio and video matching method and device, terminal and storage medium

Abstract

The invention relates to an audio and video matching method and device, a terminal and a storage medium. The method comprises the following steps: acquiring query information and a plurality of candidate videos; splitting each candidate video in the plurality of candidate videos to obtain a plurality of sub-lens segments corresponding to the candidate videos, the sub-lens segments being a segment of video under the same scene lens in the candidate videos; the sub-lens feature information of the sub-lens segments is extracted, a sub-lens feature library is constructed, and the sub-lens feature library comprises each piece of sub-lens feature information corresponding to the candidate videos; and querying the sub-mirror feature library according to the query information, and determining a target video. According to the invention, the technical problem of low audio and video matching efficiency in the prior art is solved.

Description

Technical Field

[0001] The present application relates to the technical field of audio and video processing, and in particular to an audio and video matching method, device, terminal and storage medium. Background Art

[0002] At present, the media industry is in a stage of rapid development where information can be quickly gathered, disseminated in multiple ways, and accurately reached. At the same time, the number of converged media audio and video on the Internet is increasing, and the growth rate is getting faster and faster, and the reuse of the same audio and video clips is becoming more and more common. For audio and video data, due to the inability to structure and abstract the content features of converged media audio and video, comparison and retrieval has become a major problem. It is difficult for users to determine whether existing audio and video clips are used in converged media audio and video or whether converged media audio and video are used as audio and video clips in other audio and video.

[0003] Currently, audio and video matching is mainly achieved by extracting the video frames of the audio and video to be tested as features and comparing them frame by frame with the target audio and video. However, this method requires comparing the audio and video to be tested frame by frame to see if they match the target audio and video, which results in a relatively low matching efficiency. In addition, this method only compares whether the video part of the audio and video to be tested matches the target audio and video, but cannot detect the audio part of the audio and video to be tested, which reduces the matching accuracy.

[0004] In summary, there is currently no effective solution to the problem of low efficiency and accuracy of audio and video matching in related technologies. Summary of the invention

[0005] The embodiments of the present application provide an audio and video matching method, device, terminal and storage medium to solve the technical problem of low efficiency and accuracy of audio and video matching in the prior art.

[0006] To achieve the above objectives, the embodiments of the present application adopt the following technical solutions: In a first aspect, an embodiment of the present application provides an audio and video matching method, the method comprising: Decode the audio and video to be tested to obtain the audio and video to be tested corresponding to the audio and video to be tested; Perform splitting and frame extraction processing on the video to be tested to obtain storyboard segments and video frames corresponding to the video to be tested, wherein the storyboard segment is a segment of video under the same scene lens in the video to be tested; Extracting the storyboard feature information of the storyboard segments, extracting the frame feature information of the video frames, and extracting the audio feature information of the audio to be tested; According to the storyboard feature information, frame feature information and audio feature information, the pre-built audio and video fingerprint feature library is queried to determine the target audio and video that matches the audio and video to be tested.

[0007] In combination with the first aspect, in a possible design, the pre-constructed audio and video fingerprint feature library includes a storyboard feature library, a frame feature library, and an audio feature library; the pre-constructed audio and video fingerprint feature library is queried according to the storyboard feature information, the frame feature information, and the audio feature information to determine the target audio and video matching the audio and video to be tested, including: According to the storyboard feature information, a storyboard feature library is queried to determine multiple candidate audio and video; Matching the frame feature information with the frame feature corresponding to each candidate audio and video in the frame feature library, and matching the audio feature information with the audio feature corresponding to each candidate audio and video in the audio feature library; When the frame feature of the candidate audio and video matches the frame feature information, and the audio feature of the candidate audio and video matches the audio feature information, the candidate audio and video is determined to be the target audio and video.

[0008] In combination with the first aspect, in a possible design manner, after determining that the candidate audio and video is the target audio and video, the method further includes: The storyboard feature information corresponding to the audio and video to be tested is stored in the storyboard feature library, the frame feature information corresponding to the audio and video to be tested is stored in the frame feature library, and the audio feature information corresponding to the audio and video to be tested is stored in the audio feature library.

[0009] In combination with the first aspect, in a possible design, a storyboard feature library is queried according to the storyboard feature information to determine that a plurality of candidate audio and video include: The audio and video corresponding to the storyboard features in the storyboard feature library whose similarity with the storyboard feature information is greater than a first threshold are determined as candidate audio and video.

[0010] In combination with the first aspect, in a possible design, the splitting process includes: Based on the shot boundary time points in the video to be tested, the video to be tested is split to obtain split-shot segments, wherein the shot boundary time points are the split time points of the video shots of the video to be tested; and Frame extraction processing includes: The video frames are extracted from the video to be tested according to a sampling number that is positively correlated with the duration of the video to be tested; or the video frames are extracted from the video to be tested according to a preset sampling number.

[0011] In combination with the first aspect, in a possible design, the video to be tested is split based on the shot boundary time points in the video to be tested, and the obtained storyboard segments include: Obtaining a pixel difference between two adjacent video frames in the video to be tested, wherein the pixel difference includes a grayscale value; If the pixel difference is greater than the second threshold, any time point between the timestamps of the two video frames is taken as a shot boundary time point in the video to be tested; The video to be tested is split based on the shot boundary time points to obtain storyboard segments.

[0012] In combination with the first aspect, in a possible design, extracting the storyboard feature information of the storyboard segment includes: Extracting video frames from the storyboard segments according to a number of samples that is positively correlated with the duration of the storyboard segments; Perform content recognition on the video frames to obtain the storyboard feature information of the storyboard segments.

[0013] In combination with the first aspect, in a possible design, content recognition is performed on a video frame to obtain storyboard feature information of a storyboard segment, including: Use the preset multimodal training model of image and text semantics to perform content recognition on the video frame and obtain the representation vector of the video frame; The representation vectors of each video frame are clustered to obtain a plurality of clusters, wherein the center vector in each cluster is determined as the storyboard feature information of the storyboard segment.

[0014] In combination with the first aspect, in a possible design manner, extracting frame feature information of a video frame includes: A preset residual network model is used to perform content recognition on the video frame to obtain a representation vector of the video frame, and the representation vector of the video frame is used as the frame feature information of the video frame.

[0015] In combination with the first aspect, in a possible design manner, extracting audio feature information of the audio to be tested includes: Using a preset audio fingerprint neural network model to perform content recognition on the audio to be tested, and obtaining multiple representation vectors of the audio to be tested; Each characterization vector of the audio to be tested is clustered to obtain a plurality of clusters, wherein the center vector in each cluster is determined as the audio feature information of the audio to be tested.

[0016] In a second aspect, an embodiment of the present application provides an audio and video matching device, including: A decoding module is used to decode the audio and video to be tested to obtain the audio and video to be tested corresponding to the audio and video to be tested; A processing module is used to split and extract frames of the video to be tested to obtain storyboard segments and video frames corresponding to the video to be tested, wherein the storyboard segment is a segment of video under the same scene lens in the video to be tested; An extraction module, used to extract the shot feature information of the shot segments, extract the frame feature information of the video frames, and extract the audio feature information of the audio to be tested; The query module is used to query the pre-built audio and video fingerprint feature library according to the storyboard feature information, frame feature information and audio feature information to determine the target audio and video that matches the audio and video to be tested.

[0017] In a third aspect, an embodiment of the present application provides a terminal, including a memory and a processor, wherein a computer program is stored in the memory, and the processor is configured to run the computer program to execute the method of the first aspect and possible design methods thereof.

[0018] In a fourth aspect, an embodiment of the present application provides a storage medium, in which a computer program is stored, wherein the computer program is configured to execute the method of the first aspect and its possible design method when running.

[0019] Compared with the prior art, an audio and video matching method, device, terminal and storage medium provided in an embodiment of the present application decodes the audio and video to be tested to obtain the audio and video to be tested corresponding to the audio and video to be tested, splits and extracts frames of the video to be tested to obtain the storyboard segments and video frames corresponding to the video to be tested; extracts the storyboard feature information of the storyboard segments, extracts the frame feature information of the video frames, and extracts the audio feature information of the audio to be tested; queries a pre-constructed audio and video fingerprint feature library based on the storyboard feature information, frame feature information and audio feature information to determine the target audio and video that matches the audio and video to be tested; through the present application, when matching audio and video, it is possible to query the audio and video fingerprint feature library based on the storyboard feature information, frame feature information and audio feature information to see whether there are various features that match the above feature information, and output the candidate audio and video corresponding to the various features that are successfully matched as the target audio and video. Since audio and video matching can be achieved without frame-by-frame comparison, the efficiency of audio and video matching is improved. At the same time, matching the audio and video to be tested based on the three types of feature information can split the audio and video granularity more finely, thereby improving the accuracy of audio and video matching. Therefore, this embodiment solves the problem of low efficiency and accuracy of audio and video matching in the prior art, and achieves the technical effect of improving the efficiency and accuracy of audio and video matching.

[0020] Details of one or more embodiments of the present application are set forth in the following drawings and description to make other features, objects, and advantages of the present application more readily apparent. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings: Figure 1 A hardware structure block diagram of a terminal provided in an embodiment of the present application is shown; Figure 2A flowchart of an audio and video matching method provided in an embodiment of the present application is shown; Figure 3 A system architecture diagram of an audio and video matching method provided in an embodiment of the present application is shown; Figure 4 A schematic diagram of an audio and video matching method provided in an embodiment of the present application is shown; Figure 5 A structural block diagram of an audio and video matching device provided in an embodiment of the present application is shown. DETAILED DESCRIPTION

[0022] In order to more clearly understand the purpose, technical solutions and advantages of the present application, the present application is described and illustrated below in conjunction with the accompanying drawings and embodiments.

[0023] Unless otherwise defined, the technical terms or scientific terms involved in this application shall have the general meaning understood by people with general skills in the technical field to which this application belongs. The words "one", "a", "the", "these" and the like in this application do not indicate a quantitative limitation, and they may be singular or plural. The terms "include", "comprise", "have" and any variants thereof involved in this application are intended to cover non-exclusive inclusions; for example, a process, method and system, product or device comprising a series of steps or modules (units) is not limited to the listed steps or modules (units), but may include unlisted steps or modules (units), or may include other steps or modules (units) inherent to these processes, methods, products or devices. The words "connect", "connected", "coupled" and the like involved in this application are not limited to physical or mechanical connections, but may include electrical connections, whether direct or indirect. The "multiple" involved in this application refers to two or more. "And / or" describes the association relationship of associated objects, indicating that there may be three relationships, for example, "A and / or B" may mean: A exists alone, A and B exist at the same time, and B exists alone. Generally, the character " / " indicates that the objects associated with each other are in an "or" relationship. The terms "first", "second", "third", etc. in this application are only used to distinguish similar objects and do not represent a specific ordering of the objects.

[0024] The audio and video matching method provided in the embodiment of the present application can be executed in a terminal or a computer device. Taking running on a terminal as an example, Figure 1 FIG. 1 shows a hardware structure block diagram of a terminal provided in an embodiment of the present application. Figure 1 As shown, the terminal may include one or more ( Figure 1Only one is shown in the figure) processor 102, and memory 104 for storing data. Processor 102 may include but is not limited to a processing device such as a microprocessor MCU or a programmable logic device FPGA. Optionally, the terminal may also include a transmission device 106 and an input / output device 108 for communication functions. It can be understood by those skilled in the art that Figure 1 The structure shown is for illustration only and does not limit the structure of the above terminal. Figure 1 More or fewer components as shown, or with Figure 1 Different configurations are shown.

[0025] The memory 104 can be used to store computer programs, for example, software programs and modules of application software. The processor 102 executes various functional applications and data processing by running the computer programs stored in the memory 104, that is, to implement the above method. The memory 104 may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some examples, the memory 104 may further include a memory remotely arranged relative to the processor 102, and these remote memories may be connected to the terminal via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0026] The transmission device 106 is used to receive or send data via a network. The specific example of the above network may include a wireless network provided by a communication provider of the terminal. In one example, the transmission device 106 includes a network adapter (Network Interface Controller, referred to as NIC), which can be connected to other network devices through a base station so as to communicate with the Internet. In one example, the transmission device 106 can be a radio frequency (Radio Frequency, referred to as RF) module, which is used to communicate with the Internet wirelessly.

[0027] The various technologies described in the embodiments of the present application can be used for audio and video matching, that is, the user inputs the audio and video to be tested, and the website or application will return to the user the target audio and video with the highest similarity to the audio and video to be tested. It is mainly based on the storyboard feature information, frame feature information and audio feature information of the audio and video to be tested, and queries whether there are various features that match the above feature information in the pre-built audio and video fingerprint feature library, and outputs the candidate audio and video corresponding to the various features that are successfully matched as the target audio and video. At present, in the audio and video matching scenario, the method of matching each video frame in the audio and video to be tested one by one will cause a large amount of resources to be occupied in calculation, transmission, and storage, so it is inefficient. In addition, some audio and video matching methods will achieve audio and video matching based on the hash features of the video. By extracting multiple video frames from the audio and video to be tested, the features of each extracted video frame are connected to the video feature hash layer composed of the sigmoid activation function, the adjacent frame pair loss function, and the category cross entropy loss function for calculation to obtain the hash features of the audio and video to be tested. However, this method has the problem that the video context association relationship may be lost and the high-dimensional features are insufficient. These problems also limit the efficiency and effect of video hashing technology in processing video data; In addition, some audio and video matching methods use neural networks such as ResNet50, which mainly rely on manually designed features and rules, such as edge detection, color segmentation, feature extraction, etc., so they are only suitable for processing simple images, and the recognition accuracy for complex images is low. Training a specific model dedicated to identifying the labels of video frames requires a high cost, and a small or unbalanced training data set may cause the model to overfit or lack of model generalization ability.

[0028] In view of this, the method provided in the embodiment of the present application decodes the audio and video to be tested to obtain the audio and video to be tested corresponding to the audio and video to be tested, splits and extracts frames of the video to be tested to obtain the storyboard segments and video frames corresponding to the video to be tested; extracts the storyboard feature information of the storyboard segments, extracts the frame feature information of the video frames, and extracts the audio feature information of the audio to be tested; queries a pre-constructed audio and video fingerprint feature library based on the storyboard feature information, the frame feature information, and the audio feature information to determine the target audio and video that matches the audio and video to be tested; through the present application, the target audio and video that matches the audio and video can be determined based on the storyboard feature information when matching audio and video. The audio and video fingerprint feature library is queried for various features that match the above feature information based on the information, frame feature information and audio feature information, and the candidate audio and video corresponding to the various features that match successfully are output as the target audio and video. The audio and video matching can be achieved without frame by frame comparison, thereby improving the efficiency of audio and video matching. At the same time, matching the audio and video to be tested based on the three types of feature information can split the audio and video granularity more finely and improve the accuracy of audio and video matching. Therefore, this embodiment solves the problem of low efficiency and accuracy of audio and video matching in the prior art, and achieves the technical effect of improving the efficiency and accuracy of audio and video matching.

[0029] The present application embodiment provides an audio and video matching method. Figure 2 A flowchart of an audio and video matching method provided in an embodiment of the present application is shown. Figure 2 As shown, the method includes steps S201 to S204.

[0030] Step S201: Decode the audio and video to be tested to obtain the audio and video to be tested corresponding to the audio and video to be tested.

[0031] The user can input the audio and video to be tested through the dialogue model, and can decode the audio and video to be tested through various audio and video decoders to obtain the audio and video to be tested corresponding to the audio and video to be tested.

[0032] In this step, the audio and video to be tested can be audio and video in the video library, that is, offline audio and video, and the audio and video to be tested can also be real-time audio and video streams. For example, the audio and video streams can be obtained in real time by connecting to a camera, and the audio and video streams obtained by each camera are called an audio and video to be tested.

[0033] Step S202: splitting and extracting frames of the video to be tested to obtain storyboard segments and video frames corresponding to the video to be tested, wherein the storyboard segment is a segment of video captured under the same scene in the video to be tested.

[0034] The splitting process in this step can split the video to be tested into multiple storyboard segments. It can be understood that the video frames in the same storyboard segment belong to the same shot, while the video frames in different storyboard segments belong to different shots. Generally speaking, two video frames in the same shot are shot under the same shooting environment and conditions, such as the same lighting conditions, shooting angle, shooting distance, etc., and the objects, backgrounds and other features in the video frames are similar. Therefore, the video to be tested can be split by comparing whether adjacent video frames are similar and splitting between two dissimilar video frames.

[0035] In some embodiments, the video to be tested can be split manually. Specifically, the adjacent video frames in the video to be tested are manually compared to see if they are similar, and splitting is performed between two dissimilar video frames. Different from the prior art, which directly manually reviews the candidate audio and video and defines labels for the content of each video segment, this step manually only splits the video to be tested, which has low technical difficulty, low error rate, and high speed, thus improving processing efficiency.

[0036] In other embodiments, a video segmentation device can be used to split the video to be tested. The video segmentation device has a built-in model for identifying each video frame in the video to be tested, and the feature information of the video frame is automatically identified through the model, wherein the feature information of the video frame includes the pixel value of the pixel point. Specifically, when the lens is switched, the pixel values ​​of adjacent frames usually change significantly, and this change is used by this step to accurately identify the boundary of the lens. As an example, the video segmentation device obtains the pixel value of the pixel point of the same coordinate of two adjacent video frames. When the pixel value is greater than the threshold, the two adjacent video frames are segmented, and the previous video frame of the two video frames belongs to the previous lens, and the next video frame belongs to the next lens. The use of the video segmentation device can accurately capture the slight changes between the video frames, ensure the accuracy of the segmentation, avoid misjudging the slight changes in the same lens as the lens switch, and does not require additional metadata or prior knowledge. It can be segmented only by relying on the pixel information of the video frame itself, and has better versatility.

[0037] The frame extraction process in this step may include: extracting video frames from the video to be tested according to a sampling number that is positively correlated with the duration of the video to be tested; or extracting video frames from the video to be tested according to a preset sampling number.

[0038] The frame extraction process in this step is explained below by a specific example. If the duration of the video to be tested is 12 seconds, 60 video frames can be extracted from the video to be tested at a sampling frequency of 5 frames / second, wherein the sampling frequency can be designed according to the actual application scenario, and the embodiment of the present application does not limit this; or, the preset number of samples can be determined to be 100. Assuming that the duration of the video to be tested is 50 seconds, 100 video frames can be extracted from the video to be tested at a sampling frequency of 100 / 50=2 frames / second, wherein the preset number of samples can be designed according to the actual application scenario, and the embodiment of the present application does not limit this. Specifically, OpenCV (a cross-platform computer vision and machine learning software library) can be used to perform frame extraction processing on the video to be tested.

[0039] Frame extraction reduces the total data volume of the video by reducing the number of frames in the video to be tested, thereby saving storage space. This is particularly important for scenarios that require storage of a large amount of video data, such as surveillance video storage, online video platforms, and the like. In addition, during video transmission, reducing the number of frames can reduce the amount of data required for transmission, thereby speeding up the transmission speed and reducing the transmission time. This is particularly beneficial for real-time video transmission or bandwidth-constrained environments such as mobile networks. The number of samples can be positively correlated with the duration of the video to be tested, meaning that the longer the video to be tested, the more samples are collected, in order to ensure that the number of samples extracted is reasonable, and to avoid the loss of key information in the video to be tested due to too few samples, resulting in a low subsequent recognition rate. Therefore, by using this embodiment, while ensuring the content integrity of the video to be tested, the resources consumed by calculation, transmission, and storage are reduced by extracting frames.

[0040] Step S203: extracting the storyboard feature information of the storyboard segments, extracting the frame feature information of the video frames, and extracting the audio feature information of the audio to be tested.

[0041] In step S202 to step S203, the long video to be tested can be split into multiple shorter storyboard segments through splitting processing, and the dimensional degradation processing is realized, which helps to simplify the subsequent processing steps. For example, after the video to be tested is split into storyboard segments, the content of each storyboard segment is identified and the storyboard feature information is extracted and incorporated into the preset storyboard feature library. What is stored in the library is the storyboard feature information rather than the image feature information, so the storage capacity is small, so the space occupancy is small. In addition, a small number of video frames can be extracted from the video to be tested through frame extraction processing, and there is no need to extract features one by one for each video frame of the video to be tested, thereby reducing the processing amount.

[0042] Step S204: query a pre-built audio and video fingerprint feature library based on the storyboard feature information, frame feature information, and audio feature information to determine a target audio and video that matches the audio and video to be tested.

[0043] In this embodiment, the pre-built audio and video fingerprint feature library may include multiple audio and video fingerprint features of pre-input audio and video, and the audio and video fingerprint features may include storyboard features, frame features and audio features corresponding to the pre-input audio and video.

[0044] Specifically, based on the storyboard feature information, storyboard features similar to the storyboard feature information can be searched in the audio and video fingerprint feature library, and the pre-input audio and video corresponding to the storyboard features similar to the storyboard feature information can be used as a candidate audio and video range; then, it can be determined whether the frame features corresponding to each candidate audio and video in the candidate audio and video range match the frame feature information of the audio and video to be tested, and it can be determined whether the audio features corresponding to each candidate audio and video in the candidate audio and video range match the audio feature information of the audio and video to be tested. Only when the frame features corresponding to the candidate audio and video match the frame feature information and the audio features corresponding to the candidate audio and video match the audio feature information, can the candidate audio and video be determined to be the target audio and video that matches the audio and video to be tested.

[0045] In this way, matching the audio and video to be tested based on the three types of feature information can split the audio and video granularity more finely, thereby improving the accuracy of audio and video matching; in addition, there is no coupling relationship between the three modalities corresponding to the three types of feature information, that is, a rough comparison of storyboard fragments can be achieved, and more accurate video frame matching and audio matching can be achieved, further improving the accuracy of audio and video matching.

[0046] In some embodiments, the pre-built audio and video fingerprint feature library includes a storyboard feature library, a frame feature library, and an audio feature library; the above step S204 may include the following steps: Step 1: According to the storyboard feature information, query the storyboard feature library to determine multiple candidate audio and video.

[0047] The following is an explanation of the storyboard feature library in this step through a specific example: the storyboard feature library stores storyboard features, so this step searches the storyboard feature library for storyboard features that match the storyboard feature information through the storyboard feature information, and then finds the candidate audio and video through the correspondence between the storyboard features and the candidate audio and video. Exemplarily, the storyboard feature library includes storyboard feature a1 (corresponding to the candidate audio and video A1), storyboard feature a2 (corresponding to the candidate audio and video A2), storyboard feature b1 (corresponding to the candidate audio and video B1), storyboard feature b3 (corresponding to the candidate audio and video B1), storyboard feature b4 (corresponding to the candidate audio and video B2), and storyboard feature b5 (corresponding to the candidate audio and video B4). The storyboard features a1 and a2 are similar to the storyboard feature information a, and the candidate audio and video A1 and A2 can be used as the candidate audio and video range, and the candidate audio and video in the candidate audio and video range can be matched based on frame features and audio features.

[0048] Step 2: Match the frame feature information with the frame features corresponding to each candidate audio and video in the frame feature library, and match the audio feature information with the audio features corresponding to each candidate audio and video in the audio feature library.

[0049] The frame feature library and audio feature library in this step are explained below through a specific example: the frame feature library stores frame features, so this step matches the frame feature information with the frame features of each candidate audio and video in the above-mentioned candidate audio and video range in the frame feature library. Exemplarily, the frame feature library includes frame features x1 (corresponding to candidate audio and video A1), x2 (corresponding to candidate audio and video A2). Querying from the frame feature library that the frame feature x2 is the same as the frame feature information x2, it can be determined that the candidate audio and video A2 matches the audio and video to be tested in the dimension of frame features.

[0050] Similarly, the audio feature library stores audio features, so this step matches the audio feature information with the audio features of each candidate audio and video in the above-mentioned candidate audio and video range in the audio feature library. Exemplarily, the audio feature library includes audio features y1 (corresponding to candidate audio and video A1) and y2 (corresponding to candidate audio and video A2). If the audio feature information y2 is the same as the audio feature information y2, it can be determined that the candidate audio and video A2 matches the audio and video to be tested in the dimension of audio features.

[0051] Step 3: When the frame features of the candidate audio and video successfully match the frame feature information, and the audio features of the candidate audio and video successfully match the audio feature information, determine that the candidate audio and video is the target audio and video.

[0052] In this embodiment, the candidate audio and video in the candidate audio and video range can be determined as the target audio and video only when the candidate audio and video matches the audio and video to be tested in both the frame feature and the audio feature.

[0053] In this way, matching the audio and video to be tested based on the three types of feature information can split the audio and video granularity more finely, thereby improving the accuracy of audio and video matching; in addition, there is no coupling relationship between the three modalities corresponding to the three types of feature information, that is, a rough comparison of storyboard fragments can be achieved, and more accurate video frame matching and audio matching can be achieved, further improving the accuracy of audio and video matching.

[0054] In some embodiments, step S204 may further include: storing the storyboard feature information corresponding to the audio and video to be tested in a storyboard feature library, storing the frame feature information corresponding to the audio and video to be tested in a frame feature library, and storing the audio feature information corresponding to the audio and video to be tested in an audio feature library.

[0055] In this embodiment, by storing the storyboard feature information, frame feature information and audio feature information of the audio and video to be tested in the storyboard feature library, frame feature library and audio feature library respectively, the content volume in the storyboard feature library, frame feature library and audio feature library can be enriched. When there are other audio and video that need to be matched later, the probability and accuracy of matching the target audio and video are higher.

[0056] In some of the embodiments, a storyboard feature library is queried based on storyboard feature information to determine multiple candidate audio and video including: determining the audio and video corresponding to the storyboard features in the storyboard feature library whose similarity with the storyboard feature information is greater than a first threshold as candidate audio and video.

[0057] Specifically, this embodiment determines whether the storyboard feature information matches the storyboard feature by the size of the similarity. A large similarity indicates a high matching degree, and a small similarity indicates a low matching degree. The size of the similarity is compared with a preset first threshold. If the similarity is greater than the first threshold, the audio and video corresponding to the storyboard feature is added as a candidate audio and video to the candidate audio and video range.

[0058] Specifically, the Euclidean distance (L2) vector comparison algorithm can be used to calculate the similarity between the storyboard feature information and each storyboard feature in the storyboard feature library, and the topk parameter can be used to retrieve several most similar storyboard features at a time, and the specified vec_distance function is used to calculate the distance between the two vectors in the verification result. The verification result is used to evaluate the similarity (or degree of difference) between the predicted vector and the true vector.

[0059] Among them, Euclidean distance (L2) is a way to measure the distance between two points. It is also one of the most commonly used distance calculation methods and is widely used in the field of computer vision. The algorithm calculates the shortest straight-line distance between two points. If the Euclidean distance between two sample points is small, it means that the two sample points are relatively close and may belong to the same category. The Euclidean distance is shown below:

[0060] Among them, a=(a1, a2, …, an) and b=(b1, b2, …, bn) are two points in n-dimensional Euclidean space.

[0061] In addition, the vector comparison algorithm of Euclidean distance (L2) can also be applied to the matching of frame features and audio features. Its specific application method can refer to the description of the above application to storyboard feature matching, and this application will not provide a cumbersome explanation here.

[0062] It can be understood that the first threshold is a value preset by prior knowledge, and the present application does not provide a cumbersome explanation of the value of the first threshold.

[0063] The traditional audio and video matching method mainly extracts the video frame of the audio and video to be tested as a feature, and compares it frame by frame with the target audio and video to achieve audio and video matching. However, the matching efficiency and accuracy of this method are relatively low. In the embodiment recorded in the above steps S201 to S204, when matching audio and video, it is possible to query in the audio and video fingerprint feature library whether there are various features that match the above feature information based on the storyboard feature information, frame feature information and audio feature information, and output the candidate audio and video corresponding to the various features that are successfully matched as the target audio and video, so that the audio and video matching can be achieved without frame by frame comparison, thereby improving the efficiency of audio and video matching; at the same time, matching the audio and video to be tested based on the three feature information can split the audio and video granularity more finely, and improve the accuracy of audio and video matching. Therefore, this embodiment solves the problem of low efficiency and accuracy of audio and video matching in the prior art, and achieves the technical effect of improving the efficiency and accuracy of audio and video matching.

[0064] In some embodiments, the splitting process in step S202 includes: splitting the video to be tested based on the shot boundary time points in the video to be tested to obtain storyboard segments, wherein the shot boundary time points are the segmentation time points of the video shots of the video to be tested.

[0065] Specifically, by performing shot boundary detection on the video to be tested, the shot boundary time point of the video to be tested is determined. The shot boundary time point is the segmentation time point, that is, the video frame with a timestamp before the segmentation time point is divided into the video segment (or called storyboard segment) of the previous shot, and the video frame with a timestamp after the segmentation time point is divided into the video segment of the next shot. The video content in each video segment has high similarity, and the content of the video segment can be comprehensively understood by browsing the key frame image, thereby avoiding repeated processing of a large number of similar or identical video images, simplifying the complexity of video content analysis, and improving video processing efficiency.

[0066] In some of the embodiments, based on the shot boundary time points in the video to be tested, the video to be tested is split to obtain the storyboard segments, including: obtaining the pixel difference between two adjacent video frames in the video to be tested, wherein the pixel difference includes a grayscale value; if the pixel difference is greater than a second threshold, taking any time point between the timestamps of the two video frames as the shot boundary time point in the video to be tested; and splitting the video to be tested based on the shot boundary time points to obtain the storyboard segments.

[0067] In addition, the pixel difference also includes color value, brightness value, etc., which is not limited in this application.

[0068] The following is an example of comparing the difference in grayscale values. The grayscale difference of pixels at the same coordinates in two adjacent image frames is shown in formula (1). In formula (1), f n(i, j), f n+1 (i, j) represents the grayscale value of the pixel (i, j) in the nth frame and the n+1th frame respectively. The total frame difference between two adjacent image frames is shown in formula (2). In formula (2), M and N are the sizes of the image. If the total frame difference F d (f n , f n+1 ) is greater than the first threshold, it is determined that the lens has changed.

[0069]

[0070] By comparing the grayscale values ​​of pixels, different areas or targets in the video can be accurately identified. Specifically, different objects or scenes usually show different grayscale features in grayscale images, so the video can be divided into different parts based on these features. This segmentation method based on pixel grayscale values ​​can not only accurately capture the slight changes between video frames and ensure the accuracy of the segmentation, but also does not require content recognition of the video frames of the entire video, simplifying the early video segmentation process. Therefore, while simplifying the operation, it improves the efficiency of segmentation.

[0071] In some of the embodiments, a sample of a video frame may be obtained by sampling each storyboard segment to reduce the number of samples. Specifically, extracting the storyboard feature information of the storyboard segment includes: extracting a video frame from the storyboard segment according to a number of samples that is positively correlated with the duration of the storyboard segment; and performing content recognition on the video frame to obtain the storyboard feature information of the storyboard segment.

[0072] In this embodiment, the frame extraction process reduces the total data volume of the video by reducing the number of frames in the video, thereby saving storage space. This is particularly important for scenes that need to store a large amount of video data, such as surveillance video storage, online video platforms, and the like. In addition, during video transmission, reducing the number of frames can reduce the amount of data required for transmission, thereby speeding up the transmission speed and reducing the transmission time. This is particularly beneficial for real-time video transmission or bandwidth-constrained environments such as mobile networks. The number of samples is positively correlated with the duration of the storyboard segment, which means that the longer the storyboard segment, the more samples are collected. This is to ensure that the number of samples extracted is reasonable, and to avoid the loss of key information in the storyboard segment due to too few samples, resulting in a low subsequent recognition rate. Therefore, through this embodiment, while ensuring the content integrity of the storyboard segment, the resources consumed by calculation, transmission, and storage are reduced by extracting frames.

[0073] In some of the embodiments, content recognition of the extracted video frames is achieved by training a multimodal training model of text-graphic semantics. Specifically, performing content recognition on the video frames to obtain the storyboard feature information of the storyboard segments includes: performing content recognition on the video frames using a preset multimodal training model of text-graphic semantics to obtain a representation vector of the video frames; clustering the representation vectors of each video frame to obtain a plurality of clusters, wherein the center vector in each cluster is determined as the storyboard feature information of the storyboard segment.

[0074] Specifically, a CNN neural network, such as CLIP4Clip, can be used to extract the representation vector of the video frame. Compared with conventional neural networks such as ResNet50, CLIP4Clip generates text and video embedding by jointly training the video encoder and the text encoder, that is, the query information input by the user can be embedded as text and input into the text encoder, and the storyboard clip can be embedded as video and input into the video encoder. The encoder recognizes the query information to obtain the text feature vector, and performs semantic understanding on the storyboard clip recognition to obtain the central vector of the video shot description.

[0075] In some embodiments, content recognition of video frames is achieved by training a residual network model. Specifically, extracting frame feature information of video frames includes: using a preset residual network model to perform content recognition on the video frames to obtain a representation vector of the video frames, and using the representation vector of the video frames as the frame feature information of the video frames.

[0076] Specifically, a CNN neural network, such as ResNet V2, can be used to perform content recognition on video frames and generate representation vectors corresponding to the video frames.

[0077] In some of the embodiments, the content recognition of the audio is realized by training the audio fingerprint neural network model. Specifically, extracting the audio feature information of the audio to be tested includes: using a preset audio fingerprint neural network model to perform content recognition on the audio to be tested to obtain multiple representation vectors of the audio to be tested; clustering the representation vectors of the audio to be tested to obtain multiple clusters, wherein the center vector in each cluster is determined as the audio feature information of the audio to be tested.

[0078] Specifically, a CNN neural network, such as NNFP (Neural Network FingerPrinter), can be used to convert the audio to be tested into a dense representation vector that can be used to represent the semantics of the audio segment, wherein each representation vector represents an audio segment with a fixed length of approximately 1 second.

[0079] The audio and video matching method provided in the embodiment of the present application is described and illustrated below through specific examples. Figure 3A system architecture diagram of an audio and video matching method provided in an embodiment of the present application is shown. Figure 3 As shown, the application system of the method includes three modules: a feature extraction client 310, a data layer 320 and a matching server 330.

[0080] The feature extraction client 310 is mainly responsible for decoding and integrity checking of the audio and video to be tested, performing shot analysis, cutting and frame extraction processing on the decoded video to be tested, performing audio analysis on the decoded audio to be tested, and using a CNN deep neural network (including but not limited to a multimodal training model of text-image semantics, a residual network model and an audio fingerprint neural network model) to extract features of the video and audio to be tested, extracting the audio and video fingerprint features of the audio and video to be tested (also called fingerprint feature files, fingerprint files, including storyboard feature information, frame feature information and audio feature information), and packaging the extracted audio and video fingerprint features in a multimodal data format to send to the matching server 330. In addition, the feature extraction client 310 can also perform unified format compatibility processing on the audio and video to be tested, eliminating the impact of subsequent processing caused by differences in the bit rate, resolution, encoding format, etc. of the converged media audio and video, and having higher compatibility with the converged media audio and video.

[0081] The data layer 320 mainly adopts the object file storage method, creates a vector database through MySQL, and stores the storyboard feature information in it.

[0082] The matching service end 330 includes obtaining fingerprint files through feature extraction clients, fingerprint file parsing, storyboard feature matching services, matching range delineation and modal result condition assembly, frame feature matching services, audio feature matching services, multimodal result verification, and matching result input and output. Specifically, the audio and video fingerprint features in the multimodal data format can be added to the feature library of the corresponding modality (for example, the storyboard feature information is stored in the storyboard feature library, the frame feature information is stored in the frame feature library, and the audio feature information is stored in the audio feature library); in addition, when the matching service end 330 matches the audio and video to be tested, it first matches the storyboard feature information of the audio and video to be tested, obtains the candidate audio and video range, and then uses the candidate audio and video range as a condition to search in the frame feature library and the audio feature library. After the multimodal result verification, the matched target audio and video is written into the database and output.

[0083] Figure 4 A schematic diagram of an audio and video matching method provided in an embodiment of the present application is shown, such as Figure 4As shown, the method first decompresses the fingerprint file of the audio and video to be tested, obtains the storyboard feature information, frame feature information and audio feature information; based on the obtained storyboard feature information, searches in the preset storyboard feature library to determine whether there is a candidate audio and video in the storyboard feature library; if so, matches the frame feature information with the candidate frame feature information corresponding to each candidate audio and video in the frame feature library; if they match, matches the audio and video feature information with the candidate audio feature information corresponding to each candidate audio and video in the audio feature library; if they match, outputs the candidate audio and video in the frame feature library and the audio feature library that matches both the frame feature information and the audio feature information as the target audio and video.

[0084] In summary, the audio and video matching method provided in the embodiment of the present application can query whether there are various features matching the above feature information in the audio and video fingerprint feature library based on the storyboard feature information, frame feature information and audio feature information during audio and video matching, and output the candidate audio and video corresponding to the various features that are successfully matched as the target audio and video, so that the audio and video matching can be achieved without frame-by-frame comparison, thereby improving the efficiency of audio and video matching; at the same time, matching the audio and video to be tested based on the three kinds of feature information can split the audio and video granularity more finely, and improve the accuracy of audio and video matching. Therefore, this embodiment solves the problem of low efficiency and accuracy of audio and video matching in the prior art, and achieves the technical effect of improving the efficiency and accuracy of audio and video matching.

[0085] The present application also provides an audio and video matching device in the embodiments, which is used to implement the above embodiments and preferred implementation modes, and will not be repeated hereafter. The terms "module", "unit", "subunit", etc. used below may be a combination of software and / or hardware that implements a predetermined function. Although the devices described in the following embodiments are preferably implemented in software, the implementation of hardware, or a combination of software and hardware, is also possible and conceivable.

[0086] Figure 5 A structural block diagram of an audio and video matching device provided in an embodiment of the present application is shown. Figure 5 As shown, the device comprises: The decoding module 51 is used to decode the audio and video to be tested to obtain the audio and video to be tested corresponding to the audio and video to be tested.

[0087] The processing module 52 is used to perform splitting and frame extraction processing on the video to be tested to obtain storyboard segments and video frames corresponding to the video to be tested, wherein the storyboard segment is a video segment under the same scene lens in the video to be tested.

[0088] The extraction module 53 is used to extract the shot feature information of the shot segments, extract the frame feature information of the video frames, and extract the audio feature information of the audio to be tested.

[0089] The query module 54 is used to query the pre-built audio and video fingerprint feature library according to the storyboard feature information, frame feature information and audio feature information to determine the target audio and video that matches the audio and video to be tested.

[0090] In some of the embodiments, the pre-constructed audio and video fingerprint feature library includes a storyboard feature library, a frame feature library and an audio feature library; the query module 54 is also used to query the storyboard feature library based on the storyboard feature information to determine multiple candidate audio and video; match the frame feature information with the frame feature corresponding to each candidate audio and video in the frame feature library, and match the audio feature information with the audio feature corresponding to each candidate audio and video in the audio feature library; when the frame feature of the candidate audio and video successfully matches the frame feature information, and the audio feature of the candidate audio and video successfully matches the audio feature information, determine that the candidate audio and video is the target audio and video.

[0091] In some of the embodiments, the audio and video matching device also includes a storage module, which is used to store the storyboard feature information corresponding to the audio and video to be tested into a storyboard feature library, store the frame feature information corresponding to the audio and video to be tested into a frame feature library, and store the audio feature information corresponding to the audio and video to be tested into an audio feature library.

[0092] In some of the embodiments, the query module 54 is further used to determine the audio and video corresponding to the storyboard features in the storyboard feature library whose similarity with the storyboard feature information is greater than a first threshold as candidate audio and video.

[0093] In some embodiments, splitting the video to be tested includes: splitting the video to be tested based on the shot boundary time points in the video to be tested to obtain storyboard segments, wherein the shot boundary time points are the segmentation time points of the video shots of the video to be tested; performing frame extraction processing on the video to be tested includes: extracting video frames from the video to be tested according to a sampling number that is positively correlated with the duration of the video to be tested; or extracting video frames from the video to be tested according to a preset sampling number.

[0094] In some of the embodiments, the processing module 52 is further used to obtain a pixel difference between two adjacent video frames in the video to be tested, wherein the pixel difference includes a grayscale value; if the pixel difference is greater than a first threshold, any time point between the timestamps of the two video frames is used as a shot boundary time point in the video to be tested; and the video to be tested is split based on the shot boundary time points to obtain storyboard segments.

[0095] In some of the embodiments, the extraction module 53 is further used to extract video frames from the storyboard segments according to a sampling quantity that is positively correlated with the duration of the storyboard segments; and perform content recognition on the video frames to obtain storyboard feature information of the storyboard segments.

[0096] In some of the embodiments, the extraction module 53 is further used to perform content recognition on the video frames using a preset graphic and text semantic multimodal training model to obtain representation vectors of the video frames; cluster the representation vectors of the respective video frames to obtain a plurality of clusters, wherein the center vector in each cluster is determined as the storyboard feature information of the storyboard fragment.

[0097] In some of the embodiments, the extraction module 53 is further used to perform content recognition on the video frame using a preset residual network model to obtain a representation vector of the video frame, and use the representation vector of the video frame as frame feature information of the video frame.

[0098] In some of the embodiments, the extraction module 53 is also used to use a preset audio fingerprint neural network model to perform content recognition on the audio to be tested to obtain multiple characterization vectors of the audio to be tested; cluster the various characterization vectors of the audio to be tested to obtain multiple clusters, wherein the center vector in each cluster is determined as the audio feature information of the audio to be tested.

[0099] It should be noted that the above modules can be functional modules or program modules, and can be implemented by software or hardware. For modules implemented by hardware, the above modules can be located in the same processor; or the above modules can be located in different processors in any combination.

[0100] In this embodiment, an electronic device is further provided, including a memory and a processor, wherein a computer program is stored in the memory, and the processor is configured to run the computer program to execute the steps in any one of the above method embodiments.

[0101] Optionally, the electronic device may further include a transmission device and an input / output device, wherein the transmission device is connected to the processor, and the input / output device is connected to the processor.

[0102] Optionally, in this embodiment, the processor may be configured to execute the following steps S1 to S4 through a computer program.

[0103] S1. Decode the audio and video to be tested to obtain the audio and video to be tested corresponding to the audio and video to be tested.

[0104] S2. Split and extract frames of the video to be tested to obtain storyboard segments and video frames corresponding to the video to be tested, wherein the storyboard segment is a segment of video captured under the same scene in the video to be tested.

[0105] S3, extracting the storyboard feature information of the storyboard segments, extracting the frame feature information of the video frames, and extracting the audio feature information of the audio to be tested.

[0106] S4. According to the storyboard feature information, frame feature information and audio feature information, a pre-built audio and video fingerprint feature library is queried to determine the target audio and video that matches the audio and video to be tested.

[0107] It should be noted that the specific examples in this embodiment can refer to the examples described in the above embodiments and optional implementation modes, and will not be repeated in this embodiment.

[0108] In addition, in combination with the method provided in the above embodiment, a storage medium may be provided in this embodiment to implement the method. The storage medium stores a computer program; when the computer program is executed by a processor, any audio and video matching method in the above embodiment is implemented.

[0109] The embodiment of the present application also provides a computer program product. When the computer program product is run on a computer, the computer executes each function or step executed by the processor in the above method embodiment.

[0110] It should be understood that the specific embodiments described herein are only used to explain the application, rather than to limit it. Based on the embodiments provided in this application, all other embodiments obtained by ordinary technicians in this field without creative work are within the protection scope of this application.

[0111] Obviously, the drawings are only some examples or embodiments of the present application. For ordinary technicians in the field, the present application can also be applied to other similar situations based on these drawings without creative work. In addition, it is understandable that although the work done in this development process may be complicated and lengthy, for ordinary technicians in the field, certain changes in design, manufacturing or production based on the technical content disclosed in this application are only conventional technical means and should not be regarded as insufficient content disclosed in this application.

[0112] The term "embodiment" in this application refers to a specific feature, structure or characteristic described in conjunction with the embodiment that can be included in at least one embodiment of the present application. The appearance of this phrase in various locations in the specification does not necessarily mean the same embodiment, nor does it mean that it is mutually exclusive with other embodiments and is independent or optional. It is clearly or implicitly understood by those of ordinary skill in the art that the embodiments described in this application can be combined with other embodiments without conflict.

[0113] The above-mentioned embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of patent protection. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the scope of protection of the present application. Therefore, the scope of protection of the present application shall be subject to the attached claims.

Claims

1. An audio and video matching method, characterized in that: The method comprises: Decoding the audio and video to be tested to obtain the audio and video to be tested corresponding to the audio and video to be tested; Performing splitting and frame extraction processing on the video to be tested to obtain storyboard segments and video frames corresponding to the video to be tested, wherein the storyboard segments are a segment of video under the same scene lens in the video to be tested; Extracting the storyboard feature information of the storyboard segment, extracting the frame feature information of the video frame, and extracting the audio feature information of the audio to be tested; According to the storyboard feature information, the frame feature information and the audio feature information, a pre-constructed audio and video fingerprint feature library is queried to determine a target audio and video that matches the audio and video to be tested.

2. The audio and video matching method according to claim 1, characterized in that: The pre-built audio and video fingerprint feature library includes a storyboard feature library, a frame feature library and an audio feature library; The step of querying a pre-built audio and video fingerprint feature library according to the storyboard feature information, the frame feature information, and the audio feature information to determine a target audio and video that matches the audio and video to be tested includes: According to the storyboard feature information, query the storyboard feature library to determine a plurality of candidate audio and video; Matching the frame feature information with the frame feature corresponding to each of the candidate audio and video in the frame feature library, and matching the audio feature information with the audio feature corresponding to each of the candidate audio and video in the audio feature library; When the frame feature of the candidate audio and video successfully matches the frame feature information, and the audio feature of the candidate audio and video successfully matches the audio feature information, the candidate audio and video is determined to be the target audio and video.

3. The audio and video matching method according to claim 2, characterized in that: After determining that the candidate audio and video is the target audio and video, the method further includes: The storyboard feature information corresponding to the audio and video to be tested is stored in the storyboard feature library, the frame feature information corresponding to the audio and video to be tested is stored in the frame feature library, and the audio feature information corresponding to the audio and video to be tested is stored in the audio feature library.

4. The audio and video matching method according to claim 2, characterized in that: The step of querying the storyboard feature library according to the storyboard feature information to determine a plurality of candidate audio and video includes: The audio and video corresponding to the storyboard feature in the storyboard feature library whose similarity with the storyboard feature information is greater than a first threshold are determined as candidate audio and video.

5. The audio and video matching method according to any one of claims 1 to 4, characterized in that: The splitting process of the video to be tested includes: Based on the shot boundary time points in the video to be tested, the video to be tested is split to obtain storyboard segments, wherein the shot boundary time points are the segmentation time points of the video shots of the video to be tested; The performing the frame extraction process on the video to be tested comprises: Extracting video frames from the video to be tested according to a sampling number that is positively correlated with the duration of the video to be tested; or extracting video frames from the video to be tested according to a preset sampling number.

6. The audio and video matching method according to claim 5, characterized in that: The step of splitting the video to be tested based on the shot boundary time points in the video to be tested to obtain shot segments includes: Obtaining a pixel difference between two adjacent video frames in the video to be tested, wherein the pixel difference includes a grayscale value; If the pixel difference is greater than a second threshold, any time point between the timestamps of the two video frames is used as a shot boundary time point in the video to be tested; Based on the shot boundary time points, the video to be tested is split to obtain storyboard segments.

7. The audio and video matching method according to any one of claims 1 to 4, characterized in that: The extracting the storyboard feature information of the storyboard segment comprises: Extracting video frames from the storyboard segment according to a sampling quantity that is positively correlated with the duration of the storyboard segment; Content recognition is performed on the video frame to obtain storyboard feature information of the storyboard segment.

8. The audio and video matching method according to claim 7, characterized in that: The performing content recognition on the video frame to obtain the storyboard feature information of the storyboard segment includes: Using a preset multimodal training model of image and text semantics to perform content recognition on the video frame to obtain a representation vector of the video frame; The representation vectors of each of the video frames are clustered to obtain a plurality of clusters, wherein a central vector in each of the clusters is determined as the storyboard feature information of the storyboard segment.

9. The audio and video matching method according to any one of claims 1 to 4, characterized in that: The extracting frame feature information of the video frame includes: A preset residual network model is used to perform content recognition on the video frame to obtain a representation vector of the video frame, and the representation vector of the video frame is used as frame feature information of the video frame.

10. The audio and video matching method according to any one of claims 1 to 4, characterized in that: The extracting audio feature information of the audio to be tested includes: Using a preset audio fingerprint neural network model to perform content recognition on the audio to be tested, and obtain multiple representation vectors of the audio to be tested; Clustering the characterization vectors of the audio to be tested to obtain a plurality of clusters, wherein a central vector in each cluster is determined as the audio feature information of the audio to be tested.

11. An audio and video matching device, characterized in that: include: A decoding module, used to decode the audio and video to be tested, and obtain the audio and video to be tested corresponding to the audio and video to be tested; A processing module, used for performing splitting and frame extraction processing on the video to be tested, to obtain storyboard segments and video frames corresponding to the video to be tested, wherein the storyboard segments are a segment of video under the same scene lens in the video to be tested; An extraction module, used to extract the shot feature information of the shot segment, extract the frame feature information of the video frame, and extract the audio feature information of the audio to be tested; The query module is used to query a pre-built audio and video fingerprint feature library according to the storyboard feature information, the frame feature information and the audio feature information to determine the target audio and video that matches the audio and video to be tested.

12. A terminal comprising a memory and a processor, characterized in that: A computer program is stored in the memory, and the processor is configured to run the computer program to execute the audio and video matching method according to any one of claims 1 to 10.

13. A storage medium, characterized in that: The storage medium stores a computer program, wherein the computer program is configured to execute the audio and video matching method according to any one of claims 1 to 10 when running.

Images