A feature-based audio and video retrieval method

Abstract

The present application relates to the field of audio and video retrieval technology, and provides a feature-based audio and video retrieval method. By pre-establishing an image material library and a pre-established audio material library, the samples to be detected are indexed. For example, accurate monitoring and statistics can be performed on the broadcast of radio and television program advertisements. Radio and television broadcast advertisements have the characteristic of invariance. Based on this method, the features of each TV channel are extracted to construct a material feature library, and then based on the advertisement sample feature retrieval, the advertisement broadcast situation is counted. Based on the millisecond-level characteristics of the features, advertisements can be accurately located and viewed. Through the constructed material feature library, different samples to be detected can be retrieved repeatedly. The retrieval results are output based on the judgment of time continuity. The feature ID is generated according to the program ID and feature time, and the retrieved media resources can be quickly grouped. It can be accurately located to the millisecond level, accurately matching the position where the material appears in the program, and the material can be repeatedly traced and retrieved.

Description

Technical Field

[0001] The present application relates to the field of audio and video retrieval technology, and in particular to a specific feature-based audio and video retrieval method. Background Art

[0002] With the development of modern computers and Internet technology, the types of multimedia data are becoming more and more diverse and the scale is getting larger and larger, which makes multimedia retrieval technology more and more important. For example, advertisers need to monitor whether the advertisements are broadcast as agreed. For example, film and television producers need to confirm the playback status of their works.

[0003] Currently, existing audio and video retrieval solutions are often based on single-dimensional retrieval. For example, voice is first converted to text, and then searched based on the text. This extracts only audio features, without associating them with image features. Furthermore, this recognition technology, which converts voice to text first, consumes a large amount of computing power and cannot accurately match with millisecond accuracy. Consequently, search results are unsatisfactory in noisy backgrounds. Furthermore, pure image retrieval, due to the large number of different audio and video sources using the same or similar images, results obtained using image retrieval methods are subject to significant errors, hindering efficient and accurate audio and video retrieval. Summary of the Invention

[0004] In order to provide an efficient and accurate audio and video retrieval method, an embodiment of the present application provides a feature-based audio and video retrieval method.

[0005] The first aspect of the present application provides a feature-based audio and video retrieval method, comprising:

[0006] Get the image feature points of the sample to be detected and the audio feature points of the sample to be detected;

[0007] A pre-established image material library is screened using the image feature points of the sample to be detected. If the inner product distance between the material image feature points of the image material library and the inner product distance between the sample image feature points to be detected is within a preset interval, the corresponding material image feature points are extracted, and a first feature set is established with all the extracted material image feature points; and a pre-established audio material library is screened using the audio feature points of the sample to be detected. If the inner product distance between the material audio feature points of the audio material library and the inner product distance between the sample audio feature points to be detected is within a preset interval, the corresponding material audio feature points are extracted, and a second feature set is established with all the extracted material audio feature points;

[0008] The material image feature points of the first feature set are grouped according to program ID, and the material image feature points in the same ID group are sorted according to timestamps to obtain multiple sets of material image feature sequence sets; and the material audio feature points of the second feature set are grouped according to program ID, and the audio feature points in the same ID group are sorted according to timestamps to obtain multiple sets of material audio feature sequence sets;

[0009] Under the material image feature sequence set, temporal clustering is performed on the material image feature points in the material image feature sequences grouped by the same ID to obtain multiple similar image clusters; and under the material audio feature sequence set, temporal clustering is performed on the material audio feature points in the material audio feature sequences grouped by the same ID to obtain multiple similar audio clusters;

[0010] Performing continuity verification on the material image feature points in the same image cluster and the material audio feature points in the same audio cluster, respectively, determining the start and end times of the material image feature sequences in the same image cluster to obtain a set of continuous material image feature sequences; and determining the start and end times of the material audio feature sequences in the same audio cluster to obtain a set of continuous material audio feature sequences;

[0011] According to the program ID, a union or a subset of the continuous material image feature sequence set and the continuous material audio feature sequence set is obtained.

[0012] Optionally, the step of obtaining the image feature points of the sample to be detected and the audio feature points of the sample to be detected is specifically as follows:

[0013] Obtain the audio and video of the sample to be tested;

[0014] The extracted sample image sequence and sample audio sequence of the audio and video of the sample to be detected;

[0015] Performing feature extraction on the sample image sequence 4 to generate image feature points of the sample to be detected, wherein the image feature points of the sample to be detected include program ID information, time information and fingerprint information;

[0016] Feature extraction is performed on the sample audio sequence based on spectrum peaks to generate audio feature points of the sample to be detected. The audio feature points of the sample to be detected include program ID information, time information and voiceprint information.

[0017] Optionally, the step of temporally clustering the material image feature points in the material image feature sequences of the same ID group to obtain a plurality of similar image clusters is specifically as follows:

[0018] Get the Unix timestamp information of the feature points of the material image. If the timestamps between consecutive feature points of the material image satisfy the following formula, the consecutive feature points of the material image are classified into a similar image cluster.

[0019] t n2 -t n1 <10*IA

[0020] or t n2 -t n1 >=10*IA and|(t AFn2->SAF -t AFn1->SAF )-(t n2 -t n1 )|<3*IA;

[0021] Among them, t n2 and t n1 is the Unix timestamp information of two consecutive material image feature points, IA is the preset interval time for frame extraction; t AFn2->SAF and t AFn1->SAF It is the Unix timestamp information of the feature points of the sample image to be detected corresponding to two consecutive feature points of the material image.

[0022] Optionally, the video and audio retrieval method further includes:

[0023] If there is only one feature point of a material image in the same type of image cluster, then remove the same type of image cluster;

[0024] The duration of any similar image cluster is calculated, and if the duration is less than a preset duration, the similar image cluster is removed.

[0025] Optionally, the step of temporally clustering the material audio feature points in the material audio feature sequences grouped by the same ID to obtain a plurality of similar audio clusters is specifically as follows:

[0026] Get the Unix timestamp information of the audio feature points of the material. If the timestamps between consecutive audio feature points satisfy the following formula, the consecutive audio feature points of the material are classified into a similar audio cluster.

[0027] T n2 -T n1 <10*IP

[0028] or T n2 -T n1 >=10*IP and|(T AFn2->SAF -T AFn1->SAF )-(T n2 -T n1 )|<3*IP;

[0029] Among them, T n2 and T n1 is the Unix timestamp information of two consecutive audio feature points of the material, IP is the preset interval time for frame extraction; T AFn2->SAF and T AFn1->SAF It is the Unix timestamp information of the sample audio feature points to be detected corresponding to two consecutive material audio feature points.

[0030] Optionally, the video and audio retrieval method further includes:

[0031] If there is only one material audio feature point in the similar audio cluster, remove the similar audio cluster;

[0032] The duration of any audio cluster of the same type is calculated, and if the duration is less than a preset duration, the audio cluster of the same type is removed.

[0033] Optionally, the pre-established image material library and the pre-established audio material library are established by the following method:

[0034] Acquire audio and video materials, and slice the audio and video materials according to a preset slicing duration;

[0035] Decoding the sliced audio and video materials to generate a material image sequence and a material audio sequence, wherein sequence stamps of the material image sequence and the material audio sequence are generated according to the decoding time and the natural time of the video recording;

[0036] Extract features from the material image sequence frame by frame to generate material image feature points; extract features from the material audio sequence based on spectrum peak features to generate material audio feature points;

[0037] All material image feature points are independently databased to obtain an image material library, and all material audio feature points are independently databased to obtain an audio material library; when the material image feature points and material audio feature points are independently databased, independent IDs are set, and the independent IDs include program ID information and Unix timestamp information.

[0038] A second aspect of the present application provides a feature-based audio and video retrieval device, which is used to execute the audio and video retrieval method provided in the first aspect of the present application. The audio and video retrieval device includes:

[0039] An acquisition device, used to acquire image feature points of a sample to be detected, and used to acquire audio feature points of a sample to be detected;

[0040] a screening module for screening a pre-established image material library according to the feature points of the sample image to be detected, and extracting the corresponding feature points of the material image if the inner product distance between the feature points of the material image in the image material library and the feature points of the sample image to be detected is within a preset interval, and establishing a first feature set with all the extracted feature points of the material image;

[0041] and, for screening a pre-established audio material library according to the audio feature points of the sample to be detected, and extracting the corresponding audio feature points if the inner product distance between the audio feature points of the material library and the audio feature points of the sample to be detected is within a preset interval, and establishing a second feature set with all the extracted audio feature points;

[0042] a grouping module, configured to group the feature points of the material images of the first feature set according to the program ID, and sort the feature points of the material images in the same ID group according to the timestamp, to obtain multiple sets of material image feature sequence sets;

[0043] and, grouping the material audio feature points of the second feature set according to program IDs, and sorting the audio feature points in the same ID group according to timestamps, to obtain multiple sets of material audio feature sequence sets;

[0044] A clustering module is used to perform temporal clustering on the feature points of the material images in the feature sequences of the material images grouped by the same ID under the set of the material image feature sequences, so as to obtain multiple clusters of similar images;

[0045] and, for performing temporal clustering of material audio feature points in material audio feature sequences grouped by the same ID under the material audio feature sequence set, to obtain a plurality of similar audio clusters;

[0046] A sorting module is used to verify the continuity of the feature points of the material images in the same image cluster, determine the start and end time of the feature sequence of the material images in the same image cluster, and obtain multiple sets of continuous feature sequences of the material images;

[0047] Also, it is used to verify the continuity of the material audio feature points in the same audio cluster, determine the start and end time of the material audio feature sequence in the same audio cluster, and obtain multiple sets of continuous material audio feature sequence sets;

[0048] The merging module is used to calculate a union or a subset of the continuous material image feature sequence set and the continuous material audio feature sequence set according to the program ID.

[0049] A third aspect of the present application provides a storage medium storing a computer program; when the computer program is executed, the steps of a feature-based audio and video retrieval method provided in the first aspect of the present application are implemented.

[0050] The fourth aspect of the present application provides a computer device, characterized in that it includes: a memory for storing a computer program; and a processor for implementing the steps of a feature-based audio and video retrieval method provided in the first aspect of the present application when executing the computer program.

[0051] This application provides a feature-based audio and video retrieval method that pre-establishes an image and audio library to index samples to be tested. For example, this method can accurately monitor and count the broadcast of radio and television program advertisements. Radio and television advertisements are characterized by invariance. Based on this method, features of each television channel are extracted to construct a material feature library (image and audio). Then, based on the features of the advertisement samples, the broadcast status of the advertisements is counted. Based on the millisecond-level characteristics of the features, the advertisements can be accurately located and viewed. For another example, for program broadcast monitoring, features of the channels required for broadcast can be extracted to construct a material feature library. The extracted features of the broadcasted programs can then be searched in the material library to verify whether the programs are broadcasting normally. This application provides a feature-based audio and video retrieval method that, through the constructed material feature library, can repeatedly search for different samples to be tested. The search results are output based on temporal continuity. Feature IDs are generated based on the program ID and feature time, allowing for rapid grouping of retrieved media assets. Accurately locating and matching the location of the material in the program with millisecond accuracy can be achieved, allowing for repeated retroactive retrieval of the material. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] In order to more clearly illustrate the technical solution of the present application, the following is a brief introduction to the drawings required for use in the embodiments. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0053] Figure 1 A flowchart of a feature-based audio and video retrieval method provided in an embodiment of the present application;

[0054] Figure 2 A schematic diagram of a sequence of sorted continuous audio features provided in an embodiment of the present application;

[0055] Figure 3 A flowchart of a method for establishing an image material library and an audio material library provided in an embodiment of the present application. DETAILED DESCRIPTION

[0056] In order to more clearly describe the technical solutions of the embodiments of the present application, the following agreements are made here: taking image features as an example, feature points: have three pieces of information (describing image fingerprint information, image time information and image program ID information); feature sequence: feature points have a front-and-back relationship, and such a set of feature points is called a sequence; sequence set: a sequence set composed of multiple feature sequences, and there is no absolute relationship between sequences.

[0057] In order to provide an efficient and accurate audio and video retrieval method, the embodiment of the present application provides a feature-based audio and video retrieval method. Figure 1 FIG. 1 is a flow chart of a feature-based audio and video retrieval method provided in an embodiment of the present application. In a first aspect, an embodiment of the present application provides a feature-based audio and video retrieval method, comprising:

[0058] Step S101: Acquire image feature points of a sample to be detected and audio feature points of a sample to be detected.

[0059] Specifically, first obtain the audio and video of the sample to be detected; then extract the sample image sequence and sample audio sequence of the sample to be detected. The sample image sequence and sample audio sequence generated in this step do not need to pay attention to natural time and define the resample duration. Finally, feature extraction is performed on the global information of the sample image sequence to generate image feature points of the sample to be detected, and the image feature points of the sample to be detected include program ID information, time information and fingerprint information; and feature extraction is performed on the sample audio sequence based on spectrum peaks to generate audio feature points of the sample to be detected, and the audio feature points of the sample to be detected include program ID information, time information and voiceprint information, wherein the fingerprint information is used to calculate the inner product distance between two image feature points, and the voiceprint information is used to calculate the inner product distance between two audio feature points.

[0060] In step S102, a pre-established image material library is screened using the image feature points of the sample to be detected. If the inner product distance between the material image feature points of the image material library and the sample image feature points to be detected is within a preset interval, the corresponding material image feature points are extracted, and a first feature set is established with all the extracted material image feature points; and, a pre-established audio material library is screened using the audio feature points of the sample to be detected. If the inner product distance between the material audio feature points of the audio material library and the sample audio feature points to be detected is within a preset interval, the corresponding material audio feature points are extracted, and a second feature set is established with all the extracted material audio feature points.

[0061] Among them, taking the feature points of the sample image to be detected as an example, in the pre-established image material library, the inner product distance between the material image feature points in the image material library and the sample image feature points to be detected is calculated, and after the inner product distance is normalized, it is determined whether the inner product distance is in a preset interval. The closer the normalized inner product distance is to 1, the more similar the two objects are. If the normalized inner product distance is equal to 1, it means that the two objects are completely consistent. In actual application, the preset interval can be set according to the accuracy requirements. In the embodiment of the present application, the preset interval is set to [0.8, 1.0]. It should be noted that the material image feature points in the first feature set are randomly combined together, and the material audio feature points in the second feature set are randomly combined together.

[0062] Step S103, grouping the material image feature points of the first feature set according to the program ID, sorting the material image feature points of the same ID group according to the timestamp, and obtaining multiple groups of material image feature sequence sets; and grouping the material audio feature points of the second feature set according to the program ID, sorting the audio feature points of the same ID group according to the timestamp, and obtaining multiple groups of material audio feature sequence sets.

[0063] The disordered feature points in the first feature set and the second feature set need to be integrated according to certain rules. In an embodiment of the present application, the material image feature points or audio feature points are first grouped by program ID. Under the same ID group, the material image feature points or audio feature points are sorted according to the timestamps of the material image feature points or audio feature points, thereby obtaining a set of orderly arranged material image feature sequences, multiple sets of material image feature sequence sets or multiple sets of material audio feature sequence sets.

[0064] Step S104: Under the material image feature sequence set, temporally cluster the material image feature points in the material image feature sequence of the same ID group to obtain multiple similar image clusters; and under the material audio feature sequence set, temporally cluster the material audio feature points in the material audio feature sequence of the same ID group to obtain multiple similar audio clusters.

[0065] The temporal clustering of the feature points of the material images is specifically as follows: obtaining the Unix timestamp information of the feature points of the material images, and if the timestamps between consecutive feature points of the material images satisfy the following formula, the consecutive feature points of the material images are classified into a similar image cluster.

[0066] t n2 -t n1 <10*IP

[0067] or t n2 -t n1 >=10*IP and|(tAFn2->SAF -t AFn1->SAF )-(t n2 -t n1 )|<3*IP.

[0068] Among them, t n2 and t n1 is the Unix timestamp information of two consecutive material image feature points, IP is the preset interval time for frame extraction; t AFn2->SAF and t AFn1->SAF It is the Unix timestamp information of the feature points of the sample image to be detected corresponding to two consecutive feature points of the material image.

[0069] Temporal clustering of the material audio feature points is specifically as follows: obtaining the Unix timestamp information of the material audio feature points, and if the timestamps between consecutive audio feature points satisfy the following formula, the consecutive material audio feature points are classified into a similar audio cluster.

[0070] T n2 -T n1 <10*IA

[0071] or T n2 -T n1 >=10*IA and|(T AFn2->SAF -T AFn1->SAF )-(T n2 -T n1 )|<3*IA.

[0072] Among them, T n2 and T n1 is the Unix timestamp information of two consecutive audio feature points of the material, IA is the preset interval time for frame extraction; T AFn2->SAF and T AFn1->SAF It is the Unix timestamp information of the sample audio feature points to be detected corresponding to two consecutive material audio feature points.

[0073] It should be noted that after obtaining multiple similar image clusters or multiple similar audio clusters, noise needs to be removed. For similar image clusters, if there is only one material image feature point in the similar image cluster, then the similar image cluster is removed. In addition, the duration of any similar image cluster is calculated. If the duration is less than the preset duration, then the similar image cluster is removed. For similar image clusters, the preset duration is one tenth of the image sequence of the audio and video to be detected.

[0074] For similar audio clusters, if there is only one material audio feature point in the similar audio cluster, then the similar audio cluster is removed, and the duration of any similar audio cluster is calculated. If the duration is less than the preset duration, then the similar audio cluster is removed. For similar audio clusters, the preset duration is one tenth of the audio sequence of the audio and video to be detected.

[0075] Step S105, respectively verifying the continuity of the material image feature points in the same image cluster and the material audio feature points in the same audio cluster, determining the start and end time of the material image feature sequence in the same image cluster, and obtaining a set of continuous material image feature sequences; and determining the start and end time of the material audio feature sequence in the same audio cluster, and obtaining a set of continuous material audio feature sequences.

[0076] The continuity of the feature points in the same image cluster or audio cluster is verified, and the start and end time of the matching material is determined. For example, in the embodiment of the present application, the start and end time of the audio features of the material in the same audio cluster are arranged in order from small to large, thereby obtaining a continuous material image feature sequence set. Figure 2 The following is a schematic diagram of a sorted continuous material audio feature sequence provided by an embodiment of the present application. For a homogeneous audio cluster, tn1 to tnn are n material audio feature points, AFn1 to Afnn are the timestamp information corresponding to the n material audio feature points, and SAF1 to SAFn are the timestamp information of the audio feature points of n samples to be detected. The material audio feature points are sorted according to the timestamp information of the audio feature points of the samples to be detected to obtain a continuous material audio feature sequence. Multiple continuous material audio feature sequences are combined into a continuous material audio feature sequence set.

[0077] Step S106 : according to the program ID, a union or a subset of the continuous material image feature sequence set and the continuous material audio feature sequence set is obtained.

[0078] like Figure 3 As shown in FIG, a flow chart of a method for establishing an image material library and an audio material library provided in an embodiment of the present application is provided. The image material library and the audio material library mentioned in the embodiment of the present application are pre-established by the following method:

[0079] Step S301: Acquire audio and video materials, and slice the audio and video materials according to a preset slicing duration.

[0080] In actual application, real-time streaming programs or long recorded programs need to be sliced. In the embodiment of the present application, the preset slicing time is 1 hour.

[0081] Step S302 : Decode the sliced audio and video material to generate a material image sequence and a material audio sequence. The sequence stamps of the material image sequence and the material audio sequence are generated according to the decoding time and the natural time of the video recording.

[0082] In real-world environments, audio and video may be missing. The image and audio sequence stamp times are generated by combining the decoding timestamp and the natural time of the video. For example, if the natural time of the slice start frame is 2021-3-25 00:00:00 (ts), the slice start is PTSs, and the sequence stamp of the nth frame is PTSn, then the timestamp information tn is: tn = ts + (PTSn - PTSs).

[0083] Step S303 , extracting features of the material image sequence at alternate frames to generate material image feature points, and extracting features of the material audio sequence based on spectrum peak features to generate material audio feature points.

[0084] For source image sequences, feature extraction is performed based on global image information. Due to image continuity, feature robustness, and extraction efficiency, alternate-frame feature extraction is performed. The PDQ feature extraction algorithm or the perceptual hashing and aberration feature extraction algorithms can be used. The time interval IP between each feature point is the preset interval for alternate-frame extraction of the source image feature sequence. Source audio sequences are based on spectral peak features, each of which occurs approximately every 90ms. The interval between audio feature points is IA, the preset interval for alternate-frame extraction of the source audio feature sequence.

[0085] In step S304, all the material image feature points are independently built into a library to obtain an image material library, and all the material audio feature points are independently built into a library to obtain an audio material library; when the material image feature points and the material audio feature points are independently built into a library, an independent ID is set, and the independent ID includes program ID information and Unix timestamp information.

[0086] The independent ID consists of 128 bits, of which the first 64 bits represent the program ID information and the last 64 bits are the Unix timestamp information converted from tn.

[0087] A second aspect of the present application further provides a feature-based audio and video retrieval device, which is configured to execute the audio and video retrieval method provided in the first aspect of the present application. The audio and video retrieval device includes:

[0088] An acquisition device, used to acquire image feature points of a sample to be detected, and used to acquire audio feature points of a sample to be detected;

[0089] The screening module is used to screen a pre-established image material library according to the feature points of the sample image to be detected. If the inner product distance between the feature points of the material image in the image material library and the feature points of the sample image to be detected is within a preset interval, the corresponding feature points of the material image are extracted, and a first feature set is established with all the extracted feature points of the material image;

[0090] and, for screening a pre-established audio material library according to the audio feature points of the sample to be detected, and extracting the corresponding audio feature points if the inner product distance between the audio feature points of the material library and the audio feature points of the sample to be detected is within a preset interval, and establishing a second feature set with all the extracted audio feature points;

[0091] a grouping module, used for grouping the material image feature points of the first feature set according to the program ID, and sorting the material image feature points of the same ID group according to the timestamp to obtain multiple sets of material image feature sequence sets;

[0092] and, grouping the material audio feature points of the second feature set according to program IDs, and sorting the audio feature points in the same ID group according to timestamps, to obtain multiple sets of material audio feature sequence sets;

[0093] A clustering module is used to perform temporal clustering on the feature points of the material images in the feature sequences of the material images grouped by the same ID under the set of the material image feature sequences, so as to obtain multiple clusters of similar images;

[0094] and, for performing temporal clustering of material audio feature points in material audio feature sequences grouped by the same ID under the material audio feature sequence set, to obtain a plurality of similar audio clusters;

[0095] A sorting module is used to verify the continuity of the feature points of the material images in the same image cluster, determine the start and end time of the feature sequence of the material images in the same image cluster, and obtain multiple sets of continuous feature sequences of the material images;

[0096] Also, it is used to verify the continuity of the material audio feature points in the same audio cluster, determine the start and end time of the material audio feature sequence in the same audio cluster, and obtain multiple sets of continuous material audio feature sequence sets;

[0097] The merging module calculates a union or a subset of the continuous material image feature sequence set and the continuous material audio feature sequence set according to the program ID.

[0098] The third aspect of the embodiment of the present application further provides a storage medium, which stores a computer program; when the computer program is executed, it implements the steps of a feature-based audio and video retrieval method provided in the first aspect of the embodiment of the present application.

[0099] The third aspect of the embodiment of the present application also provides a computer device, which includes: a memory and a processor; the memory is used to store a computer program; the processor is used to implement the steps of a feature-based audio and video retrieval method provided in the first aspect of the embodiment of the present application when executing the computer program.

[0100] The present invention provides a feature-based audio and video retrieval method that pre-establishes an image and audio library to index samples to be tested. For example, this method can accurately monitor and count the broadcast of radio and television program advertisements. Radio and television advertisements are characterized by invariance. Based on this method, features of each television channel are extracted to construct a material feature library (image and audio). Then, based on the features of the advertisement samples, advertisement broadcast statistics are collected. Based on the millisecond-level characteristics of the features, advertisements can be accurately located and viewed. For another example, for program broadcast monitoring, features of the channels required for broadcast can be extracted to construct a material feature library. The extracted features of the broadcasted programs can then be searched in the material library to verify whether the programs are broadcasting normally. The feature-based audio and video retrieval method provided by this application can repeatedly search different samples to be tested using the constructed material feature library. The search results are output based on temporal continuity. Feature IDs are generated based on the program ID and feature time, allowing for rapid grouping of retrieved media assets. Accurately locating and matching the location of the material in the program with millisecond-level accuracy is possible, and the material can be repeatedly retrieved.

[0101] The present application has been described in detail above with reference to specific embodiments and exemplary examples. However, these descriptions should not be construed as limiting the present application. Those skilled in the art will appreciate that, without departing from the spirit and scope of the present application, various equivalent substitutions, modifications, or improvements may be made to the technical solutions and implementations of the present application, all of which fall within the scope of the present application. The scope of protection of the present application shall be determined by the appended claims.

Claims

1. A feature-based audio and video retrieval method, characterized in that: include: Obtaining the image feature points of the sample to be detected and the audio feature points of the sample to be detected; Using the feature points of the sample image to be detected to screen a pre-established image material library, if the inner product distance between the material image feature points of the image material library and the feature points of the sample image to be detected is within a preset interval, then extract the corresponding material image feature points, and establish a first feature set with all the extracted material image feature points; and, using the audio feature points of the sample to be detected to screen a pre-established audio material library, if the inner product distance between the material audio feature points of the audio material library and the audio feature points of the sample to be detected is within a preset interval, extracting the corresponding material audio feature points, and establishing a second feature set with all the extracted material audio feature points; The material image feature points of the first feature set are grouped according to program ID, and the material image feature points in the same ID group are sorted according to timestamps to obtain a set of material image feature sequences, and multiple sets of material image feature sequences are combined into a material image feature sequence set; and, grouping the material audio feature points of the second feature set according to program ID, sorting the audio feature points in the same ID group according to timestamps to obtain a set of material audio feature sequences, and combining the multiple sets of material audio feature sequences into a set of material audio feature sequences; Under the material image feature sequence set, temporal clustering is performed on the material image feature points in the material image feature sequences grouped by the same ID to obtain multiple similar image clusters; and under the material audio feature sequence set, temporal clustering is performed on the material audio feature points in the material audio feature sequences grouped by the same ID to obtain multiple similar audio clusters; Perform continuity verification on the material image feature points in the same image cluster and the material audio feature points in the same audio cluster, determine the start and end time of the material image feature sequence in the same image cluster, and obtain a set of continuous material image feature sequences; and, determining the start and end times of the material audio feature sequences in the same audio cluster to obtain a set of continuous material audio feature sequences; According to the program ID, a union of the continuous material image feature sequence set and the continuous material audio feature sequence set is obtained.

2. A feature-based audio and video retrieval method according to claim 1, characterized in that: The steps of obtaining the image feature points of the sample to be detected and the audio feature points of the sample to be detected are specifically as follows: Obtain the audio and video of the sample to be tested; The extracted sample image sequence and sample audio sequence of the audio and video of the sample to be detected; Extracting features from the sample image sequence to generate image feature points of the sample to be detected, wherein the image feature points of the sample to be detected include program ID information, time information, and fingerprint information; Feature extraction is performed on the sample audio sequence based on spectrum peaks to generate audio feature points of the sample to be detected. The audio feature points of the sample to be detected include program ID information, time information and voiceprint information.

3. The feature-based audio and video retrieval method according to claim 1, characterized in that: The step of temporally clustering the material image feature points in the material image feature sequences grouped by the same ID to obtain a plurality of similar image clusters is specifically as follows: Get the Unix timestamp information of the feature points of the material image. If the timestamps between consecutive feature points of the material image satisfy the following formula, the consecutive feature points of the material image are classified into a similar image cluster. ； or and ; in, and It is the Unix timestamp information of two consecutive feature points of the material image. The preset interval time for frame extraction; and It is the Unix timestamp information of the feature points of the sample image to be detected corresponding to two consecutive feature points of the material image.

4. A feature-based audio and video retrieval method according to claim 3, characterized in that: The video and audio retrieval method further includes: If there is only one feature point of a material image in the same type of image cluster, then remove the same type of image cluster; The duration of any similar image cluster is calculated, and if the duration is less than a preset duration, the similar image cluster is removed.

5. The feature-based audio and video retrieval method according to claim 1, characterized in that: The step of temporally clustering the material audio feature points in the material audio feature sequences grouped by the same ID to obtain a plurality of similar audio clusters is specifically as follows: Get the Unix timestamp information of the audio feature points of the material. If the timestamps between consecutive audio feature points satisfy the following formula, the consecutive audio feature points of the material are classified into a similar audio cluster. ； or and ; in, and The Unix timestamp information of two consecutive material audio feature points. The preset interval time for frame extraction; and It is the Unix timestamp information of the sample audio feature points to be detected corresponding to two consecutive material audio feature points.

6. A feature-based audio and video retrieval method according to claim 5, characterized in that: The video and audio retrieval method further includes: If there is only one material audio feature point in the similar audio cluster, remove the similar audio cluster; The duration of any audio cluster of the same type is calculated, and if the duration is less than a preset duration, the audio cluster of the same type is removed.

7. The feature-based audio and video retrieval method according to claim 1, characterized in that: The pre-established image material library and the pre-established audio material library are established by the following method: Acquire audio and video materials, and slice the audio and video materials according to a preset slicing duration; Decoding the sliced audio and video materials to generate a material image sequence and a material audio sequence, wherein sequence stamps of the material image sequence and the material audio sequence are generated according to the decoding time and the natural time of the video recording; Extract features from the material image sequence frame by frame to generate material image feature points; extract features from the material audio sequence based on spectrum peak features to generate material audio feature points; All material image feature points are independently databased to obtain an image material library, and all material audio feature points are independently databased to obtain an audio material library; when the material image feature points and material audio feature points are independently databased, independent IDs are set, and the independent IDs include program ID information and Unix timestamp information.

8. A feature-based audio and video retrieval device, characterized in that: The video and audio retrieval device is used to execute the video and audio retrieval method according to any one of claims 1 to 7, and the video and audio retrieval device includes: An acquisition device, used to acquire image feature points of a sample to be detected, and used to acquire audio feature points of a sample to be detected; a screening module for screening a pre-established image material library according to the feature points of the sample image to be detected, and extracting the corresponding feature points of the material image if the inner product distance between the feature points of the material image in the image material library and the feature points of the sample image to be detected is within a preset interval, and establishing a first feature set with all the extracted feature points of the material image; and, for screening a pre-established audio material library according to the audio feature points of the sample to be detected, and extracting the corresponding audio feature points if the inner product distance between the audio feature points of the material library and the audio feature points of the sample to be detected is within a preset interval, and establishing a second feature set with all the extracted audio feature points; a grouping module, configured to group the feature points of the material images of the first feature set according to the program ID, and sort the feature points of the material images in the same ID group according to the timestamp, to obtain multiple sets of material image feature sequence sets; and, grouping the material audio feature points of the second feature set according to program IDs, and sorting the audio feature points in the same ID group according to timestamps, to obtain multiple sets of material audio feature sequence sets; A clustering module is used to perform temporal clustering on the feature points of the material images in the feature sequences of the material images grouped by the same ID under the set of the material image feature sequences, so as to obtain multiple clusters of similar images; and, for performing temporal clustering of material audio feature points in material audio feature sequences grouped by the same ID under the material audio feature sequence set, to obtain a plurality of similar audio clusters; A sorting module is used to verify the continuity of the feature points of the material images in the same image cluster, determine the start and end time of the feature sequence of the material images in the same image cluster, and obtain multiple sets of continuous feature sequences of the material images; Also, it is used to verify the continuity of the material audio feature points in the same audio cluster, determine the start and end time of the material audio feature sequence in the same audio cluster, and obtain multiple sets of continuous material audio feature sequence sets; The merging module is used to obtain a union of a continuous material image feature sequence set and a continuous material audio feature sequence set according to a program ID.

9. A storage medium, characterized in that: The storage medium stores a computer program; when the computer program is executed, the steps of the feature-based audio-visual retrieval method according to any one of claims 1 to 7 are implemented.

10. A computer device, characterized in that: include: memory for storing computer programs; A processor is configured to implement the steps of a feature-based audio and video retrieval method as described in any one of claims 1 to 7 when executing the computer program.

Images