Multi-language audio and video content retrieval method based on voice recognition

Abstract

The invention discloses a multi-language audio and video content retrieval method based on voice recognition, and relates to the technical field of voice recognition, and the method comprises the following steps: calling a predetermined spokesman detection strategy to analyze multi-language audio and video content, and obtaining a target spokesman time sequence; extracting the spokesman and the speaking content thereof at any time point, analyzing the speaking audio through the matched semantic recognition channel, and generating a semantic feature vector; meanwhile, feature extraction is carried out on the speaking video, and a visual feature vector is generated; and matching and retrieving target contents in a multi-language audio and video database by combining the two types of features. Therefore, the technical effects of improving the retrieval efficiency and improving the retrieval accuracy are achieved.

Description

Technical Field

[0001] This invention relates to the field of speech recognition technology, and in particular to a method for retrieving multilingual audio and video content based on speech recognition. Background Technology

[0002] Existing audio and video retrieval methods mainly rely on manual annotation or feature extraction of a single modality, such as speech recognition based solely on audio or image recognition based solely on video. This results in incomplete retrieval results and limited ability to recognize multiple languages, making it difficult to adapt to content of different language types. In particular, when faced with mixed languages, the recognition accuracy drops significantly, resulting in technical problems of low retrieval efficiency and insufficient accuracy. Summary of the Invention

[0003] This invention provides a multilingual audio and video content retrieval method based on speech recognition to solve the technical problems of low retrieval efficiency and insufficient accuracy in the prior art, thereby achieving the technical effects of improving retrieval efficiency and accuracy.

[0004] The multilingual audio and video content retrieval method based on speech recognition provided by this invention includes: The predetermined speaker detection strategy is retrieved to detect and analyze the target multilingual audio and video content, and the target speaker time sequence is obtained.

[0005] Extract any speaker corresponding to any time in the target speaker time sequence, and obtain any speech content of the arbitrary speaker.

[0006] By identifying and analyzing any spoken audio in any spoken content through any semantic recognition channel corresponding to any matched language type, an arbitrary semantic feature vector is obtained.

[0007] Features are collected from any video of any spoken content to obtain an arbitrary visual feature vector.

[0008] The target retrieval content is obtained by traversing and matching the arbitrary semantic feature vector and the arbitrary visual feature vector in a multilingual audio and video content database.

[0009] In one feasible implementation, a predetermined speaker detection strategy is invoked to detect and analyze the target multilingual audio and video content, obtaining the target speaker time sequence, including: The target multilingual audio and video content is segmented to obtain the target segmentation result.

[0010] Extract the first audio and video segment from the target segmentation result, and obtain the first adjacent audio and video segment of the first audio and video segment.

[0011] Obtain the predetermined speech feature indicators stored in the predetermined speaker detection strategy.

[0012] The first speech feature parameters of the first audio and video segment are obtained based on the predetermined speech feature indicators, and the first speech feature value is generated.

[0013] Based on the predetermined speech feature indicators, the first adjacent speech feature parameters of the first adjacent audio and video segments are collected, and the first adjacent speech feature value is generated.

[0014] Calculate the first feature difference between the first speech feature value and the first adjacent speech feature value.

[0015] Determine whether the first feature difference meets the predetermined speech feature threshold, and obtain the determination result.

[0016] The target speaker's timeline is obtained based on the judgment result.

[0017] In one feasible implementation, the predetermined speech characteristic indicators include predetermined physical characteristics, predetermined acoustic characteristics, predetermined psychoacoustic characteristics, and predetermined emotional characteristics. The predetermined physical characteristics include at least frequency, amplitude, and timbre. The predetermined acoustic characteristics include at least fundamental frequency, formants, bandwidth, and harmonics. The predetermined psychoacoustic characteristics include at least pitch perception, loudness perception, and timbre perception. The predetermined emotional characteristics refer to the state type of the speaker's emotional expression.

[0018] In one feasible implementation, the first speech feature parameters of the first audio / video segment are acquired based on the predetermined speech feature indicators, and a first speech feature value is generated, including: The first speech feature parameters are normalized to obtain the first normalized speech feature parameters.

[0019] The first normalized speech feature parameter is subjected to a variation weighted calculation to obtain the first speech feature value.

[0020] In one feasible implementation, determining whether the first feature difference meets a predetermined speech feature threshold to obtain a determination result includes: if it does not meet the threshold, then the first audio / video segment and the first adjacent audio / video segment are from different speakers; if it does meet the threshold, then the first audio / video segment and the first adjacent audio / video segment are from the same speaker.

[0021] In one feasible implementation, features are collected from any video recording of any spoken content to obtain an arbitrary visual feature vector, including: The arbitrary speech video is compressed to obtain an arbitrarily compressed video.

[0022] Extract the first real frame from the arbitrary compressed video, and construct key compressed frames based on the first real frame.

[0023] Read the predetermined visual feature index, and collect features from the key compressed frame based on the predetermined visual feature index to obtain the arbitrary visual feature vector.

[0024] The process includes reading predetermined visual feature indicators and collecting features from the key compressed frame based on these indicators to obtain the arbitrary visual feature vector, including: The first visual feature parameters of the first real frame are obtained based on the predetermined visual feature index.

[0025] The second visual feature parameters of the second real frame are collected based on the predetermined visual feature index, and the second real frame is different from the first real frame.

[0026] The average value of the first visual feature parameter and the second visual feature parameter is taken as an arbitrary visual feature parameter.

[0027] The arbitrary visual feature vector is generated based on the arbitrary visual feature parameters.

[0028] In one feasible implementation, the predetermined visual feature indicators include predetermined frame features and predetermined scene features, wherein the predetermined frame features include frame tone and frame texture, and the predetermined scene features include at least emotional atmosphere, spatial relationship, brightness distribution, and optical flow.

[0029] In one feasible implementation, the arbitrary semantic feature vector and the arbitrary visual feature vector are traversed and matched in a multilingual audio and video content database to obtain the target retrieval content, including: An arbitrary feature vector is constructed based on the arbitrary semantic feature vector and the arbitrary visual feature vector.

[0030] Extract the first content from the multilingual audio and video content database, where the first content corresponds to the first feature vector.

[0031] Calculate the first similarity between the arbitrary feature vector and the first feature vector.

[0032] If the first similarity reaches a predetermined similarity threshold, the first content will be used as the target retrieval content.

[0033] This invention discloses a multilingual audio and video content retrieval method based on speech recognition, comprising: invoking a predetermined speaker detection strategy to detect and analyze target multilingual audio and video content to obtain the target speaker time sequence; extracting any speaker corresponding to any time point in the target speaker time sequence and obtaining their speech content; based on the matched language type, recognizing and analyzing the audio information of the speech content through the corresponding semantic recognition channel to generate a semantic feature vector; extracting features from the video information of the speech content to generate a visual feature vector; and traversing and matching the semantic feature vector and the visual feature vector in a multilingual audio and video content database to obtain the target retrieval content. The multilingual audio and video content retrieval method based on speech recognition disclosed in this invention solves the technical problems of low retrieval efficiency and insufficient accuracy, and achieves the technical effects of improving retrieval efficiency and retrieval accuracy. Attached Figure Description

[0034] Figure 1 This is a flowchart illustrating the multilingual audio and video content retrieval method based on speech recognition according to the present invention. Figure 2 This is a flowchart illustrating the multilingual audio and video content retrieval method based on speech recognition according to the present invention. Detailed Implementation

[0035] The above technical solutions will now be described in detail with reference to the accompanying drawings and specific embodiments to provide a better understanding of them. Obviously, the described embodiments are only a part of the embodiments of the present invention, and not all of them. It should be understood that the present invention is not limited to the exemplary embodiments used only to explain the present invention. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without creative effort are within the scope of protection of the present invention. Furthermore, it should be noted that, for ease of description, only the parts related to the present invention are shown in the drawings, not all of them.

[0036] Example, Figure 1 This is a flowchart illustrating the multilingual audio and video content retrieval method based on speech recognition according to the present invention. The method includes: S100: Retrieve the predetermined speaker detection strategy to detect and analyze the target multilingual audio and video content, and obtain the target speaker time sequence.

[0037] Specifically, target multilingual audio and video content refers to audio and video materials containing multiple languages ​​that users wish to search for, such as meeting minutes, multimedia teaching materials, and live broadcasts of international sporting events. Target speaker chronology refers to the information obtained through detection and analysis regarding the chronological order of each speaker's appearance and the segments of their speech in the audio and video, providing a precise positioning basis for subsequent content retrieval.

[0038] For example, a pre-defined speaker detection strategy is a set of rules and methods for identifying and distinguishing different speakers in multilingual audio and video content. It defines the process of detecting and analyzing target multilingual audio and video content, including detection steps, the types of languages ​​to be detected, and the detection order.

[0039] In some embodiments, a predetermined speaker detection strategy is invoked to detect and analyze the target multilingual audio and video content to obtain the target speaker time sequence, including: The target multilingual audio and video content is segmented to obtain a target segmentation result; a first audio and video segment is extracted from the target segmentation result, and a first adjacent audio and video segment is obtained; a predetermined speech feature index is obtained from the memory of the predetermined speaker detection strategy; a first speech feature parameter is collected based on the predetermined speech feature index, and a first speech feature value is generated; a first adjacent speech feature parameter is collected based on the predetermined speech feature index, and a first adjacent speech feature value is generated; a first feature difference is calculated between the first speech feature value and the first adjacent speech feature value; it is determined whether the first feature difference meets a predetermined speech feature threshold, and a judgment result is obtained; the target speaker time sequence is obtained based on the judgment result.

[0040] Specifically, firstly, the audio and video content is segmented to obtain a series of target segmentation results, namely continuous audio and video segments. This segmentation is achieved by feature recognition of the audio and video content, including acoustic feature-based segmentation (such as audio energy and speech start and end point detection) and visual feature-based segmentation (such as face detection and mouth movement trajectory analysis). Preferably, preliminary segmentation is performed through audio content to improve processing efficiency. Then, the first audio and video segment is extracted sequentially from the segmentation results, and its corresponding first adjacent audio and video segment is obtained. Through a predetermined speaker detection strategy, feature parameters are collected for the two segments respectively to obtain the first speaker feature value and the first adjacent speaker feature value.

[0041] For example, the pre-defined speaker detection strategy includes the feature indicators of the speaker to be extracted, such as voice features (timbre, intonation, speech rate, formants, pronunciation model), semantic features (speech style, verbal tics), and emotional features.

[0042] Further, calculate the first feature difference value between the first speech feature value and the first adjacent speech feature value to quantify the difference between the two feature values. For example, first, represent the first speech feature value and the first adjacent speech feature value in vector form, which involves the standardization of the feature value. Then, by calculating the vector difference, obtain the difference feature vector, and calculate the norm of the difference feature vector as the first feature difference value.

[0043] Exemplarily, select 5-dimensional feature parameters (such as pitch, speech rate, formant, lip movement change, semantic keyword frequency) to compare the first audio-visual segment (F1) and the first adjacent audio-visual segment (F2): Table 1 Exemplary first speech feature value and first adjacent speech feature value

[0044] Perform standardization processing to obtain the mean value of each dimension feature μ j and standard deviation σ j (assumed to be known): Table 2 Exemplary feature value mean and feature value standard deviation

[0045] Use the standardization formula to calculate the standardized F1 and F2: Table 3 Exemplary standardized values of the first speech feature value and the first adjacent speech feature value

[0046] Next, calculate the feature difference vector: Table 4 Exemplary component values of the difference vector

[0047] Further, according to the above-mentioned component of the difference vector, calculate the Euclidean norm:

[0048] Assume that the preset speaker feature difference threshold T = 3.0, and if D < T, it is considered that the two segments belong to the same speaker; if D ≥ T, it is considered that the two segments belong to different speakers. Then, since the above D = 3.82 is greater than 3.0, it can be considered that the first audio-visual segment and the first adjacent audio-visual segment belong to different speakers, and it is necessary to further analyze whether it is a speaker switching point.

[0049] Specifically, perform iterative extraction on the above-mentioned target segmentation result in the time sequence direction until all audio-visual segments have completed the above-mentioned discrimination based on the predetermined speech feature threshold, and then obtain the target speaker time sequence including the speech distribution of multiple speakers in the time direction.

[0050] The above process provides clear speakers and corresponding time periods for subsequent semantic and visual feature extraction, enabling more precise retrieval targeting specific speakers and their content. This avoids blindly searching through lengthy videos, thereby improving the efficiency and accuracy of retrieval.

[0051] In some implementations, the predetermined speech characteristic indicators include predetermined physical characteristics, predetermined acoustic characteristics, predetermined psychoacoustic characteristics, and predetermined emotional characteristics. The predetermined physical characteristics include at least frequency, amplitude, and timbre. The predetermined acoustic characteristics include at least fundamental frequency, formants, bandwidth, and harmonics. The predetermined psychoacoustic characteristics include at least pitch perception, loudness perception, and timbre perception. The predetermined emotional characteristics refer to the state type of the speaker's emotional expression.

[0052] Specifically, in order to accurately identify the target speaker, it is necessary to establish a complete set of predetermined speech characteristic index system (predetermined speech characteristic index). This index system integrates physical characteristics, acoustic characteristics, psychoacoustic characteristics and emotional characteristics, and quantifies and analyzes the speaker's voice characteristics from multiple angles and levels to improve the accuracy and robustness of speaker identification.

[0053] Specifically, physical characteristics refer to the measurable parameters of a sound signal at the objective physical level, including at least frequency, amplitude, and timbre, which can be obtained and characterized through short-time Fourier transform, root mean square energy analysis, sound pressure level measurement, linear predictive cepstral coefficients, and Mel cepstral coefficients, respectively.

[0054] Specifically, the predetermined acoustic features refer to the key frequency components and their distribution in the speaker's speech signal, including at least the fundamental frequency, formants, bandwidth, and harmonics, which can be obtained and characterized through autocorrelation analysis, time-frequency joint analysis, linear predictive coding, cepstral analysis, short-time Fourier transform, and harmonic-noise ratio analysis, respectively.

[0055] Specifically, the predetermined psychoacoustic features refer to the subjective perception of sound by the human ear, including at least pitch perception, loudness perception, and timbre perception, which can be obtained and characterized by Mel-scale frequency transformation, A-weighted filtering, Gammatone filtering, etc.

[0056] Specifically, emotional features refer to the speaker's emotional expression state, including emotional categories such as anger, joy, sadness, surprise, and neutrality; for example, emotional features are defined through multimodal sentiment analysis and deep learning.

[0057] By utilizing the aforementioned physical, acoustic, psychoacoustic, and emotional features, a highly stable and adaptable, multi-dimensional speaker detection system can be established, improving the accuracy of target speaker identification and enhancing recognition performance in multilingual, multi-environmental, and multi-emotional speech analysis scenarios.

[0058] In some implementations, the first speech feature parameters of the first audio / video segment are collected based on the predetermined speech feature indicators, and a first speech feature value is generated, including: The first speech feature parameter is normalized to obtain the first normalized speech feature parameter; the first normalized speech feature parameter is then subjected to a variation weighted calculation to obtain the first speech feature value.

[0059] Optionally, to obtain the first speech feature value, firstly, the extracted first speech feature parameters are normalized using methods such as min-max normalization, Z-score normalization, and wavelet transform normalization to eliminate numerical biases between different features and ensure the stability and fairness of the calculation. Then, the standard deviation or coefficient of variation of each normalized feature parameter is calculated to quantify the instability of the feature. Based on the value of the standard deviation or coefficient of variation, the weights for weighted calculation are defined. In other words, the weights for the variation-weighted calculation are related to the variability to ensure that features with lower variability receive higher weights, emphasizing stable features and improving the reliability of the calculation.

[0060] For example, weight W i The calculation formula is as follows: ; Among them, CV i The coefficient of variation characterizing the i-th feature dimension.

[0061] In some implementations, determining whether the first feature difference meets a predetermined speech feature threshold and obtaining a determination result includes: if it does not meet the threshold, then the first audio / video segment and the first adjacent audio / video segment are different speakers; if it does meet the threshold, then the first audio / video segment and the first adjacent audio / video segment are the same speaker.

[0062] Optionally, the predetermined speech feature threshold is a statistical feature value calculated based on sample data: First, a large number of voice segments of the labeled speakers are selected and their speech feature values ​​are calculated; then, the feature difference between adjacent segments is calculated, the feature difference distribution is statistically analyzed for the same speaker and different speakers, and the predetermined speech feature threshold is set accordingly. By using the above method based on empirical values, the rationality of the threshold is ensured, which helps to reduce the error of human subjective setting and improve the stability of discrimination.

[0063] Specifically, if the first feature difference does not meet the requirements of the predetermined speech feature threshold, such as if the first feature difference is greater than or equal to the predetermined speech feature threshold, then the first audio-visual segment and the first adjacent audio-visual segment are considered to have a large difference and are the speech results of different speakers; correspondingly, if the first feature difference meets the requirements of the predetermined speech feature threshold, such as if the first feature difference is less than the predetermined speech feature threshold, then the first audio-visual segment and the first adjacent audio-visual segment are considered to be similar and can be considered to be the speech results of the same speaker.

[0064] S200: Extract any speaker corresponding to any time in the target speaker time sequence, and obtain any speech content of the arbitrary speaker.

[0065] Specifically, random sampling is performed on the target speakers' time sequence mentioned above, and any speaker corresponding to any time point is selected as the analysis object, and the audio and video content corresponding to that time period is extracted as the analysis object.

[0066] Optionally, the strategy for extracting speakers and speakers from the target speaker time sequence can be flexibly defined according to the requirements of the scenario, such as extracting the speaker who is currently speaking or extracting the speaker who spoke earliest.

[0067] The above process plays a crucial role in content location and extraction in the entire retrieval solution, and can flexibly respond to the diverse retrieval needs of users. Whether users want to extract the opening remarks, the discussion session, or the concluding summary, and regardless of the language of the speech, it can provide the information quickly and accurately, thus improving the flexibility and practicality of the retrieval.

[0068] S300: By using any semantic recognition channel corresponding to any matched language type, any spoken audio in the arbitrary spoken content is identified and analyzed to obtain an arbitrary semantic feature vector.

[0069] Specifically, after obtaining any of the above-mentioned spoken content, firstly, based on phoneme features, language models, character distribution patterns, etc., the language type corresponding to the spoken content is identified. For example, this includes the specific language to which the spoken content belongs, such as Chinese, English, French, Japanese, etc.; specific dialects, industry terminology systems, etc.; then, according to the identified language type, the corresponding semantic recognition channel is selected, and the speech signal is converted into text (speech-text information). This semantic recognition channel is a dedicated speech recognition and natural language processing pipeline for a specific language type.

[0070] Furthermore, natural language processing (NLP) operations are performed on the obtained speech and text information. Taking Chinese as an example, firstly, word segmentation methods based on dictionary matching and statistical modeling, such as maximum probability path segmentation and hidden Markov models, are used to segment the sentence into individual lexical units. For example, "I am learning natural language processing technology" is segmented into "I / are / learning / natural language / processing / technology". Then, conditional random fields or deep learning models are used to label the part of speech of each lexical unit, such as "I" being a pronoun, "are" being an adverb, and "learning" being a verb. Next, the sentence structure is established through dependency parsing or constituent parsing, the subject-verb-object relationship is identified, and key information, such as names, locations, times, and organization names, is extracted through named entity recognition. Finally, sentiment analysis is used to determine the sentiment tendency of the text.

[0071] Furthermore, the text content obtained from the above identification and analysis is numerically represented to generate semantic feature vectors, enabling computers to understand, store, and compute semantic information. For example, methods such as Word2Vec, GloVe, and FastText are used to map lexical units to a fixed-dimensional vector space; pre-trained language models such as BERT (Bidirectional Encoder Representations from Transformers) are used to encode the entire sentence into a fixed-length vector, i.e., an arbitrary semantic feature vector.

[0072] By converting the linguistic information in audio into numerical data (arbitrary semantic feature vectors) that can be processed by computers, a foundation is laid for subsequent semantic matching and retrieval, enabling computer systems to understand the semantic meaning of the spoken content and thus perform content retrieval more accurately.

[0073] S400: Collect features from any video of any speech in any speech content to obtain any visual feature vector.

[0074] Specifically, arbitrary speech video in arbitrary speech content refers to a video segment of a specific speaker extracted from multilingual audio and video content after the previous steps. Through specific technical means, such as image processing algorithms and computer vision technology, key feature information that can characterize the video content can be extracted from the aforementioned video segment.

[0075] Specifically, a visual feature vector is a data structure that represents collected visual features in a numerical way, forming a vector form that can quantitatively reflect the visual content of a video segment, enabling computers to efficiently store, process, and match videos.

[0076] In some embodiments, feature collection is performed on any video recording of any spoken content to obtain an arbitrary visual feature vector, including: The arbitrary speech video is compressed to obtain an arbitrary compressed video; a first real frame is extracted from the arbitrary compressed video, and a key compressed frame is constructed based on the first real frame; a predetermined visual feature index is read, and features are collected from the key compressed frame based on the predetermined visual feature index to obtain the arbitrary visual feature vector; wherein, reading the predetermined visual feature index and collecting features from the key compressed frame based on the predetermined visual feature index to obtain the arbitrary visual feature vector includes: collecting a first visual feature parameter of the first real frame based on the predetermined visual feature index; collecting a second visual feature parameter of a second real frame based on the predetermined visual feature index, wherein the second real frame is different from the first real frame; taking the average value of the first visual feature parameter and the second visual feature parameter as the arbitrary visual feature parameter; and generating the arbitrary visual feature vector based on the arbitrary visual feature parameter.

[0077] Specifically, firstly, the original speech video is compressed to generate a compressed video. This reduces computational load; compression can be achieved by reducing resolution, frame rate, color depth, or bitrate. Then, the first real frame is extracted from the compressed video—that is, the keyframe (I-frame, containing complete image information) or actual image frame from the original video—to form key compressed frames, which guide other frames. Next, based on predetermined visual feature indicators, features are extracted from the key compressed frames to obtain arbitrary visual feature vectors.

[0078] Specifically, to obtain an arbitrary visual feature vector, firstly, the visual feature parameters of the first real frame are extracted. Then, the visual feature parameters of the second real frame (different from the first frame) are extracted. Next, the mean of the first and second visual feature parameters is calculated. The calculated visual feature parameters are combined into a vector as the final visual feature vector to fuse information from multiple frames, reduce the bias of features in a single frame, and improve the reliability of subsequent processing and analysis.

[0079] In some implementations, the predetermined visual feature indicators include predetermined frame features and predetermined scene features, wherein the predetermined frame features include frame tone and frame texture, and the predetermined scene features include at least emotional atmosphere, spatial relationship, brightness distribution and optical flow.

[0080] Specifically, the predetermined visual feature index is a standard used to characterize the visual features of video content, covering predetermined frame features and predetermined scene features. Among them, the predetermined frame features include frame tone and frame texture. Frame tone refers to the overall color tendency and distribution of the video frame, such as warm tone, cool tone, etc., which can be quantified by methods such as color histogram. Frame texture refers to the texture characteristics of the surface of the video frame, such as smooth, rough, regular texture patterns, etc., which can be extracted by texture analysis algorithms.

[0081] Specifically, the features of a pre-defined scene include at least emotional atmosphere, spatial relationships, brightness distribution, and light flow. Emotional atmosphere refers to the emotional tendency conveyed by the scene, such as joy, sadness, or seriousness, which can be determined through scene semantic understanding and analysis of character expressions and actions. Spatial relationships refer to the position and layout relationships between objects and characters in the scene, such as the position of characters in the frame and the relative distance between objects. Brightness distribution refers to the distribution of light and shadow in the scene, including the position of the main light source and shadow areas. Light flow refers to the movement trajectory and speed of pixels between consecutive frames, used to capture dynamic changes in the scene, such as the movement of objects and the movement of the camera.

[0082] By using the aforementioned predetermined visual feature indicators, the rich visual content in the video can be transformed into numerical data that can be processed by a computer, providing important visual basis for subsequent multimodal matching and retrieval, and thus helping to comprehensively utilize voice and visual information for more comprehensive and accurate content retrieval.

[0083] S500: The arbitrary semantic feature vector and the arbitrary visual feature vector are traversed and matched in the multilingual audio and video content database to obtain the target retrieval content.

[0084] Specifically, the arbitrary semantic feature vector and arbitrary visual feature vector obtained above are the retrieval objects for content retrieval. By comparing them one by one in the database, the data record with the highest similarity to the arbitrary semantic feature vector and arbitrary visual feature vector is found to obtain the target retrieval content.

[0085] In some embodiments, such as Figure 2 As shown, the arbitrary semantic feature vector and the arbitrary visual feature vector are traversed and matched in a multilingual audio and video content database to obtain the target retrieval content, including: S510: Construct an arbitrary feature vector based on the arbitrary semantic feature vector and the arbitrary visual feature vector; S520: Extract the first content from the multilingual audio and video content database, the first content corresponding to the first feature vector; S530: Calculate the first similarity between the arbitrary feature vector and the first feature vector; S540: If the first similarity reaches a predetermined similarity threshold, use the first content as the target retrieval content.

[0086] Specifically, firstly, semantic and visual feature vectors are fused using methods such as concatenation, weighted averaging, and principal component analysis to generate a comprehensive feature vector (arbitrary feature vector). This feature vector contains both semantic and visual information of the audio / video segment, providing a more comprehensive feature representation. Then, in a multilingual audio / video database, an audio / video segment (first content) is extracted based on the target search criteria. Using the same feature extraction strategies and dimensions as those used to obtain arbitrary semantic and visual feature vectors, features are extracted from the extracted first content to obtain a first feature vector. This first feature vector contains both semantic and visual information of the content. Next, the similarity between the arbitrary feature vector and the first feature vector is calculated to determine their similarity. Optional similarity metrics include cosine similarity and Euclidean distance. Finally, the calculated similarity is compared with a predetermined similarity threshold. If the similarity is greater than or equal to the threshold, the two audio / video contents are considered similar, a successful match is achieved, and the first content is returned as the target search content. Otherwise, they are considered mismatched, and further matching is performed in the database or no matching content is returned.

[0087] By combining semantic and visual features through the above steps and using similarity calculation methods, content retrieval can be performed efficiently in multilingual audio and video content databases, which helps to improve the matching accuracy of multimodal audio and video content and enhance retrieval efficiency.

[0088] In summary, the multilingual audio and video content retrieval method based on speech recognition provided by this invention has the following technical effects: By invoking a predetermined speaker detection strategy, the target multilingual audio and video content is detected and analyzed to obtain the target speaker time sequence; any speaker corresponding to any time point in the target speaker time sequence is extracted, and their speech content is obtained; based on the matched language type, the audio information of the speech content is identified and analyzed through the corresponding semantic recognition channel to generate a semantic feature vector; features are extracted from the video information of the speech content to generate a visual feature vector; the semantic feature vector and the visual feature vector are traversed and matched in the multilingual audio and video content database to obtain the target retrieval content, thereby achieving the technical effect of improving retrieval efficiency and retrieval accuracy.

[0089] It should be understood that the embodiments disclosed in this invention and the above description enable those skilled in the art to implement this invention. However, this invention is not limited to the embodiments mentioned above. It should be understood that those skilled in the art can still modify the technical solutions described in the foregoing embodiments or make equivalent substitutions for some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this invention, and should all be included within the protection scope of this invention.

Claims

1. A multilingual audio and video content retrieval method based on speech recognition, characterized in that, include: The predetermined speaker detection strategy is retrieved to detect and analyze the target multilingual audio and video content, and the target speaker time sequence is obtained; Extract any speaker corresponding to any time in the target speaker time sequence, and obtain any speech content of the arbitrary speaker; By identifying and analyzing any spoken audio in any spoken content through any semantic recognition channel corresponding to any matched language type, an arbitrary semantic feature vector is obtained; Feature collection is performed on any video of any speech content to obtain an arbitrary visual feature vector; The target retrieval content is obtained by traversing and matching the arbitrary semantic feature vector and the arbitrary visual feature vector in a multilingual audio and video content database.

2. The multilingual audio and video content retrieval method based on speech recognition according to claim 1, characterized in that, The predetermined speaker detection strategy is retrieved to detect and analyze the target multilingual audio and video content, obtaining the target speaker's time sequence, including: The target multilingual audio and video content is segmented to obtain the target segmentation result; Extract the first audio and video segment from the target segmentation result, and obtain the first adjacent audio and video segment of the first audio and video segment; Obtain the predetermined speech feature indicators stored in the predetermined speaker detection strategy; Based on the predetermined speech feature indicators, the first speech feature parameters of the first audio and video segment are collected, and the first speech feature value is generated; Based on the predetermined speech feature indicators, the first adjacent speech feature parameters of the first adjacent audio and video segments are collected, and the first adjacent speech feature value is generated. Calculate the first feature difference between the first spoken feature value and the first adjacent spoken feature value; Determine whether the first feature difference meets the predetermined speech feature threshold to obtain the determination result; The target speaker's timeline is obtained based on the judgment result.

3. The multilingual audio and video content retrieval method based on speech recognition according to claim 2, characterized in that, The predetermined speech characteristic indicators include predetermined physical characteristics, predetermined acoustic characteristics, predetermined psychoacoustic characteristics, and predetermined emotional characteristics. The predetermined physical characteristics include at least frequency, amplitude, and timbre. The predetermined acoustic characteristics include at least fundamental frequency, formants, bandwidth, and harmonics. The predetermined psychoacoustic characteristics include at least pitch perception, loudness perception, and timbre perception. The predetermined emotional characteristics refer to the state type of the speaker's emotional expression.

4. The multilingual audio and video content retrieval method based on speech recognition according to claim 2, characterized in that, Based on the predetermined speech feature indicators, the first speech feature parameters of the first audio-visual segment are collected, and the first speech feature value is generated, including: The first speech feature parameters are normalized to obtain the first normalized speech feature parameters; The first normalized speech feature parameter is subjected to a variation weighted calculation to obtain the first speech feature value.

5. The multilingual audio and video content retrieval method based on speech recognition according to claim 2, characterized in that, Determine whether the first feature difference meets the predetermined speech feature threshold to obtain a determination result, including: if it does not meet the threshold, then the first audio / video segment and the first adjacent audio / video segment are different speakers; if it does meet the threshold, then the first audio / video segment and the first adjacent audio / video segment are the same speaker.

6. The multilingual audio and video content retrieval method based on speech recognition according to claim 1, characterized in that, Feature collection is performed on any video recording of any given speech content to obtain an arbitrary visual feature vector, including: The arbitrary speech video is compressed to obtain an arbitrarily compressed video; Extract the first real frame from the arbitrary compressed video, and construct key compressed frames based on the first real frame; Read the predetermined visual feature index, and collect features from the key compressed frame based on the predetermined visual feature index to obtain the arbitrary visual feature vector; The process includes reading predetermined visual feature indicators and collecting features from the key compressed frame based on these indicators to obtain the arbitrary visual feature vector, including: The first visual feature parameters of the first real frame are collected based on the predetermined visual feature indicators. The second visual feature parameters of the second real frame are collected based on the predetermined visual feature index, and the second real frame is different from the first real frame. Take the average value of the first visual feature parameter and the second visual feature parameter as an arbitrary visual feature parameter; The arbitrary visual feature vector is generated based on the arbitrary visual feature parameters.

7. The multilingual audio and video content retrieval method based on speech recognition according to claim 6, characterized in that, The predetermined visual feature indicators include predetermined frame features and predetermined scene features. The predetermined frame features include frame tone and frame texture, and the predetermined scene features include at least emotional atmosphere, spatial relationship, brightness distribution, and optical flow.

8. The multilingual audio and video content retrieval method based on speech recognition according to claim 1, characterized in that, The target retrieval content is obtained by traversing and matching the arbitrary semantic feature vector and the arbitrary visual feature vector in a multilingual audio and video content database, including: Construct an arbitrary feature vector based on the arbitrary semantic feature vector and the arbitrary visual feature vector; Extract the first content from the multilingual audio and video content database, where the first content corresponds to a first feature vector; Calculate the first similarity between the arbitrary feature vector and the first feature vector; If the first similarity reaches a predetermined similarity threshold, the first content will be used as the target retrieval content.

Images