Media data fusion method and system, core voice switching equipment and storage medium

Abstract

The invention provides a media data fusion method and system, core voice switching equipment and a storage medium, and the method comprises the following steps: carrying out the feature extraction processing of multi-source heterogeneous audio data and video data, and obtaining an audio data stream, a video key frame and the semantic features of the data; performing synchronization processing on the audio data stream and the video key frame based on the semantic features of the data to obtain synchronized audio and video data; the method comprises the following steps: synchronizing audio and video data, extracting key features of the synchronized audio and video data, obtaining content correlation features in the audio and video data, performing feature selection and feature combination processing on the key features of the synchronized audio and video data, and performing content correlation maximization optimization on the content correlation features according to weights of the content correlation features. Obtaining an optimized audio and video feature set; and constructing a data mixed stream optimization model, and carrying out fusion mixed stream processing on the optimized audio and video feature set to obtain fused audio and video mixed stream data.

Description

Technical Field

[0001] The present application relates to the field of data processing technology, and in particular to a media data fusion method, system, core voice switching device and storage medium. Background Art

[0002] The core voice switching gateway is a key device in the communication network. It is responsible for processing and forwarding voice signals to ensure the interconnection between different communication systems. In the context of media convergence, the core voice switching gateway must not only handle traditional voice communications, but also support the transmission of video, text and other multimedia data. Such a gateway can achieve seamless connection between different communication networks and devices, improve communication efficiency and service quality. Media convergence processing involves the integration of multiple communication media, including but not limited to audio, video, data and text.

[0003] There are significant differences between voice data and video data in terms of encoding method, data volume, timestamp, etc., which leads to the problem of audio and video asynchrony when synchronizing the two in timestamp, resulting in poor mixing effect of audio and video mixed stream data. Summary of the invention

[0004] The technical problem to be solved by the present application is that there are significant differences between voice data and video data in terms of encoding method, data volume, timestamp, etc., which leads to the problem of audio and video being out of sync when the two are synchronized in timestamp, resulting in poor mixing effect of audio and video mixed stream data.

[0005] In order to solve the above problems, in order to solve the above technical problems or at least partially solve the above technical problems, the present application provides a media data fusion method, system, core voice switching device and storage medium.

[0006] In a first aspect, the present invention discloses a media data fusion method, which specifically comprises the following steps:

[0007] Perform feature extraction processing on multi-source heterogeneous audio and video data to obtain audio data streams, video key frames and semantic features of data;

[0008] Synchronize the audio data stream and the video key frame based on the semantic features of the data to obtain synchronized audio and video data;

[0009] Extract key features of the synchronized audio and video data, obtain content relevance features in the audio and video data, perform feature selection and feature combination processing on the key features of the synchronized audio and video data, and optimize the content relevance features to maximize the content relevance according to the weight of the content relevance features to obtain an optimized audio and video feature set;

[0010] A data mixing optimization model is constructed, and the optimized audio and video feature set is fused and mixed to obtain fused audio and video mixed data.

[0011] Preferably, the feature extraction process is performed on multi-source heterogeneous audio data and video data to obtain the audio data stream, video key frames and semantic features of the data, specifically including the following steps:

[0012] Acquire multi-source heterogeneous audio and video data, adopt an adaptive decoding algorithm, dynamically select the optimal decoding method according to the voice pause interval and video scene switching frequency of the audio and video data, and obtain the decoded audio data stream and video key frame;

[0013] The decoded audio data stream is subjected to speech recognition to extract text content, speech rate change rate and emotion intensity features, the video key frames are subjected to image understanding to obtain the motion trajectory of the main body of the picture, color saturation change and visual attention distribution, and semantic features are comprehensively generated, wherein the semantic features include keywords, emotion tags and scene tags.

[0014] Preferably, the step of synchronizing the audio data stream and the video key frame based on the semantic features of the data to obtain synchronized audio and video data specifically includes the following steps:

[0015] Extracting the timbre spectrum feature vector of the audio according to the semantic features of the audio data;

[0016] Extracting content label feature vectors of video key frames based on semantic features of video data;

[0017] Construct an audio and video feature similarity calculation model, use a dynamic time warping algorithm, and calculate the similarity between the audio feature vector and the video feature vector within the time window by sliding the time window;

[0018] Determine the initial synchronization position of the audio stream and the video frame according to the calculated audio and video feature similarity;

[0019] Based on the initial synchronization position, the interpolation compensation technology is used to build an audio and video frame mapping model to achieve precise synchronization alignment between the audio stream and the video frame.

[0020] Adopt data repair algorithm to compensate lost audio or video data by interpolating features of adjacent frames;

[0021] The synchronized audio stream and video frame are synthesized to obtain synchronized audio and video data output.

[0022] Preferably, the extracting of key features of the synchronized audio and video data, obtaining content relevance features in the audio and video data, performing feature selection and feature combination processing on the key features of the synchronized audio and video data, and optimizing the content relevance features to maximize the content relevance according to the weights of the content relevance features to obtain an optimized audio and video feature set specifically includes the following steps:

[0023] For the synchronized audio and video data, a multimodal feature fusion method based on deep learning is used to extract the voiceprint features of the audio and the style features of the background music, as well as the features such as the picture composition and the foreground-background ratio of the video;

[0024] According to the preset feature selection strategy, the extracted audio and video features are screened to obtain the feature subset with the strongest relevance to the content;

[0025] For the screened audio and video features, multiple feature combination methods are used for fusion, including series and parallel connection, to obtain the combined audio and video feature vector;

[0026] Input the combined audio and video feature vector into a pre-trained support vector machine classifier to determine the content relevance score of the audio and video;

[0027] If the content relevance score is lower than the preset threshold, a genetic algorithm is used to iteratively optimize the feature weights;

[0028] The feature weight is used as the optimization variable of the genetic algorithm, and the content relevance score is used as the fitness function. New feature weight combinations are generated through evolution operations such as selection, crossover and mutation.

[0029] According to the updated feature weights, the content relevance score of the audio and video is recalculated until the score reaches a preset threshold or the number of iterations exceeds the upper limit, and the optimized audio and video feature set is used as the representation of the audio and video content relevance.

[0030] Preferably, the constructing of the data mixing optimization model, performing fusion mixing processing on the optimized audio and video feature set to obtain fused audio and video mixed data, specifically comprises the following steps:

[0031] Obtain the audio and video data to be mixed, extract key features, and construct an audio and video feature set;

[0032] Input the audio and video feature set into a pre-trained deep learning model, and determine the objective function according to the mixed stream picture layout optimization degree and the mixed stream audio volume balance degree;

[0033] A reinforcement learning algorithm is used, the objective function is taken as the optimization target, and the optimal mixing parameter combination is determined through intelligent search;

[0034] For mixed-stream images, the optimization degree of mixed-stream image layout is quantitatively evaluated by calculating the balance and visual guidance of image element distribution.

[0035] For mixed audio, the volume balance of mixed audio is evaluated by measuring the volume difference and dynamic range between different audio tracks.

[0036] During the mixing process, the visual coherence and content relevance of the mixed stream are analyzed in real time, and the mixing parameters are adaptively adjusted to minimize the switching frequency of the mixed stream data source;

[0037] The optimal mixing parameter combination obtained in the above steps is applied to the audio and video mixing to obtain the fused audio and video mixing data output.

[0038] Preferably, after obtaining the fused audio and video mixed stream data, the following steps are also included:

[0039] Perform quality assessment on the fused audio and video mixed stream data, preset a threshold for the quality assessment result, adjust the mixed stream data according to the quality assessment result, obtain the adjusted mixed stream data if the quality assessment result of the adjusted mixed stream data exceeds the preset threshold, and calculate the mixing effect score for the adjusted mixed stream data.

[0040] Preferably, the method of performing quality assessment on the fused audio and video mixed stream data, presetting a threshold for the quality assessment result, adjusting the mixed stream data according to the quality assessment result to obtain mixed stream data that meets the requirements, and calculating a mixed stream effect score for the mixed stream data that meets the requirements specifically includes the following steps:

[0041] Performing a quality assessment on the fused audio and video mixed stream data, calculating picture clarity, audio signal-to-noise ratio, and content coherence indicators, and if the quality assessment result does not reach a preset threshold, triggering an optimization mechanism, adjusting the mixed stream parameters, and regenerating the mixed stream data until the quality requirements are met;

[0042] A multi-level cache mechanism is used to store audio and video mixed stream data that meets the quality standards. According to the switching frequency of the mixed stream objects and the data heat, the cache strategy is dynamically adjusted to improve the cache hit rate and data reading efficiency. After the storage is completed, the mixed stream effect score is immediately calculated. The score indicators include the rationality of the picture layout, audio balance and content relevance. The objective indicators are converted into subjective scores through the weighted average method.

[0043] According to the mixing effect scoring result, the feature weights and mixing parameters of the audio and video feature set are dynamically adjusted, the optimization intensity is controlled by the weight attenuation coefficient, and the optimization result is fed back to the audio and video feature extraction and mixing parameter selection stage. At the same time, according to the data complexity and processing efficiency, the feature dimension compression rate is updated, the feature dimension is reduced by the principal component analysis method, the noise filtering intensity is adjusted, and the low signal-to-noise ratio components are removed by the adaptive threshold method, so as to continuously improve the intelligent level of audio and video fusion processing until the mixing quality is stable and meets the preset requirements.

[0044] In a second aspect, the present invention discloses a media data fusion system, which includes the steps of any one of the above-mentioned media data fusion methods.

[0045] In a third aspect, the present invention discloses a core voice switching device, which includes a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other via the communication bus;

[0046] Memory, used to store computer programs;

[0047] The processor is used to implement the steps of any one of the media data fusion methods when executing the program stored in the memory.

[0048] In a fourth aspect, the present invention discloses a storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the steps of the media data fusion method as described in any one of the above items.

[0049] The above technical solution provided by this application has the following advantages compared with the prior art:

[0050] The present invention discloses a method for intelligent fusion processing of multi-source heterogeneous audio and video data. Aiming at the complex scenario of synchronization and mixing of multi-source heterogeneous audio and video data, the method extracts audio and video features through adaptive decoding, speech recognition and image understanding, and uses a synchronization algorithm based on timbre spectrum and content labels to achieve precise alignment. Deep learning and reinforcement learning are further used to optimize the mixing parameters, with the optimization of picture layout and volume balance as the objective function, to avoid the problem of audio and video asynchrony in the mixed data and generate high-quality mixed data.

[0051] Furthermore, a multi-level cache mechanism and dynamic feature weight adjustment are introduced to continuously optimize feature extraction and mixing parameters. In this way, the present invention effectively solves the problems of synchronization, mixing quality and processing efficiency in heterogeneous audio and video data fusion, significantly improves the intelligent level and mixing quality of audio and video fusion processing, and provides strong support for audio and video fusion applications in complex scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the invention and, together with the description, serve to explain the principles of the invention.

[0053] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0054] Figure 1 A schematic diagram of a media data fusion method provided in this application Figure 1 ;

[0055] Figure 2 A schematic diagram of a media data fusion method provided in this application Figure 2 ;

[0056] Figure 3 A schematic diagram of a specific step flow of step S1 of a media data fusion method provided in this application;

[0057] Figure 4 A schematic diagram of the specific steps of step S5 of the media data fusion method provided in this application. DETAILED DESCRIPTION

[0058] In order to make the purpose, technical solutions and advantages of this application clearer, the technical solutions in this application will be clearly and completely described below in conjunction with the drawings in this application. Obviously, the described embodiments are part of the embodiments of this application, not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of this application.

[0059] First, see Figure 1-2 The present invention discloses a media data fusion method, which specifically includes the following steps:

[0060] Step S1: extracting features from multi-source heterogeneous audio data and video data to obtain audio data streams, video key frames, and semantic features of the data;

[0061] Step S2: Synchronize the audio data stream and the video key frame based on the semantic features of the data to obtain synchronized audio and video data;

[0062] Step S3: extract key features of the synchronized audio and video data, obtain content relevance features in the audio and video data, perform feature selection and feature combination processing on the key features of the synchronized audio and video data, and optimize the content relevance features by maximizing the content relevance according to the weights of the content relevance features to obtain an optimized audio and video feature set;

[0063] Step S4: construct a data mixing optimization model, perform fusion and mixing processing on the optimized audio and video feature set, and obtain fused audio and video mixed data;

[0064] Step S5: perform quality assessment on the fused audio and video mixed stream data, preset a threshold for the quality assessment result, adjust the mixed stream data according to the quality assessment result, obtain the adjusted mixed stream data if the quality assessment result of the adjusted mixed stream data exceeds the preset threshold, and calculate the mixed stream effect score for the adjusted mixed stream data.

[0065] Specifically, in step S1, multi-source heterogeneous audio and video data are obtained, and an adaptive decoding algorithm is used to dynamically select the optimal decoding method according to the voice pause interval and video scene switching frequency of the audio and video data to obtain the decoded audio data stream and video key frames, perform speech recognition on the decoded audio data stream, extract text content, speech rate change rate and emotion intensity characteristics, perform image understanding on the video key frames, obtain the main motion trajectory of the picture, color saturation change and visual attention distribution, and comprehensively generate semantic features, which include keywords, emotion tags and scene tags.

[0066] Specifically, in step S2, according to the semantic features, an audio and video synchronization algorithm based on timbre and spectrum features and video content tags is adopted, and through dynamic time windows and interpolation compensation, the audio data stream and the video key frames are accurately aligned to eliminate the audio and video asynchrony problem and obtain synchronized audio and video data.

[0067] Specifically, in step S3, key features are extracted from the synchronized audio and video data, and data mining and pattern recognition techniques are used to obtain content-relevance features such as speaker voiceprint features, background music style, picture composition rules, and foreground-background ratio. According to preset feature selection strategies and feature combination methods, the weights of the content-relevance features are dynamically updated through an iterative optimization algorithm, with maximizing the relevance of audio and video content as the optimization goal, to obtain an optimized audio and video feature set.

[0068] Specifically, in step S4, the optimized audio and video feature set is input into a pre-trained deep learning model, and the optimization degree of the mixed stream picture layout and the mixed stream audio volume balance are used as objective functions. The optimal mixed stream parameter combination is searched through a reinforcement learning algorithm. The mixed stream picture layout optimization degree is quantified by calculating the balance and visual guidance of the picture element distribution, and the mixed stream audio volume balance is evaluated by measuring the volume difference and dynamic range between different audio tracks. While ensuring the visual coherence and content relevance of the mixed stream, the switching frequency of the mixed stream data source is minimized, and the fused audio and video mixed stream data is obtained by adaptively adjusting the mixed stream parameters.

[0069] Specifically, in step S5, the quality of the fused audio and video mixed stream data is evaluated, and the picture clarity, audio signal-to-noise ratio and content coherence indicators are calculated. If the quality evaluation result does not reach the preset threshold, the optimization mechanism is triggered to adjust the mixed stream parameters and regenerate the mixed stream data until the quality requirements are met.

[0070] It can be understood that this method targets complex scenarios of synchronization and mixing of multi-source heterogeneous audio and video data, extracts audio and video features through adaptive decoding, speech recognition and image understanding, and uses a synchronization algorithm based on timbre spectrum and content labels to achieve precise alignment. Deep learning and reinforcement learning are further used to optimize mixing parameters, with the optimization of screen layout and volume balance as the objective function to generate high-quality mixed data.

[0071] See also Figure 3 , step S1 specifically includes the following steps:

[0072] Step S11: Acquire multi-source heterogeneous audio and video data, adopt an adaptive decoding algorithm, dynamically select the optimal decoding method according to the voice pause interval and video scene switching frequency of the audio and video data, and obtain the decoded audio data stream and video key frame;

[0073] Step S12: Perform speech recognition on the decoded audio data stream to extract text content, speech rate change rate and emotion intensity features, perform image understanding on the video key frames, obtain the motion trajectory of the main body of the picture, color saturation changes and visual attention distribution, and comprehensively generate semantic features, which include keywords, emotion tags and scene tags.

[0074] It can be understood that for complex scenarios of synchronization and mixing of multi-source heterogeneous audio and video data, audio and video features are extracted through adaptive decoding, speech recognition and image understanding, so as to facilitate the subsequent synchronization of audio data and video data based on the features.

[0075] Step S11 specifically includes the following steps:

[0076] Step S111: Acquire multi-source heterogeneous audio and video data, and extract the voice pause interval and video scene switching frequency characteristics of the audio and video data;

[0077] Step S112: classifying the audio and video data using a support vector machine algorithm according to the extracted voice pause interval and video scene switching frequency characteristics to obtain audio and video data of different categories;

[0078] Step S113: for each type of audio and video data, an adaptive decoding model is established based on a hidden Markov model, and an optimal decoding path is solved by a Viterbi algorithm;

[0079] Step S114: dynamically selecting optimal decoding parameters according to the solved optimal decoding path, and performing adaptive decoding processing on the audio and video data;

[0080] Step S115: performing voice activity detection on the decoded audio data, extracting continuous voice segments, and obtaining an audio data stream; performing shot boundary detection on the decoded video data, extracting key frames, and obtaining a video key frame sequence;

[0081] Step S116: Synchronize the obtained audio data stream and video key frames to construct a multimodal data set for subsequent multimodal analysis and application.

[0082] Specifically, first, audio and video data in different formats are obtained from multiple sources, such as MP4, AVI, WAV, etc. Then, the FFmpeg tool is used to extract the features of the audio and video data, such as the speech pause interval and the video scene switching frequency. Among them, the speech pause interval can be calculated by a voice activity detection algorithm (such as a double threshold method), and the video scene switching frequency can be obtained by analyzing the difference between video frames (such as a histogram difference method). Then, the extracted features are input into a support vector machine (SVM) model for classification. The SVM model can effectively divide the audio and video data into different categories, such as news, movies, sports events, etc., through training. For each type of audio and video data, an adaptive decoding model is established based on a hidden Markov model (HMM), the parameters of the HMM are estimated by the Baum-Welch algorithm, and then the Viterbi algorithm is used to solve the optimal decoding path. According to the optimal decoding path solved, the parameters of the decoder, such as the bit rate, frame rate, etc., are dynamically adjusted to achieve adaptive decoding. For the decoded audio data, a voice activity detection algorithm based on short-time energy and zero-crossing rate is used to extract continuous voice segments and construct an audio data stream. For the decoded video data, a shot boundary detection algorithm based on color histogram and motion vector is used to extract video key frames and generate video key frame sequences. Finally, the audio data stream and video key frames are synchronized by timestamp alignment to construct a multimodal dataset. This dataset can be used for various multimodal analysis tasks, such as speech recognition, face recognition, sentiment analysis, etc., to provide data support for subsequent intelligent applications.

[0083] Step S12 specifically includes the following steps:

[0084] Step S121: segmenting the decoded audio data stream according to a predetermined time, and using a deep learning speech recognition model to transcribe each audio segment into text to obtain the text content of the segment;

[0085] Step S122: Calculate the average speech rate of each audio segment based on the transcribed text. If the speech rate of the current segment changes by more than a predetermined threshold compared with the previous segment, it is determined to be a speech rate change point, and the change rate is recorded for comparison;

[0086] Step S123: performing emotion recognition on the audio signal of each audio clip, using a pre-trained emotion classification model to determine whether the emotion of the clip is positive, neutral or negative, and obtaining an emotion intensity value;

[0087] Step S124: for each key frame image, a target detection algorithm is used to detect the main object in the picture to obtain its bounding box position;

[0088] Step S125: by tracking the position change of the bounding box of the same subject in consecutive key frames, the motion trajectory of the subject in the picture is obtained. If the motion amplitude exceeds a threshold, it is determined to be a significant motion;

[0089] Step S126: extracting color histogram features for each key frame, calculating the histogram similarity between the current frame and the previous frame, and if the similarity is lower than a threshold, determining that the color saturation of the picture has changed significantly;

[0090] Step S127: Use a saliency detection algorithm to process each key frame to obtain a visual attention distribution map of the frame, and combine keywords, emotion tags and scene tags to comprehensively generate a semantic feature description of each segment of the video.

[0091] Specifically, the audio data is segmented every predetermined time (such as 5 seconds), and the text is transcribed using the DeepSpeech speech recognition model. The speech rate is calculated by dividing the number of words by the duration. If the speech rate of adjacent segments changes by more than 20%, it is marked as a speech rate change point. The pre-trained LSTM emotion classification model is used to perform positive, neutral, and negative emotion recognition on the Mel frequency cepstral coefficient features of each segment to obtain the emotion intensity value. A key frame is extracted from the video every 5 seconds, and the bounding box of the main object is detected using YOLOv3. The position change of the continuous bounding box is tracked by Kalman filtering to obtain the motion trajectory. The displacement of more than 50 pixels is judged as significant movement. The HSV color histogram of the key frame is extracted, and the similarity of the adjacent frame histogram is calculated using the Euclidean distance. If it is less than 7, it is judged as a significant change in color saturation. The spectral clustering saliency detection algorithm is used to obtain the visual attention distribution map of the key frame. Finally, the audio text keywords, emotion tags, and video scene tags are combined to generate a semantic feature description of each segment to form a multimodal analysis result of the video.

[0092] Step S2 specifically includes the following steps:

[0093] Step S21: extracting the timbre spectrum feature vector of the audio according to the semantic features of the audio data;

[0094] Step S22: extracting content label feature vectors of video key frames according to semantic features of the video data;

[0095] Step S23: constructing an audio and video feature similarity calculation model, using a dynamic time warping algorithm, and calculating the similarity between the audio feature vector and the video feature vector within the time window by sliding the time window;

[0096] Step S24: determining the initial synchronization position of the audio stream and the video frame according to the calculated audio and video feature similarity;

[0097] Step S25: Based on the initial synchronization position, an interpolation compensation technology is used to construct an audio and video frame mapping model to achieve accurate synchronization alignment between the audio stream and the video frame;

[0098] Step S26: using a data repair algorithm to compensate for lost audio or video data by interpolating features of adjacent frames;

[0099] Step S27: synthesize the synchronized audio stream and video frame to obtain synchronized audio and video data output.

[0100] Specifically, the audio data is first preprocessed, and the timbre spectrum features of the audio are extracted by short-time Fourier transform to obtain a 128-dimensional timbre spectrum feature vector. At the same time, the key frame is extracted from the video data, and the content of the key frame is identified by a convolutional neural network to extract a 256-dimensional content label feature vector. Then, an audio and video feature similarity calculation model is constructed, and a dynamic time warping algorithm is used to set a 5-second sliding time window to calculate the cosine similarity between the audio feature vector and the video feature vector in the time window. According to the calculated audio and video feature similarity matrix, the initial synchronization position of the audio stream and the video frame, that is, the corresponding position with the highest similarity, is determined by a dynamic programming algorithm. On the basis of the initial synchronization position, a cubic spline interpolation algorithm is used to construct an audio and video frame mapping model, and the audio stream and the video frame are precisely synchronized and aligned, so that the timestamp error of the audio and video frame is controlled within 10 milliseconds. In view of the problem of audio or video frame loss in the audio and video synchronization process, a data repair algorithm is used to compensate for the lost data through feature interpolation of adjacent frames to ensure the continuity of audio and video synchronization. Finally, the synchronized audio stream and video frames are synthesized, and the synchronized audio and video data are output in H.264 encoding format and AAC encoding format through the FFmpeg multimedia framework to achieve high-quality synchronized audio and video playback.

[0101] Step S3 specifically includes the following steps:

[0102] Step S31: for the synchronized audio and video data, a multimodal feature fusion method based on deep learning is used to extract the voiceprint features of the audio and the style features of the background music, as well as the features such as the picture composition and the foreground-background ratio of the video;

[0103] Step S32: Screening the extracted audio and video features according to a preset feature selection strategy to obtain a feature subset with the strongest relevance to the content;

[0104] Step S33: for the screened audio and video features, multiple feature combination methods are used for fusion, including series connection and parallel connection, to obtain a combined audio and video feature vector;

[0105] Step S34: inputting the combined audio and video feature vector into a pre-trained support vector machine classifier to determine the content relevance score of the audio and video;

[0106] Step S35: if the content relevance score is lower than a preset threshold, a genetic algorithm is used to iteratively optimize the feature weights;

[0107] Step S36: Using the feature weight as the optimization variable of the genetic algorithm and the content relevance score as the fitness function, a new feature weight combination is generated through evolution operations such as selection, crossover and mutation;

[0108] Step S37: recalculate the content relevance score of the audio and video according to the updated feature weights until the score reaches a preset threshold or the number of iterations exceeds an upper limit, and the optimized audio and video feature set is used as a representation of the audio and video content relevance.

[0109] Specifically, the synchronized audio and video data are first preprocessed, the Mel frequency cepstral coefficient is used to extract the voiceprint features of the audio, and the convolutional neural network is used to extract the style features of the background music. In terms of video, the main target in the picture is detected by the target detection algorithm YOLO v3, and its proportion of the picture is calculated as the foreground background ratio feature, and the picture composition feature is extracted using the VGG-19 network. Then, according to the variance size and mutual information of the features, the 50 features with the strongest content relevance are selected as the feature subset using the variance selection method and the maximum relevance minimum redundancy method. Then, the selected audio features and video features are normalized respectively, and then fused into a 200-dimensional audio and video feature vector by concatenation. The feature vector is classified using the pre-trained support vector machine model to obtain the content relevance score. If the score is lower than the threshold of 6, the genetic algorithm is used to optimize the feature weight. The feature weight is encoded as a binary string of length 50, and 100 initial weights are randomly generated as the population. In each iteration, the 20 individuals with the highest content relevance score are selected from the population, and new offspring are generated through genetic operations with a crossover probability of 8 and a mutation probability of 1. After 50 iterations, the weight corresponding to the individual with the highest fitness function value in the population is used as the optimal weight for feature importance adjustment. Finally, the weighted audio and video features are used as a representation of the relevance of the audio and video content for subsequent multimedia analysis tasks.

[0110] Step S4 specifically includes the following steps:

[0111] Step S41: inputting the optimized audio and video feature set into the pre-trained deep learning model, and determining the objective function according to the mixed stream picture layout optimization degree and the mixed stream audio volume balance degree;

[0112] Step S43: using a reinforcement learning algorithm, taking the objective function as the optimization target, and determining the optimal combination of mixing parameters through intelligent search;

[0113] Step S44: for the mixed-stream picture, quantitatively evaluate the optimization degree of the mixed-stream picture layout by calculating the balance and visual guidance of the picture element distribution;

[0114] Step S45: for the mixed stream audio, by measuring the volume difference and dynamic range between different audio tracks, the volume balance of the mixed stream audio is evaluated;

[0115] Step S46: During the mixing process, the mixing parameters are adaptively adjusted by analyzing the visual coherence and content relevance of the mixed stream in real time to minimize the switching frequency of the mixed stream data source;

[0116] Step S47: Apply the optimal mixing parameter combination obtained in the above steps to the audio and video mixing to obtain the fused audio and video mixing data output.

[0117] Specifically, first, key features are extracted from the audio and video data to be mixed, such as the brightness, contrast, motion vector, etc. of the video, as well as the pitch, volume, spectrum, etc. of the audio. The extracted features are analyzed and represented using deep learning models such as convolutional neural networks (CNN) and long short-term memory networks (LSTM), and a set of audio and video features containing spatiotemporal information is constructed. Then, the feature set is input into a pre-trained deep reinforcement learning model, which takes the optimization degree of the mixed stream screen layout and the balance degree of the mixed stream audio volume as the optimization goals. Through reinforcement learning algorithms such as policy gradient, the model intelligently searches in the high-dimensional parameter space to find the optimal combination of mixed stream parameters. Among them, the optimization degree of the mixed stream screen layout can be quantitatively evaluated by calculating indicators such as the distribution entropy and visual significance of the screen elements. For example, the Gini coefficient is used to measure the balance of the distribution of the screen elements, and the value range is 0 to 1. The closer to 1, the more balanced the distribution; and the balance degree of the mixed stream audio volume can be evaluated by measuring the relative volume difference and dynamic range between different tracks. For example, the root mean square energy ratio of the audio signal is calculated. If the ratio is between 8 and 2, the volume is considered to be relatively balanced. In the actual mixing process, factors such as visual coherence and content relevance also need to be considered. By analyzing the similarity between video frames and the semantic relevance of audio in real time, the mixing parameters can be adaptively adjusted. For example, when the similarity is less than 6, the switching of the mixed screen is triggered to minimize the switching frequency of the mixed data source and improve the user experience. Finally, the mixed parameter combination optimized by the above steps is applied to the encoding, decoding and rendering of audio and video mixing, and the fused high-quality mixed result can be output.

[0118] See also Figure 4 , step S5 specifically includes the following steps:

[0119] Step S51: performing a quality assessment on the fused audio and video mixed stream data, calculating the picture clarity, audio signal-to-noise ratio and content coherence index; if the quality assessment result does not reach a preset threshold, triggering an optimization mechanism, adjusting the mixed stream parameters and regenerating the mixed stream data until the quality requirements are met;

[0120] Step S52: A multi-level cache mechanism is used to store audio and video mixed stream data that meets the quality standards. According to the switching frequency of the mixed stream objects and the data heat, the cache strategy is dynamically adjusted to improve the cache hit rate and data reading efficiency. After the storage is completed, the mixed stream effect score is immediately calculated. The score indicators include the rationality of the picture layout, the audio balance and the content relevance. The objective indicators are converted into subjective scores through the weighted average method;

[0121] Step S53: According to the mixing effect scoring result, dynamically adjust the feature weights and mixing parameters of the audio and video feature set, control the optimization intensity through the weight attenuation coefficient, and feed back the optimization results to the audio and video feature extraction and mixing parameter selection stage. At the same time, according to the data complexity and processing efficiency, update the feature dimension compression rate, reduce the feature dimension through the principal component analysis method, adjust the noise filtering intensity, and use the adaptive threshold method to remove low signal-to-noise ratio components, so as to continuously improve the intelligent level of audio and video fusion processing until the mixing quality is stable and meets the preset requirements.

[0122] It can be understood that the introduction of multi-level cache mechanism and dynamic feature weight adjustment continuously optimizes feature extraction and mixing parameters. In this way, the present invention effectively solves the problems of synchronization, mixing quality and processing efficiency in heterogeneous audio and video data fusion, significantly improves the intelligent level and mixing quality of audio and video fusion processing, and provides strong support for audio and video fusion applications in complex scenarios.

[0123] Step S51 specifically includes the following steps:

[0124] Step S511: obtaining the fused audio and video mixed stream data, extracting the video frames and audio frames from the mixed stream data, and performing quality assessment processing on each of them;

[0125] Step S512: for the extracted video frame, using an image quality assessment algorithm, calculating the clarity index of the video frame to obtain a video quality score;

[0126] Step S513: for the extracted audio frame, using an audio quality assessment algorithm, calculating the signal-to-noise ratio index of the audio frame to obtain an audio quality score;

[0127] Step S514: judging the coherence of the audio and video content according to the video quality score and the audio quality score, combined with the timestamp information of the video and audio frames, and obtaining a content coherence index;

[0128] Step S515: performing weighted summation on the video quality score, the audio quality score and the content coherence index to obtain a comprehensive quality assessment result of the mixed stream data;

[0129] Step S516: determining whether the comprehensive quality evaluation result reaches a preset threshold; if not, triggering an optimization mechanism to determine the mixing parameters that need to be optimized based on the comprehensive quality evaluation result;

[0130] Step S517: adjusting the audio and video encoding parameters, frame rate and bit rate, etc. according to the mixed stream parameters to be optimized, and regenerating the optimized mixed stream data;

[0131] Step S518: perform quality assessment on the optimized mixed stream data again, repeat the above steps until the comprehensive quality assessment result meets the preset quality requirements, and output the final high-quality mixed stream data.

[0132] Specifically, first obtain the fused audio and video mixed stream data, and use FFmpeg and other tools to extract video frames and audio frames for the mixed stream data. For the extracted video frames, use a deep learning-based image quality assessment algorithm, such as BRISQUE or NIMA, to calculate the clarity index of each frame, and take the average value as the video quality score, with a score range of 0 to 100 points. For the extracted audio frames, use an audio quality assessment algorithm based on PESQ or POLQA to calculate the signal-to-noise ratio index of each frame, and take the average value as the audio quality score, with a score range of 0 to 5 points. According to the video quality score and audio quality score, combined with the timestamp information of the video and audio frames, the coherence of the audio and video content is judged by calculating the difference between the timestamps of the video frame and the audio frame. If the difference is less than 100 milliseconds, it is considered that the coherence is good, and the content coherence index is 1, otherwise it is 0. The video quality score, audio quality score, and content coherence index are multiplied by weights 5, 3, and 2 respectively, and then summed to obtain the comprehensive quality assessment result of the mixed stream data. Determine whether the comprehensive quality assessment result reaches the preset threshold of 85 points. If not, the optimization mechanism is triggered. According to the comprehensive quality assessment result, the decision tree algorithm is used to determine the mixed stream parameters that need to be optimized, such as video encoding parameters, audio encoding parameters, frame rate, bit rate, etc. According to the mixed stream parameters that need to be optimized, call the relevant interface of FFmpeg to dynamically adjust the mixed stream parameters, such as adjusting the video encoding from H. 264 to H. 265, the audio encoding from AAC to Opus, the video frame rate from 25 to 30, and the audio bit rate from 128kbps to 192kbps, and regenerate the optimized mixed stream data. The optimized mixed stream data is evaluated again for quality, and the above steps are repeated until the comprehensive quality assessment result meets the preset quality requirements, and the final high-quality mixed stream data is output.

[0133] Step S52 specifically includes the following steps:

[0134] Step S521: obtaining audio and video mixed stream data, and determining whether the mixed stream data meets the quality standard according to a preset quality threshold. If the mixed stream data meets the quality standard, storing the mixed stream data in a multi-level cache;

[0135] Step S522: for the mixed flow data in the multi-level cache, obtain the corresponding mixed flow object switching frequency and data heat, and dynamically adjust the cache strategy according to the preset strategy rules to improve the cache hit rate and data reading efficiency;

[0136] Step S523: After the mixed stream data is stored, the mixed stream effect scoring process is immediately triggered to obtain multiple scoring indicators such as the rationality of the picture layout, the audio balance and the content relevance;

[0137] Step S524: for the obtained multiple scoring indicators, a weighted average algorithm is used to calculate the weight of each indicator, and a weighted average value is obtained as a subjective score of the mixing effect;

[0138] Step S525: Compare the subjective score with a preset threshold. If the score is higher than the threshold, it is determined to be a high-quality mixed flow effect, and the corresponding service process is triggered;

[0139] Step S526: If the value is lower than the threshold, it is determined as a low-quality mixing effect, triggering optimization or re-mixing processes;

[0140] Step S527: continuously monitor the mixed stream data in the multi-level cache, dynamically update the cache according to the timeliness of the data, eliminate low-heat data, and ensure efficient use of storage resources;

[0141] Step S528: Use machine learning algorithms such as decision trees and support vector machines to train and optimize cache strategies and scoring weights based on historical mixed flow data and scoring indicators to improve the system's adaptability and intelligence level.

[0142] Specifically, when obtaining audio and video mixed stream data, the H. 264 encoding format is used, the resolution is set to 1920×1080, the frame rate is 60fps, and the bit rate is controlled within 8Mbps. The FFmpeg tool is used to analyze the quality of the mixed stream data, extract objective quality indicators such as PSNR and SSIM, and if the PSNR is higher than 35dB and the SSIM is greater than 9, it is determined to be qualified mixed stream data. The qualified data is stored in a two-level cache architecture consisting of Redis and Memcached. Among them, the hot data, such as the data with more than 10 access times in the past 5 minutes, is stored in Redis first, and the rest of the data is stored in Memcached. For the cached data, the switching frequency of the mixed stream object and the data heat are counted every 1 minute. If the switching frequency exceeds 5 times / minute or the heat is less than 100 times / day, the cache strategy adjustment is triggered to reduce the data expiration time or eliminate cold data. In the evaluation of the mixing effect, the hierarchical analysis method is used to compare the three indicators of picture layout rationality, audio balance and content relevance in pairs, build a judgment matrix and calculate the weight vector, and finally get the weights of 54, 16 and 30 respectively. The score of each indicator is multiplied by the weight and then summed to get the subjective score of the mixing effect. If the subjective score is higher than 2 points (out of 5 points), it is judged as high-quality mixing, triggering subsequent processes such as content distribution and user push; if the score is lower than 5 points, it is judged as low-quality mixing, triggering audio and video parameter tuning or re-mixing. The system scans the cache data every 5 minutes, and performs heat decay on data with a storage time of more than 1 hour, with a decay rate of 8. After decay, the data with a heat of less than 50 times / day is deleted, and the cache utilization rate is ensured to reach more than 80% through regular cleaning. At the same time, historical mixing data is collected as training samples, and features such as mixing object type, audio and video parameters, and cache access mode are extracted. The CART decision tree algorithm is used to optimize the cache strategy, and the SVM algorithm is used to optimize the scoring weight. The model is updated once a day to achieve adaptive adjustment of strategies and weights and improve the intelligence level of the system.

[0143] Step S53 specifically includes the following steps:

[0144] Step S531: Obtain the score result of the mixing effect, determine the adjustment direction and amplitude of the feature weight and the mixing parameter according to the score result, and control the optimization intensity through the weight attenuation coefficient;

[0145] Step S532: Feedback the optimized feature weights and mixing parameters to the audio and video feature extraction and mixing parameter selection module to update the feature extraction and parameter selection strategy;

[0146] Step S533: dynamically adjust the feature dimension compression rate according to data complexity and processing efficiency, and use principal component analysis to reduce the dimension of high-dimensional features to improve the simplicity and robustness of feature representation;

[0147] Step S534: performing noise filtering on the audio and video data, automatically adjusting the filtering intensity through the adaptive threshold method, removing interference components with low signal-to-noise ratio, and improving the audio and video quality;

[0148] Step S535: applying the optimized audio and video feature set and mixing parameters to audio and video fusion processing, generating mixed audio and video data, and evaluating the mixing effect;

[0149] Step S536: If the mixing effect does not meet the preset requirements, return to step 1 and continue to iteratively optimize the feature weights, mixing parameters, feature dimensions and noise filtering strength until the mixing quality is stable and meets the preset requirements;

[0150] Step S537: output the optimized audio and video feature set, mixing parameters and high-quality audio and video data after mixing, and complete the intelligent audio and video fusion processing.

[0151] Specifically, after obtaining the score results of the mixing effect, the system determines the adjustment direction and amplitude of the feature weight and the mixing parameters according to the score results. For example, when the score is lower than 8, the feature weight adjustment amplitude is set to 05, and the mixing parameter adjustment amplitude is set to 1; when the score is between 8 and 9, the feature weight adjustment amplitude is set to 03, and the mixing parameter adjustment amplitude is set to 05; when the score is higher than 9, the feature weight adjustment amplitude is set to 01, and the mixing parameter adjustment amplitude is set to 02. At the same time, by setting the weight decay coefficient to 95, the intensity of each iteration optimization is controlled. The optimized feature weights and mixing parameters are fed back to the audio and video feature extraction and mixing parameter selection module to update the feature extraction and parameter selection strategy. The system uses principal component analysis to reduce the dimensionality of high-dimensional features, and dynamically adjusts the feature dimension compression rate according to the data complexity and processing efficiency. When the data complexity is high and the processing efficiency is low, the compression rate is increased to 80%; when the complexity is moderate and the efficiency is acceptable, the compression rate is set to 50%; when the data is simple and the processing is efficient, the compression rate can be reduced to 20%. Adaptive threshold noise filtering is performed on audio and video data. The threshold is automatically adjusted according to the signal-to-noise ratio. When the signal-to-noise ratio is lower than 10dB, the filtering strength is increased to 90%; when the signal-to-noise ratio is between 10dB and 20dB, the filtering strength is set to 70%; when the signal-to-noise ratio is higher than 20dB, the filtering strength is reduced to 50%. The optimized audio and video feature set and mixing parameters are applied to the audio and video fusion processing to generate the mixed audio and video data and evaluate the mixing effect. If the effect does not meet the preset requirements, it returns to continue iterative optimization until the mixing quality is stable and meets the requirements. Finally, the system outputs the optimized audio and video feature set, mixing parameters and high-quality mixed audio and video data to achieve intelligent audio and video fusion processing.

[0152] In a second aspect, the present invention discloses a media data fusion system, which includes the steps of any one of the above-mentioned media data fusion methods.

[0153] Specifically, the system targets complex scenarios of synchronization and mixing of multi-source heterogeneous audio and video data, extracts audio and video features through adaptive decoding, speech recognition and image understanding, and uses a synchronization algorithm based on timbre spectrum and content tags to achieve precise alignment. Deep learning and reinforcement learning are further used to optimize the mixing parameters, with the degree of picture layout optimization and volume balance as the objective function to avoid the problem of audio and picture asynchrony in the mixed data and generate high-quality mixed data. A multi-level cache mechanism and dynamic feature weight adjustment are introduced to continuously optimize feature extraction and mixing parameters. In this way, the present invention effectively solves the problems of synchronization, mixing quality and processing efficiency in the fusion of heterogeneous audio and video data, significantly improves the intelligence level and mixing quality of audio and video fusion processing, and provides strong support for audio and video fusion applications in complex scenarios.

[0154] In a third aspect, the present invention discloses a core voice switching device, which includes a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other via the communication bus;

[0155] Memory, used to store computer programs;

[0156] The processor is used to implement the steps of any one of the media data fusion methods when executing the program stored in the memory.

[0157] It can be understood that the core voice switching equipment extracts audio and video features through adaptive decoding, voice recognition and image understanding for complex scenarios of synchronization and mixing of multi-source heterogeneous audio and video data, and uses a synchronization algorithm based on timbre spectrum and content tags to achieve precise alignment. Deep learning and reinforcement learning are further used to optimize the mixing parameters, with the optimization of picture layout and volume balance as the objective function to avoid the problem of audio and picture asynchrony in the mixed data and generate high-quality mixed data. In addition, a multi-level cache mechanism and dynamic feature weight adjustment are introduced to continuously optimize feature extraction and mixing parameters. In this way, the present invention effectively solves the problems of synchronization, mixing quality and processing efficiency in the fusion of heterogeneous audio and video data, significantly improves the intelligence level and mixing quality of audio and video fusion processing, and provides strong support for audio and video fusion applications in complex scenarios.

[0158] In a fourth aspect, the present invention discloses a storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the steps of the media data fusion method as described in any one of the above items.

[0159] It can be understood that the storage medium stores a media data fusion method, which can synchronously mix audio and video data, synchronize the audio data with the video data in terms of timestamps, and output high-quality mixed data.

[0160] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0161] In the description of the present invention, it should be understood that the terms "center", "longitudinal", "lateral", "length", "width", "thickness", "up", "down", "front", "back", "left", "right", "vertical", "horizontal", "top", "bottom", "inside", "outside", "clockwise", "counterclockwise" and the like indicate orientations or positional relationships based on the orientations or positional relationships shown in the accompanying drawings, and are only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the referred device or element must have a specific orientation, be constructed and operated in a specific orientation, and therefore should not be understood as limiting the present invention.

[0162] In addition, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined as "first" and "second" may explicitly or implicitly include one or more of the features. In the description of the present invention, the meaning of "plurality" is two or more, unless otherwise clearly and specifically defined.

[0163] In the present invention, unless otherwise clearly specified and limited, the terms "installed", "connected", "connected", "fixed" and the like should be understood in a broad sense, for example, it can be connected, detachably connected, or integrated; it can be mechanically connected or electrically connected; it can be directly connected or indirectly connected through an intermediate medium, it can be the internal connection of two elements or the interaction relationship between two elements. For ordinary technicians in this field, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.

[0164] In the present invention, unless otherwise clearly specified and limited, a first feature being "above" or "below" a second feature may include that the first and second features are in direct contact, or may include that the first and second features are not in direct contact but are in contact through another feature between them. Moreover, a first feature being "above", "above" and "above" a second feature includes that the first feature is directly above and obliquely above the second feature, or simply indicates that the first feature is higher in level than the second feature. A first feature being "below", "below" and "below" a second feature includes that the first feature is directly below and obliquely below the second feature, or simply indicates that the first feature is lower in level than the second feature.

[0165] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representation of the above terms should not be understood as necessarily referring to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art may combine and combine the different embodiments or examples described in this specification.

[0166] Obviously, those skilled in the art can make various changes and modifications to the present invention without departing from the spirit and scope of the present invention. Thus, if these modifications and variations of the present invention fall within the scope of the claims of the present invention and their equivalents, the present invention is also intended to include these modifications and variations.

[0167] The above is a specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any technician familiar with the technical field can easily think of various equivalent modifications or replacements within the technical scope disclosed by the present invention, and these modifications or replacements should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention shall be based on the protection scope of the claims.

Claims

1. A media data fusion method, characterized in that: The specific steps include: Perform feature extraction processing on multi-source heterogeneous audio and video data to obtain audio data streams, video key frames and semantic features of data; Synchronize the audio data stream and the video key frame based on the semantic features of the data to obtain synchronized audio and video data; Extract key features of the synchronized audio and video data, obtain content relevance features in the audio and video data, perform feature selection and feature combination processing on the key features of the synchronized audio and video data, and optimize the content relevance features to maximize the content relevance according to the weight of the content relevance features to obtain an optimized audio and video feature set; A data mixing optimization model is constructed, and the optimized audio and video feature set is fused and mixed to obtain fused audio and video mixed data.

2. The media data fusion method according to claim 1, characterized in that: The feature extraction process is performed on multi-source heterogeneous audio data and video data to obtain the audio data stream, video key frames and semantic features of the data, specifically including the following steps: Acquire multi-source heterogeneous audio and video data, adopt an adaptive decoding algorithm, dynamically select the optimal decoding method according to the voice pause interval and video scene switching frequency of the audio and video data, and obtain the decoded audio data stream and video key frame; The decoded audio data stream is subjected to speech recognition to extract text content, speech rate change rate and emotion intensity features, the video key frames are subjected to image understanding to obtain the motion trajectory of the main body of the picture, color saturation change and visual attention distribution, and semantic features are comprehensively generated, wherein the semantic features include keywords, emotion tags and scene tags.

3. The media data fusion method according to claim 1, characterized in that: The method of synchronizing the audio data stream and the video key frame based on the semantic features of the data to obtain synchronized audio and video data specifically includes the following steps: Extracting the timbre spectrum feature vector of the audio according to the semantic features of the audio data; Extracting content label feature vectors of video key frames based on semantic features of video data; Construct an audio and video feature similarity calculation model, use a dynamic time warping algorithm, and calculate the similarity between the audio feature vector and the video feature vector within the time window by sliding the time window; Determine the initial synchronization position of the audio stream and the video frame according to the calculated audio and video feature similarity; Based on the initial synchronization position, the interpolation compensation technology is used to build an audio and video frame mapping model to achieve precise synchronization alignment between the audio stream and the video frame. Adopt data repair algorithm to compensate lost audio or video data by interpolating features of adjacent frames; The synchronized audio stream and video frame are synthesized to obtain synchronized audio and video data output.

4. The media data fusion method according to claim 1, characterized in that: The extracting of key features of the synchronized audio and video data, obtaining content relevance features in the audio and video data, performing feature selection and feature combination processing on the key features of the synchronized audio and video data, and optimizing the content relevance of the content relevance features according to the weights of the content relevance features to obtain an optimized audio and video feature set specifically includes the following steps: For the synchronized audio and video data, a multimodal feature fusion method based on deep learning is used to extract the voiceprint features of the audio and the style features of the background music, as well as the features such as the picture composition and the foreground-background ratio of the video; According to the preset feature selection strategy, the extracted audio and video features are screened to obtain the feature subset with the strongest relevance to the content; For the screened audio and video features, multiple feature combination methods are used for fusion, including series and parallel connection, to obtain the combined audio and video feature vector; Input the combined audio and video feature vector into a pre-trained support vector machine classifier to determine the content relevance score of the audio and video; If the content relevance score is lower than the preset threshold, a genetic algorithm is used to iteratively optimize the feature weights; The feature weight is used as the optimization variable of the genetic algorithm, and the content relevance score is used as the fitness function. New feature weight combinations are generated through evolution operations such as selection, crossover and mutation. According to the updated feature weights, the content relevance score of the audio and video is recalculated until the score reaches a preset threshold or the number of iterations exceeds the upper limit, and the optimized audio and video feature set is used as the representation of the audio and video content relevance.

5. The media data fusion method according to claim 1, characterized in that: The constructing data mixing optimization model, performing fusion mixing processing on the optimized audio and video feature set, and obtaining fused audio and video mixed stream data specifically includes the following steps: Obtain the audio and video data to be mixed, extract key features, and construct an audio and video feature set; Input the audio and video feature set into a pre-trained deep learning model, and determine the objective function according to the mixed stream picture layout optimization degree and the mixed stream audio volume balance degree; A reinforcement learning algorithm is used, the objective function is taken as the optimization target, and the optimal mixing parameter combination is determined through intelligent search; For mixed-stream images, the optimization degree of mixed-stream image layout is quantitatively evaluated by calculating the balance and visual guidance of image element distribution. For mixed audio, the volume balance of mixed audio is evaluated by measuring the volume difference and dynamic range between different audio tracks. During the mixing process, the visual coherence and content relevance of the mixed stream are analyzed in real time, and the mixing parameters are adaptively adjusted to minimize the switching frequency of the mixed stream data source; The optimal mixing parameter combination obtained in the above steps is applied to the audio and video mixing to obtain the fused audio and video mixing data output.

6. The media data fusion method according to claim 1, characterized in that: After obtaining the fused audio and video mixed stream data, the following steps are also included: Perform quality assessment on the fused audio and video mixed stream data, preset a threshold for the quality assessment result, adjust the mixed stream data according to the quality assessment result, obtain the adjusted mixed stream data if the quality assessment result of the adjusted mixed stream data exceeds the preset threshold, and calculate the mixing effect score for the adjusted mixed stream data.

7. The media data fusion method according to claim 6, characterized in that: The method of performing quality assessment on the fused audio and video mixed stream data, presetting a threshold for the quality assessment result, adjusting the mixed stream data according to the quality assessment result to obtain mixed stream data that meets the requirements, and calculating a mixed stream effect score for the mixed stream data that meets the requirements specifically includes the following steps: Performing a quality assessment on the fused audio and video mixed stream data, calculating picture clarity, audio signal-to-noise ratio, and content coherence indicators, and if the quality assessment result does not reach a preset threshold, triggering an optimization mechanism, adjusting the mixed stream parameters, and regenerating the mixed stream data until the quality requirements are met; A multi-level cache mechanism is used to store audio and video mixed stream data that meets the quality standards. According to the switching frequency of the mixed stream objects and the data heat, the cache strategy is dynamically adjusted to improve the cache hit rate and data reading efficiency. After the storage is completed, the mixed stream effect score is immediately calculated. The score indicators include the rationality of the picture layout, audio balance and content relevance. The objective indicators are converted into subjective scores through the weighted average method. According to the mixing effect scoring result, the feature weights and mixing parameters of the audio and video feature set are dynamically adjusted, the optimization intensity is controlled by the weight attenuation coefficient, and the optimization result is fed back to the audio and video feature extraction and mixing parameter selection stage. At the same time, according to the data complexity and processing efficiency, the feature dimension compression rate is updated, the feature dimension is reduced by the principal component analysis method, the noise filtering intensity is adjusted, and the low signal-to-noise ratio components are removed by the adaptive threshold method, so as to continuously improve the intelligent level of audio and video fusion processing until the mixing quality is stable and meets the preset requirements.

8. A media data feature analysis system, characterized in that: The method comprises the steps of the media data fusion method described in any one of claims 1 to 7.

9. A core voice switching device, characterized in that: It includes a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other through the communication bus; Memory, used to store computer programs; The processor is used to implement the steps of the media data fusion method described in any one of claims 1 to 7 when executing the program stored in the memory.

10. A storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the media data fusion method as described in any one of claims 1 to 7 are implemented.

Images