Video polyphonic ringtone label generation method, system and device and medium

Abstract

The invention discloses a video polyphonic ringtone label generation method, system and device and a medium, and the method comprises the steps: obtaining multi-mode video polyphonic ringtone data, carrying out the preprocessing of the multi-mode video polyphonic ringtone data, obtaining the preprocessing data, carrying out the spatial-temporal feature extraction of the preprocessing data, obtaining a video spatial-temporal feature, and obtaining a video polyphonic ringtone label; and performing multi-modal feature fusion processing on the video spatial-temporal features to obtain multi-modal features, constructing a dynamic semantic map, and performing label query processing on the multi-modal features according to the dynamic semantic map to obtain video polyphonic ringtone labels. The embodiment of the invention can improve the consistency of labels and content semantics, and can be widely applied to the technical field of video processing.

Description

Technical Field

[0001] The present application relates to the field of video processing technology, and in particular to a method, system, device and medium for generating a video ring back tone tag. Background Art

[0002] With the popularization of 5G communication technology and the explosive growth of video ringback tone services, users are increasingly demanding personalized and scenario-based video ringback tone content. In related technologies, there is a method of generating video ringback tone labels by text recognition, which generates labels by performing text recognition on the video subtitles of the video ringback tone. However, in actual applications, it has been found that related text recognition technologies are insufficient in handling interference from complex background videos, resulting in low recognition accuracy. At the same time, the independent processing of audio and video features in related technologies leads to a high mismatch rate between labels and content semantics, resulting in low accuracy of generated labels, which affects user experience and the efficiency of content recommendation. In summary, the technical problems existing in related technologies need to be improved. Summary of the Invention

[0003] The main purpose of the embodiments of the present application is to provide a method, system, device and medium for generating video ringback tone tags, which can improve the accuracy of generated tags.

[0004] To achieve the above-mentioned purpose, an embodiment of the present application provides a method for generating a video ringback tone tag, the method comprising: Obtain multimodal video ringback tone data; Preprocessing the multimodal video ringback tone data to obtain preprocessed data; Performing spatiotemporal feature extraction processing on the pre-processed data to obtain video spatiotemporal features; Performing multimodal feature fusion processing on the video spatiotemporal features to obtain multimodal features; A dynamic semantic graph is constructed, and label query processing is performed on the multimodal features according to the dynamic semantic graph to obtain the video ringback tone label.

[0005] In some embodiments, the multimodal video ringback tone data includes a video stream, an audio stream, and metadata, and preprocessing the multimodal video ringback tone data to obtain preprocessed data includes the following steps: Performing spatiotemporal slicing and dynamic background suppression processing on the video stream to obtain a preprocessed image; Performing acoustic feature extraction and speech recognition processing on the audio stream to obtain speech text; Data alignment and packaging processing are performed on the preprocessed image and the speech text according to the metadata to obtain the preprocessed data.

[0006] In some embodiments, performing spatiotemporal slicing and dynamic background suppression on the video stream to obtain a preprocessed image comprises the following steps: Decoding and format conversion are performed on the video stream to obtain a converted video; Performing a slicing segmentation process on the converted video according to a sliding step size to obtain segmented slices; Performing fast Fourier transform processing on the segmented slices to obtain a spectrum diagram; Performing inverse fast Fourier transform processing on the spectrum image according to a low-pass filter to obtain a reconstructed image; The reconstructed image is input into a generative adversarial network for denoising to obtain the preprocessed image.

[0007] In some embodiments, performing spatiotemporal feature extraction processing on the pre-processed data to obtain video spatiotemporal features includes the following steps: Performing image decoding processing on the preprocessed data to obtain a preprocessed image; Performing displacement compensation processing on the pre-processed image according to an optical flow field algorithm to obtain a compensated image; The compensated image is subjected to spatiotemporal feature fusion processing through a three-dimensional convolutional network to obtain the video spatiotemporal features.

[0008] In some embodiments, performing displacement compensation on the pre-processed image according to the optical flow algorithm to obtain a compensated image includes the following steps: Performing optical flow calculation on the preprocessed image to obtain an optical flow field matrix; Performing text location processing on the pre-processed image using a text detection model to obtain text area coordinates; Affine transformation is performed on the optical flow field matrix according to the coordinates of the text area to obtain the compensated image.

[0009] In some embodiments, performing multimodal feature fusion processing on the video spatiotemporal features to obtain multimodal features includes the following steps: Performing text decoding processing on the pre-processed data to obtain speech text; Performing feature mapping processing on the speech text and the video spatiotemporal features to obtain text features and visual features; Performing cross-modal alignment processing on the text features and the visual features according to a multi-head attention mechanism to obtain associated features; The associated features are subjected to semantic consistency verification processing and residual enhancement output processing to obtain the multimodal features.

[0010] In some embodiments, constructing a dynamic semantic graph includes the following steps: Acquiring multi-source data, including behavioral data, Internet buzzwords, and dialect data; Performing semantic relationship analysis on the Internet buzzwords and the dialect data according to the graph structure to obtain an initial graph; Performing node update processing on the initial graph according to the behavior data to obtain an updated graph; The updated graph is processed to eliminate semantic conflicts and construct the dynamic semantic graph.

[0011] To achieve the above-mentioned purpose, another aspect of the present application provides a video ringback tone tag generation system, the system comprising: The first module is used to obtain multimodal video ringback tone data; The second module is used to preprocess the multimodal video ringback tone data to obtain preprocessed data; The third module is used to perform spatiotemporal feature extraction processing on the pre-processed data to obtain video spatiotemporal features; The fourth module is used to perform multimodal feature fusion processing on the video spatiotemporal features to obtain multimodal features; The fifth module is used to construct a dynamic semantic graph, perform label query processing on the multimodal features according to the dynamic semantic graph, and obtain the video ringtone label.

[0012] To achieve the above-mentioned purpose, another aspect of an embodiment of the present application provides an electronic device, which includes a memory and a processor, wherein the memory stores a computer program, and the processor implements the above-mentioned method when executing the computer program.

[0013] To achieve the above objectives, another aspect of an embodiment of the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method described above is implemented.

[0014] Embodiments of the present application include at least the following beneficial effects: This application provides a method, system, device, and medium for generating video ringback tone tags. This solution improves recognition accuracy in complex scenarios by acquiring multimodal video ringback tone data, preprocessing the multimodal video ringback tone data to obtain preprocessed data, extracting spatiotemporal features from the preprocessed data to obtain video spatiotemporal features, and then performing multimodal feature fusion on the video spatiotemporal features to obtain multimodal features. Furthermore, this solution improves the consistency between tags and content semantics by constructing a dynamic semantic graph and performing tag query processing on the multimodal features based on the dynamic semantic graph to obtain video ringback tone tags. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] Figure 1This is a flow chart of a method for generating a video ringback tone tag provided by an embodiment of the present application; Figure 2 yes Figure 1 Flowchart of step S102 in FIG. Figure 3 yes Figure 1 Flowchart of step S103 in FIG. Figure 4 yes Figure 1 Flowchart of step S104 in FIG. Figure 5 This is a multimodal feature fusion flow chart provided in an embodiment of the present application; Figure 6 yes Figure 1 Flowchart of step S105 in FIG. Figure 7 This is a system architecture flow chart provided by an embodiment of the present application; Figure 8 This is a structural diagram of a video ringback tone label generation system provided by an embodiment of the present application; Figure 9 This is a schematic diagram of the hardware structure of the electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0016] In order to make the purpose, technical solutions and advantages of the present application clearer, the present application is further described in detail below with reference to the accompanying drawings and examples. It should be understood that the specific embodiments described herein are only used to explain the present application and are not intended to limit the present application. When the following description refers to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the embodiments of the present application. They are merely examples of systems and methods consistent with some aspects of the embodiments of the present application as detailed in the appended claims.

[0017] It will be understood that the terms "first", "second", etc. used in this application may be used herein to describe various concepts, but unless otherwise specified, these concepts are not limited by these terms. These terms are only used to distinguish one concept from another. For example, without departing from the scope of the embodiments of the present application, the first information may also be referred to as the second information, and similarly, the second information may also be referred to as the first information. Depending on the context, the words "if" and "if" as used herein may be interpreted as "at the time of" or "when" or "in response to determining".

[0018] The terms "at least one", "plurality", "each", "any", etc. used in this application include "at least one", "two" or more, "plurality" or "each", "any" or "any one", "each" or "any one" as used herein.

[0019] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by those skilled in the art to which this application pertains. The terms used herein are for the purpose of describing the embodiments of this application only and are not intended to limit this application.

[0020] Before explaining the embodiments of the present application in detail, some of the nouns and terms involved in the embodiments of the present application are first explained. The nouns and terms involved in the embodiments of the present application are subject to the following explanations.

[0021] Multimodal feature fusion: Improve the accuracy of semantic understanding by integrating features from multiple data modalities such as video, audio, and text.

[0022] Dynamic semantic graph: A semantic network structure that is updated in real time, containing entities, phrases, and their relationships, supporting dynamic evolution of context.

[0023] Edge computing optimization architecture: A distributed computing framework that combines terminal devices, edge nodes, and the cloud to achieve low-latency and high-efficiency data processing.

[0024] With the popularization of 5G communication technology and the explosive growth of video ringback tone services, users are increasingly demanding personalized and scenario-based video ringback tone content. In related technologies, there is a method of generating video ringback tone labels by text recognition, which generates labels by performing text recognition on the video subtitles of the video ringback tone. However, in actual applications, it has been found that related text recognition technologies are insufficient in handling interference from complex background videos, resulting in low recognition accuracy. At the same time, the independent processing of audio and video features in related technologies leads to a high mismatch rate between labels and content semantics, resulting in low accuracy of generated labels, which affects user experience and the efficiency of content recommendation. In summary, the technical problems existing in related technologies need to be improved.

[0025] For example, relevant text recognition technologies are inadequately able to handle interference from complex background videos (such as dynamic effects and gradient colors), resulting in an industry-average subtitle recognition accuracy rate of less than 60%. Furthermore, these technologies independently process audio and video features, resulting in a high rate of mismatch between labels and content semantics. For example, if the video image is "Spring Festival Reunion" and the audio text is "Promotional Event," the generated label will be "Holiday Promotion." Furthermore, these technologies rely on manually maintained tag libraries with long update cycles, making them unable to adapt to the rapid evolution of internet slang and emerging buzzwords, and providing insufficient support for dialects and niche languages. Furthermore, differences in device performance lead to a fragmented experience. The decoding delay between flagship phones and low-end models can vary by 400-800ms, resulting in asynchronous tag generation and a poorly consistent user experience.

[0026] In light of this, embodiments of the present application provide a method, system, device, and medium for generating video ringback tone tags. This solution obtains multimodal video ringback tone data, preprocesses the multimodal video ringback tone data to obtain preprocessed data, extracts spatiotemporal features from the preprocessed data to obtain video spatiotemporal features, and then fuses the spatiotemporal features to obtain multimodal features. This method improves recognition accuracy in complex scenarios. Furthermore, this solution constructs a dynamic semantic graph and performs tag query processing on the multimodal features based on the dynamic semantic graph to obtain video ringback tone tags, thereby improving the consistency between tags and content semantics.

[0027] The embodiment of the present application provides a method for generating a video ringback tone label, which relates to the field of video processing technology. The embodiment of the present application provides a method for generating a video ringback tone label that can be applied to a terminal, a server, or software running in a terminal or a server. In some embodiments, the terminal can be a smart phone, a tablet computer, a laptop computer, a desktop computer, a smart speaker, a smart watch, and a car terminal, etc., but is not limited to this; the server side can be configured as an independent physical server, or as a server cluster or distributed system composed of multiple physical servers, or as a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms. The server can also be a node server in a blockchain network; the software can be an application that implements a method for generating a video ringback tone label, etc., but is not limited to the above forms.

[0028] The present application can be used in many general or special computer system environments or configurations. For example: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, and the like. The present application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, and the like that perform specific tasks or implement specific abstract data types. The present application can also be practiced in distributed computing environments in which tasks are performed by remote processing devices connected via a communication network. In a distributed computing environment, program modules can be located in local and remote computer storage media, including storage devices.

[0029] It should be noted that in each specific embodiment of the present application, when it comes to the need to perform relevant processing based on data related to the user's identity or characteristics, such as user information, user behavior data, user historical data, and user location information, the user's permission or consent will be obtained first, and the collection, use, and processing of such data will comply with relevant laws, regulations, and standards. In addition, when the embodiment of the present application needs to obtain the user's sensitive personal information, the user's separate permission or consent will be obtained through a pop-up window or by jumping to a confirmation page. After clearly obtaining the user's separate permission or consent, the necessary user-related data for the normal operation of the embodiment of the present application will be obtained.

[0030] Figure 1 This is an optional flowchart of a method for generating a video ringback tone tag provided in an embodiment of the present application. Figure 1 The method may include but is not limited to steps S101 to S105.

[0031] Step S101, obtaining multimodal video ringback tone data; Step S102, preprocessing the multimodal video ringback tone data to obtain preprocessed data; Step S103, performing spatiotemporal feature extraction processing on the pre-processed data to obtain video spatiotemporal features; Step S104, performing multimodal feature fusion processing on the video spatiotemporal features to obtain multimodal features; Step S105: construct a dynamic semantic graph, perform label query processing on the multimodal features according to the dynamic semantic graph, and obtain the video ringback tone label.

[0032] In the embodiment of the present application, steps S101 to S105 are performed by obtaining multimodal video ringback ring data and performing label generation to obtain a video ringback ring label, wherein the multimodal video ringback ring data includes video stream data, audio stream data and metadata of the video ringback ring. The video ringback ring label is a label related to the corresponding video ringback ring. For example, if the video screen in the video ringback ring is "Spring Festival Reunion", the video ringback ring label generation method can generate corresponding video ringback ring labels such as "Festival", "Spring Festival", "Family", etc. In order to improve the processing efficiency of the data, the embodiment of the present application also pre-processes the multimodal video ringback ring data, for example, by performing pre-processing such as decoding and format conversion on the video stream data, and performing pre-processing such as decoding and resampling on the audio stream to obtain pre-processed data. Then, the pre-processed data is subjected to spatiotemporal feature extraction processing to obtain video spatiotemporal features and the video spatiotemporal features are subjected to multimodal feature fusion processing. By fusing the speech and text features in the pre-processed data with the video spatiotemporal features, the visual, audio and text features can be dynamically associated through the multi-head attention mechanism to eliminate semantic mismatch. Furthermore, by constructing a dynamic semantic graph that is updated in real time, label query processing is performed on multimodal features based on the dynamic semantic graph to obtain video ringback tone labels, which can improve the consistency between labels and content semantics. It should be noted that the embodiments of this application can also design an edge computing optimization architecture, through an end-edge-cloud hierarchical processing architecture, hierarchical processing to reduce latency and adapt to the performance of different devices.

[0033] One of the above technical solutions has the following advantages or beneficial effects: This embodiment of the application improves the accuracy of subtitle tag recognition in complex scenarios through multimodal feature fusion. Furthermore, by constructing a dynamic semantic graph, it can accurately identify video ringtones, improving the consistency between tags and content semantics, thereby improving user experience and the accuracy of content recommendations.

[0034] In step S101 of some embodiments, the multimodal video ringback tone data may be obtained through a video ringback tone library, or may be obtained through user upload, but is not limited thereto.

[0035] See also Figure 2 In step S102 of some embodiments, the multimodal video ringback tone data includes a video stream, an audio stream, and metadata, and preprocessing the multimodal video ringback tone data to obtain preprocessed data includes the following steps: Step S201, performing spatiotemporal slicing and dynamic background suppression processing on the video stream to obtain a preprocessed image; Step S202: performing acoustic feature extraction and speech recognition processing on the audio stream to obtain speech text; Step S203 , performing data alignment and packaging processing on the pre-processed image and the speech text according to the metadata to obtain the pre-processed data.

[0036] In this embodiment of the present application, multimodal video ringback tone data includes a video stream, an audio stream, and metadata for the ringback tone to be labeled. This embodiment uses a video processing tool to decode the video stream and perform color space conversion, followed by spatiotemporal slicing and dynamic background suppression. Specifically, a segmentation strategy is used to generate spatiotemporal slices, and background noise suppression is performed on the slices to obtain a preprocessed image. This embodiment performs acoustic feature extraction and speech recognition on the audio stream. Specifically, the video processing tool decodes the audio stream into PCM format, resamples it to 16kHz, and then frames it into 25ms / frames with a step size of 10ms, outputting an audio frame sequence. Acoustic feature extraction is performed on the audio frame sequence, and 39-dimensional Mel-Frequency Cepstral Coefficient (MFCC) features are extracted using an audio processing library. The audio processing library parameters are set as follows: number of filter banks = 40, fast Fourier transform (FFT) window length = 400 (corresponding to 25ms), and frequency band range: 0-8kHz. Silence detection and filtering are also performed, marking frames with less than 20% of the average energy as silence and discarding areas with continuous silence frames >200ms. For speech recognition processing, a pre-trained speech recognition model can be selected. By inputting an MFCC feature sequence, the output is a text segment and its timestamp, which is then processed based on the confidence level, retaining only text segments with a confidence level ≥0.8. If a single audio recognition result contains contradictory semantics, a multimodal verification process is triggered. This embodiment of the present application also performs data alignment and packaging on the pre-processed image and speech text based on metadata to obtain pre-processed data. The video and audio streams are aligned based on the NTP protocol synchronization device clock information based on the metadata timestamps, and soft alignment compensation is performed. Specifically, the video keyframe timestamps and the audio recognition text timestamps are extracted for precise matching. If the video frame timestamp overlaps with the audio word timestamp by ≥80%, a direct correlation is established. When the error is >50ms, the dynamic time warping (DTW) distance between the video frame and the audio segment is calculated, allowing a maximum offset compensation of ±100ms. When the temporal overlap between the video OCR text and the ASR text is less than 50%, the video text is prioritized and a semantic mismatch log is recorded for subsequent model optimization. Finally, the cross-modal data is packaged and processed. In one feasible embodiment, the output data structure is as follows: { "video_frames": [frame_1, ..., frame_16], # Preprocessed RGB frames "audio_features": [mfcc_1, ..., mfcc_N], # MFCC feature sequence "asr_text": [ {"text": "Yangtze River Bridge", "start": 120ms, "end": 680ms, "conf": 0.85}, # Other text snippets... ], "metadata": { "resolution": "1080P", "device_id": "XYZ123", "timestamp": 1630000000123 } }; One of the above technical solutions has the following advantages or beneficial effects: the embodiment of the present application can improve data processing efficiency by preprocessing multimodal data.

[0037] In some embodiments, performing spatiotemporal slicing and dynamic background suppression on the video stream to obtain a preprocessed image comprises the following steps: Decoding and format conversion are performed on the video stream to obtain a converted video; Performing a slicing segmentation process on the converted video according to a sliding step size to obtain segmented slices; Performing fast Fourier transform processing on the segmented slices to obtain a spectrum diagram; Performing inverse fast Fourier transform processing on the spectrum image according to a low-pass filter to obtain a reconstructed image; The reconstructed image is input into a generative adversarial network for denoising to obtain the preprocessed image.

[0038] In an embodiment of the present application, a video stream is decoded and color-space converted using a video processing tool to obtain a converted video. The converted video is then segmented according to a preset segmentation strategy, such as a fixed 16-frame / unit (640ms) with a sliding step of 8 frames (50% overlap), to obtain segmented slices. The slice size is 224×224×3 (H×W×C), and black frames are added if the number of frames is less than 16. Each segmented slice is then fast Fourier transformed to obtain a spectrogram. The spectrogram is then inversely fast Fourier transformed using a low-pass filter to obtain a reconstructed image. The low-pass filter parameters are set to a cutoff frequency of 0.3Hz and an order of 4. Finally, the reconstructed image is input into a generative adversarial network for denoising. The training process of the generative adversarial network involves inputting a noisy frame (an RGB image with particle effects superimposed) into the generator and outputting a denoised text region mask. The generator and discriminator are alternately trained to maximize the accuracy of real / generated image classification by minimizing the L1 loss and the adversarial loss. Upon completion of the training, the generative adversarial network is obtained.

[0039] One of the above technical solutions has the following advantages or beneficial effects: the embodiment of the present application can perform dynamic background suppression on videos with complex backgrounds by preprocessing the video stream, thereby improving the accuracy of video processing.

[0040] See also Figure 3 In step S103 of some embodiments, performing spatiotemporal feature extraction on the pre-processed data to obtain spatiotemporal features of the video includes the following steps: Step S301, performing image decoding processing on the pre-processed data to obtain a pre-processed image; Step S302, performing displacement compensation processing on the pre-processed image according to an optical flow algorithm to obtain a compensated image; Step S303: Perform spatiotemporal feature fusion processing on the compensated image through a three-dimensional convolutional network to obtain the video spatiotemporal features.

[0041] In an embodiment of the present application, preprocessed data is decoded to extract a preprocessed image. Displacement compensation is then performed on the pixels in the text area of the preprocessed image using an optical flow algorithm combined with a text detection model to obtain a compensated image. The compensated image is then subjected to spatiotemporal feature fusion processing using a three-dimensional convolutional network to obtain video spatiotemporal features. The present embodiment of the present application makes corresponding improvements to the three-dimensional convolutional network by adding a spatial attention module after the convolution layer, calculating text area features using spatial attention weights, performing discrete cosine transform (DCT) processing on the feature map, and performing inverse DCT reconstruction after replacing it with a component >0.3Hz. This fusion of features yields video spatiotemporal features.

[0042] One of the above technical solutions has the following advantages or beneficial effects: the embodiment of the present application can extract the spatiotemporal features in the video ringback tone by performing spatiotemporal feature extraction on the preprocessed data, and can more accurately identify the visual features of the video ringback tone, thereby improving the accuracy of generated labels.

[0043] In some embodiments, performing displacement compensation on the pre-processed image according to the optical flow algorithm to obtain a compensated image includes the following steps: Performing optical flow calculation on the preprocessed image to obtain an optical flow field matrix; Performing text location processing on the pre-processed image using a text detection model to obtain text area coordinates; Affine transformation is performed on the optical flow field matrix according to the coordinates of the text area to obtain the compensated image.

[0044] In this embodiment of the present application, two adjacent preprocessed images are obtained from preprocessed data, and optical flow calculation is performed on the preprocessed images according to an optical flow algorithm to obtain an optical flow matrix. The parameters of the optical flow algorithm are set to 3 pyramid levels, a window size of 15×15 per level, and a Gaussian kernel σ of 1.5. The coordinates of the text area are then located using a text detection model. When the average displacement within the area is ≥5 pixels and the consistency of the displacement direction is >80% (excluding random noise), a compensation condition is triggered to apply an inverse displacement matrix to the pixels in the text area for an affine transformation to obtain a compensated image.

[0045] One of the above technical solutions has the following advantages or beneficial effects: The embodiment of the present application uses a spatiotemporal feature extraction method that combines the optical flow method with a three-dimensional convolutional network to solve the problem of text missed detection caused by dynamic background interference.

[0046] See also Figure 4 In step S104 of some embodiments, performing multimodal feature fusion processing on the spatiotemporal features of the video to obtain multimodal features includes the following steps: Step S401, performing text decoding processing on the pre-processed data to obtain speech text; Step S402: performing feature mapping processing on the speech text and the spatiotemporal features of the video to obtain text features and visual features; Step S403: performing cross-modal alignment processing on the text features and the visual features according to a multi-head attention mechanism to obtain associated features; Step S404: performing semantic consistency verification processing and residual enhancement output processing on the associated features to obtain the multimodal features.

[0047] In the embodiment of the present application, the pre-processed data is subjected to text decoding and extraction to obtain speech text, which is text data obtained by performing speech recognition on the audio data. The embodiment of the present application performs feature mapping processing on the speech text and video spatiotemporal features to obtain text features and visual features, and then uses a multi-head attention mechanism to align the text features with the visual features. The formula of the multi-head attention mechanism is: Attention(Q, K, V) = softmax(QK^T / √d_k) V; Among them, Q is the text feature, K / V is the visual feature, the dimension d_k=512, and the text feature (Q) is dynamically associated with the visual feature (K / V). The embodiment of the present application also performs a semantic consistency check on the associated features, calculates the cross-modal confidence, and combines it with the dynamic threshold. When the cross-modal confidence is less than the dynamic threshold, the correction strategy is triggered. Among them, the correction strategy is to extract video text through text recognition technology, calculate the similarity between video text and voice text, and regenerate text features when the similarity is greater than 0.6, otherwise it is marked as "cross-modal conflict", and manual detection and other processing processes can be performed later. In addition, the embodiment of the present application also sets a residual connection, and performs residual calculation on the associated features to obtain multimodal features. Please refer to Figure 5 In the embodiment of the present application, a video stream and an audio stream are extracted, video features are extracted from the video stream through a three-dimensional convolutional network (3D-CNN), text features are generated from the audio stream through automatic speech recognition (ASR), and finally, based on multi-head attention alignment, label matching is performed through a dynamic semantic graph to generate a video ringback tone label.

[0048] One of the above technical solutions has the following advantages or beneficial effects: The embodiment of the present application uses a dynamic alignment method of a multi-head attention mechanism to non-linearly associate video spatiotemporal features and speech text features, which can reduce the semantic mismatch problem of unimodal analysis.

[0049] See also Figure 6 In step S105 of some embodiments, constructing a dynamic semantic graph includes the following steps: Step S601: Acquire multi-source data, where the multi-source data includes behavioral data, Internet buzzwords, and dialect data; Step S602: performing semantic relationship analysis on the Internet buzzwords and the dialect data according to the graph structure to obtain an initial graph; Step S603, performing node update processing on the initial graph according to the behavior data to obtain an updated graph; Step S604: performing semantic conflict elimination processing on the updated graph to construct the dynamic semantic graph.

[0050] In this embodiment of the present application, by acquiring multi-source data, the present application embodiment can collect click / share logs once an hour through Kafka real-time stream to obtain behavioral data, obtain online hot words by calling the entire network search trend API every 10 minutes, and obtain dialect data by synchronizing minority language dictionaries every morning. Then, semantic relationship analysis is performed on online hot words and dialect data based on the graph structure, where the graph structure is as follows: G = (E, R), E = {entity|phrase|network term}, R = {synonym|related|antonym}; The embodiment of this application also updates semantic nodes daily based on user behavior feedback and filters illegal content in real time based on regular expressions and deep learning models (BiLSTM+CRF). The update weight ΔW = 0.6*(search volume / maximum search volume) + 0.4*(user feedback score). Before weight calculation, the search volume and user score must be normalized to ensure the weight range. The update formula for the semantic node is as follows: W_{new} = 0.7*W_old + 0.3*ΔW; Where W_{new} represents the semantic node in the updated graph, W_old represents the semantic node in the initial graph, and ΔW represents the weight. Finally, semantic conflict resolution is performed on the updated graph. Two candidate entities A and B are input. The cosine similarity of their word vectors is calculated using the RoBERTa model. The number of co-occurrences in the same video is counted. Words with a confidence level of 0.7-0.85 are manually reviewed by a manual review queue. When nodes have both synonymous and antonymous relationships, high-frequency relationships (>60%) from the last seven days are prioritized. Adversarial training verification is initiated: adversarial examples are generated and associated words are replaced. If the confidence level of the classifier (BiLSTM) drops by >30%, the conflicting relationship is deemed invalid.

[0051] One of the above technical solutions has the following advantages or beneficial effects: The embodiment of the present application supports the dynamic evolution of dialects and Internet terms by constructing a real-time updated semantic graph, breaking through the update cycle limitations of the static tag library.

[0052] The following is a detailed description of the embodiments of the present application with reference to specific application examples: The embodiments of this application can be applied to the field of video processing technology and can be widely used in scenarios where video tags are generated. By designing a three-tiered processing model of end-edge-cloud and combining it with a dynamic resource allocation algorithm, it can adapt to different device performances. This embodiment of the application can improve the core aspects of user experience and accurate content recommendation. Its application scenarios include personalized tag generation. In one feasible embodiment, this embodiment quantitatively evaluates device performance using a computing capability index (CI) = 0.4*CPU frequency + 0.6*GPU TFLOPS and a memory score (MI) = log2(available memory / 500MB). Devices are graded based on the calculated capability index: high-end devices: CI ≥ 8.0 and MI ≥ 3.0; mid-range devices: 5.0 ≤ CI < 8.0 and 2.0 ≤ MI < 3.0; and low-end devices: CI < 5.0 or MI < 2.0. This tiered processing then involves lightweight pre-processing and real-time response on the end-side, including video decoding, basic optical flow compensation, low-resolution spatiotemporal slicing (480P), and device performance awareness. Multimodal feature fusion and semantic screening are performed at the edge nodes, along with spatiotemporal feature compression, cross-modal alignment (lightweight version), low-confidence label filtering, and dynamic resource allocation decisions. Global semantic understanding and graph optimization are performed in the cloud, along with 3D-CNN deep feature extraction, dynamic semantic graph updates, conflict resolution, and hot word weight calculation. The embodiments of this application also dynamically downgrade end-side tasks, such as dynamically adjusting and allocating computing tasks in real time based on device performance (computing power index, memory score), further reducing latency on low-end models. The dynamic downgrade strategy is as follows: { if device level == "low-end": Enable frame skipping (process one frame for every two frames, reducing computational effort by 50%) The resolution is forced down to 480P (memory usage is reduced by 70%) Basic optical flow compensation module elif device level == "mid-range": Enable half-precision inference (FP16, 40% reduction in memory usage) The optical flow compensation threshold is relaxed to ±8 pixels (the computational complexity is reduced by 30%). else: Full-featured mode (including GAN denoising and high-frequency feature retention) }; Dynamic resource allocation is then performed, and the CPU utilization and remaining memory on the end side, the number of queued tasks on the edge node, the GPU memory occupancy rate, the network throughput, the cluster load rate, storage IOPS, and API response latency of the cloud are perceived in real time through resource status. Communication detection is performed between the end side and the edge: once per second (heartbeat packet + performance data), and from the edge to the cloud: the regional status is summarized every 5 seconds. The embodiment of the present application also sets a task migration decision model, by setting the migration trigger condition (any one of which is met is triggered) as follows: Edge node latency > 200ms for 10 seconds; The client-side CPU utilization is >90%; The cloud cluster load rate is <50% (can take over additional tasks); If the edge latency is 300ms (threshold 200ms) and the idle cloud resources account for 60%, then the migration ratio is 0.5, which means that 50% of the tasks are migrated to the cloud. The process controller logic is as follows: { while True: Real-time status of the collection end, edge, and cloud If the edge load is light and the end-side device is a high-end device: Enable "device-edge" collaboration mode (preprocessing and feature extraction are performed on the device side, and only fusion is performed on the edge) elif edge overload: Enable "device-cloud" direct connection mode (only raw data is uploaded on the device side, and the cloud processes the entire process) else: Default three-stage pipeline mode Re-evaluate decisions every 30 seconds }; See also Figure 7In this embodiment, a terminal device inputs device information, including a 1080P video stream with dynamic special effects (gradient background color and scrolling text). Additional information includes: 1920×1080 resolution, 5Mbps bitrate, and 16kHz audio stream sampling rate. The video ringtone playback screen displays the special effects text "bridge," "building," and "TV tower" in sequence, with a colorful dynamic city skyline background. Multiple edge nodes then process the input multimodal data. Submodule 1: On-device processing (MobileNetV3 dynamic downscaling) provides lightweight spatiotemporal slicing and optical flow compensation. The spatiotemporal slicing input is a downsampled 480P video stream (854×480 resolution), frame skipping (16 frames / cell → 8 frames / cell), and a segmentation strategy of 8 frames / cell (320ms) with no overlap. The resulting spatiotemporal matrix (dimensions 8×224×224×3) is then generated. Basic optical flow compensation is then performed. Motion vectors of adjacent frames are calculated using the Farneback algorithm with a 15×15 window size and three iterations, compensating for offsets of ±5 pixels. Text region displacement is detected (coordinate range x:100-900, y:200-800), with a compensation threshold of ±5 pixels. Finally, the FFT cutoff frequency is set to 0.3Hz, and special effects are suppressed, eliminating high-frequency noise from particle effects (improving PSNR by 6.2dB). The compensated spatiotemporal feature block (dimension 256×16) is output. Sub-module 2 performs edge processing, achieving preliminary feature fusion and anomaly filtering. Specifically, the input is the spatiotemporal features uploaded from the client (dimension 256×8). PCA dimensionality reduction (256→128) is used to reduce memory usage. Low-confidence features (threshold < 0.7) are filtered based on anomalies to reduce the computing load on the cloud. The resulting compressed feature matrix (dimension 128×16) is output. Finally, feature extraction is performed on a cloud server. A ResNet3D-50 model is used to extract deep spatiotemporal features (output dimension 512×16). Frequency domain filtering (FFT cutoff frequency 0.3Hz) is then performed to remove high-frequency noise. The audio ASR text and video features are then aligned using a multi-head attention mechanism (8 heads). The alignment error is controlled within ±50ms and the semantic mismatch rate is less than 8%. The fused multimodal features (dimension 512) are output. Finally, a dynamic semantic graph is searched. The resulting graph is shown below: { Entity node: Bridge (Type: Landmark, Attributes: Transportation Hub, Historical Building).

[0053] Building (Type: Landmark, Attributes: Historical Building, Cultural Symbol).

[0054] TV Tower (Type: Landmark, Attributes: Modern Architecture, City Symbol).

[0055] Related Relationships: Bridge--Same Category-->Landmark Building--Cultural Relations-->Historic District }; The query logic is then set to a search depth of 3 and a confidence threshold of ≥ 0.85. The associated nodes are expanded: "Historical Scenic Spots," "City Tourism," and "Transportation Hub." The output is a list of candidate tags: ["city landmark," "historical building," "city tourism," "transportation hub"]. Sorted by semantic relevance (BERTScore) and real-time popularity (number of searches within 24 hours), the output is tag 1: "city landmark" (confidence 0.92), tag 2: "historical building" (confidence 0.89), and tag 3: "city tourism" (confidence 0.85).

[0056] See also Figure 8 The embodiment of the present application further provides a video ring back tone tag generation system, which can implement the above-mentioned video ring back tone tag generation method, and the system includes: The first module 801 is used to obtain multimodal video ringback tone data; The second module 802 is configured to pre-process the multimodal video ringback tone data to obtain pre-processed data; The third module 803 is used to perform spatiotemporal feature extraction processing on the pre-processed data to obtain video spatiotemporal features; The fourth module 804 is configured to perform multimodal feature fusion processing on the video spatiotemporal features to obtain multimodal features; The fifth module 805 is used to construct a dynamic semantic graph, perform label query processing on the multimodal features according to the dynamic semantic graph, and obtain the video ringback tone label.

[0057] It can be understood that the contents of the above method embodiments are all applicable to the present system embodiments, the functions specifically implemented by the present system embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0058] The present application also provides an electronic device comprising a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the above-mentioned method for generating a video ringback tone tag. The electronic device can be any smart terminal including a tablet computer, an in-vehicle computer, or the like.

[0059] It can be understood that the contents of the above method embodiments are applicable to the present device embodiments, the functions specifically implemented by the present device embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0060] See also Figure 9 , Figure 9 The hardware structure of an electronic device according to another embodiment is shown. The electronic device includes: The processor 901 can be implemented as a general-purpose CPU (Central Processing Unit), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of the present application. The memory 902 can be implemented in the form of a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 902 can store an operating system and other application programs. When the technical solutions provided in the embodiments of this specification are implemented through software or firmware, the relevant program code is stored in the memory 902 and is called by the processor 901 to execute the video ringback tone label generation method of the embodiments of this application. Input / output interface 903, used to implement information input and output; Communication interface 904, used to implement communication interaction between this device and other devices, which can be achieved through wired means (such as USB, network cable, etc.) or wireless means (such as mobile network, WiFi, Bluetooth, etc.); Bus 905 , which transmits information between various components of the device (e.g., processor 901 , memory 902 , input / output interface 903 , and communication interface 904 ); The processor 901 , the memory 902 , the input / output interface 903 and the communication interface 904 are connected to each other in communication within the device via a bus 905 .

[0061] An embodiment of the present application further provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, the above-mentioned video ringback tone tag generation method is implemented.

[0062] It can be understood that the contents of the above method embodiments are all applicable to the present storage medium embodiment, the functions specifically implemented by the present storage medium embodiment are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0063] The memory, as a non-transient computer-readable storage medium, can be used to store non-transient software programs and non-transient computer executable programs. In addition, the memory may include a high-speed random access memory and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some embodiments, the memory may optionally include a memory remotely arranged relative to the processor, and these remote memories may be connected to the processor via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0064] Embodiments of the present application provide a method, system, device, and medium for generating video ringback tone tags. This solution obtains multimodal video ringback tone data, preprocesses the multimodal video ringback tone data to obtain preprocessed data, extracts spatiotemporal features from the preprocessed data to obtain video spatiotemporal features, and then fuses the spatiotemporal features to obtain multimodal features. This method improves recognition accuracy in complex scenarios. Furthermore, this solution constructs a dynamic semantic graph and performs tag query processing on the multimodal features based on the dynamic semantic graph to obtain video ringback tone tags, thereby improving the consistency between tags and content semantics.

[0065] The embodiments described in the embodiments of this application are intended to more clearly illustrate the technical solutions of the embodiments of this application and do not constitute a limitation on the technical solutions provided by the embodiments of this application. Those skilled in the art will appreciate that with the evolution of technology and the emergence of new application scenarios, the technical solutions provided in the embodiments of this application are also applicable to similar technical problems.

[0066] Those skilled in the art will understand that the technical solutions shown in the figures do not constitute a limitation on the embodiments of the present application, and may include more or fewer steps than shown in the figures, or a combination of certain steps, or different steps.

[0067] The system embodiment described above is merely illustrative. The units described as separate components may or may not be physically separate, i.e., they may be located in one place or distributed across multiple network units. Some or all of the modules may be selected based on actual needs to achieve the objectives of this embodiment.

[0068] Those skilled in the art will appreciate that all or some of the steps in the methods, systems, and functional modules / units in the devices disclosed above may be implemented as software, firmware, hardware, or appropriate combinations thereof.

[0069] The terms "first", "second", "third", "fourth", etc. (if any) in the specification of the present application and the above-mentioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequential order. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0070] It should be understood that in this application, "at least one (item)" means one or more, and "plurality" means two or more. "And / or" is used to describe the association relationship of associated objects, indicating that three relationships may exist. For example, "A and / or B" can mean: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally indicates that the previous and next associated objects are in an "or" relationship. "At least one of the following items" or similar expressions refers to any combination of these items, including any combination of single items or plural items. For example, at least one of a, b or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple.

[0071] In the several embodiments provided in this application, it should be understood that the disclosed systems and methods can be implemented in other ways. For example, the system embodiments described above are merely illustrative. For example, the division of the above units is merely a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of systems or units, which can be electrical, mechanical or other forms.

[0072] The units described above as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0073] In addition, the functional units in the various embodiments of the present application may be integrated into a single processing unit, or each unit may exist physically separately, or two or more units may be integrated into a single unit. The aforementioned integrated units may be implemented in the form of hardware or software functional units.

[0074] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, which is stored in a storage medium and includes multiple instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of various embodiments of the present application. The aforementioned storage medium includes: various media that can store programs, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0075] The preferred embodiments of the present invention are described above with reference to the accompanying drawings, but are not intended to limit the scope of the present invention. Any modifications, equivalent substitutions, and improvements made by those skilled in the art without departing from the scope and essence of the present invention should be within the scope of the present invention.

Claims

1. A method for generating a video ring back tone tag, characterized in that: The method comprises the following steps: Obtain multimodal video ringback tone data; Preprocessing the multimodal video ringback tone data to obtain preprocessed data; Performing spatiotemporal feature extraction processing on the pre-processed data to obtain video spatiotemporal features; Performing multimodal feature fusion processing on the video spatiotemporal features to obtain multimodal features; A dynamic semantic graph is constructed, and label query processing is performed on the multimodal features according to the dynamic semantic graph to obtain the video ringback tone label.

2. The method according to claim 1, characterized in that The multimodal video ringback ring data includes a video stream, an audio stream, and metadata. The preprocessing of the multimodal video ringback ring data to obtain preprocessed data includes the following steps: Performing spatiotemporal slicing and dynamic background suppression processing on the video stream to obtain a preprocessed image; Performing acoustic feature extraction and speech recognition processing on the audio stream to obtain speech text; Data alignment and packaging processing are performed on the preprocessed image and the speech text according to the metadata to obtain the preprocessed data.

3. The method according to claim 2, characterized in that The step of performing spatiotemporal slicing and dynamic background suppression processing on the video stream to obtain a preprocessed image comprises the following steps: Decoding and format conversion are performed on the video stream to obtain a converted video; Performing a slicing segmentation process on the converted video according to a sliding step size to obtain segmented slices; Performing fast Fourier transform processing on the segmented slices to obtain a spectrum diagram; Performing inverse fast Fourier transform processing on the spectrum image according to a low-pass filter to obtain a reconstructed image; The reconstructed image is input into a generative adversarial network for denoising to obtain the preprocessed image.

4. The method according to claim 1, wherein The step of performing spatiotemporal feature extraction on the pre-processed data to obtain spatiotemporal features of the video comprises the following steps: Performing image decoding processing on the preprocessed data to obtain a preprocessed image; Performing displacement compensation processing on the pre-processed image according to an optical flow field algorithm to obtain a compensated image; The compensated image is subjected to spatiotemporal feature fusion processing through a three-dimensional convolutional network to obtain the video spatiotemporal features.

5. The method according to claim 4, characterized in that The method of performing displacement compensation on the pre-processed image according to the optical flow algorithm to obtain a compensated image includes the following steps: Performing optical flow calculation on the preprocessed image to obtain an optical flow field matrix; Performing text location processing on the pre-processed image using a text detection model to obtain text area coordinates; Affine transformation is performed on the optical flow field matrix according to the coordinates of the text area to obtain the compensated image.

6. The method according to claim 1, characterized in that The multimodal feature fusion processing is performed on the video spatiotemporal features to obtain multimodal features, comprising the following steps: Performing text decoding processing on the pre-processed data to obtain speech text; Performing feature mapping processing on the speech text and the video spatiotemporal features to obtain text features and visual features; Performing cross-modal alignment processing on the text features and the visual features according to a multi-head attention mechanism to obtain associated features; The associated features are subjected to semantic consistency verification processing and residual enhancement output processing to obtain the multimodal features.

7. The method according to any one of claims 1 to 6, characterized in that The construction of the dynamic semantic graph includes the following steps: Acquiring multi-source data, including behavioral data, Internet buzzwords, and dialect data; Performing semantic relationship analysis on the Internet buzzwords and the dialect data according to the graph structure to obtain an initial graph; Performing node update processing on the initial graph according to the behavior data to obtain an updated graph; The updated graph is processed to eliminate semantic conflicts and construct the dynamic semantic graph.

8. A video ring back tone label generation system, characterized in that: The system comprises: The first module is used to obtain multimodal video ringback tone data; The second module is used to preprocess the multimodal video ringback tone data to obtain preprocessed data; The third module is used to perform spatiotemporal feature extraction processing on the pre-processed data to obtain video spatiotemporal features; The fourth module is used to perform multimodal feature fusion processing on the video spatiotemporal features to obtain multimodal features; The fifth module is used to construct a dynamic semantic graph, perform label query processing on the multimodal features according to the dynamic semantic graph, and obtain the video ringtone label.

9. An electronic device, characterized in that: The electronic device includes a memory and a processor, the memory stores a computer program, and the processor implements the method according to any one of claims 1 to 7 when executing the computer program.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.

Images