Multimedia content processing method and device, computer equipment, storage medium and computer program product

Abstract

The invention relates to a multimedia content processing method and device, computer equipment, a storage medium and a computer program product. The method comprises the following steps: extracting a text feature vector of text metadata of multimedia content, and extracting an image feature vector of a target image included in the multimedia content; performing fusion processing on the text feature vector and the image feature vector to obtain a fusion feature vector, and determining a first confidence coefficient of the multimedia content according to the fusion feature vector; the first confidence is used for representing the possibility that the multimedia content belongs to the preset content type; inputting the text metadata and the target image into a trained semantic reasoning model to obtain a second confidence coefficient of the multimedia content; the second confidence is used for representing the possibility that the multimedia content belongs to the preset content type; and determining a target processing action on the multimedia content according to the first confidence coefficient and the second confidence coefficient. By adopting the method, the processing accuracy of the multimedia content can be improved.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and in particular to a multimedia content processing method, apparatus, computer equipment, computer-readable storage medium, and computer program product. Background Technology

[0002] With the rapid development of multimedia platforms, user-generated multimedia content (including audio, video, text, images, etc.) is experiencing explosive growth, making accurate processing of this content crucial. Traditional technologies typically employ unimodal analysis for multimedia content processing; however, this approach struggles to capture cross-modal correlation features, leading to errors and low processing accuracy. Summary of the Invention

[0003] Therefore, it is necessary to provide a multimedia content processing method, apparatus, computer equipment, computer-readable storage medium, and computer program product that can improve the accuracy of multimedia content processing in response to the above-mentioned technical problems.

[0004] Firstly, this application provides a multimedia content processing method, including:

[0005] Extract the text feature vector of the text metadata of the multimedia content, and extract the image feature vector of the target image included in the multimedia content;

[0006] The text feature vector and the image feature vector are fused to obtain a fused feature vector. A first confidence level of the multimedia content is determined based on the fused feature vector. The first confidence level is used to indicate the probability that the multimedia content belongs to a preset content type.

[0007] The text metadata and the target image are input into a trained semantic reasoning model to obtain a second confidence level of the multimedia content; the second confidence level is used to indicate the probability that the multimedia content belongs to the preset content type.

[0008] Based on the first confidence level and the second confidence level, the target processing action for the multimedia content is determined.

[0009] Secondly, this application also provides a multimedia content processing apparatus, comprising:

[0010] The vector extraction module is used to extract the text feature vector of the text metadata of the multimedia content, and to extract the image feature vector of the target image included in the multimedia content.

[0011] The first determining module is used to fuse the text feature vector and the image feature vector to obtain a fused feature vector, and to determine a first confidence level of the multimedia content based on the fused feature vector; the first confidence level is used to indicate the probability that the multimedia content belongs to a preset content type;

[0012] The second determining module is used to input the text metadata and the target image into a trained semantic reasoning model to obtain a second confidence level of the multimedia content; the second confidence level is used to indicate the probability that the multimedia content belongs to the preset content type.

[0013] The content processing module is used to determine the target processing action for the multimedia content based on the first confidence level and the second confidence level.

[0014] Thirdly, this application also provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to perform the following steps:

[0015] Extract the text feature vector of the text metadata of the multimedia content, and extract the image feature vector of the target image included in the multimedia content;

[0016] The text feature vector and the image feature vector are fused to obtain a fused feature vector. A first confidence level of the multimedia content is determined based on the fused feature vector. The first confidence level is used to indicate the probability that the multimedia content belongs to a preset content type.

[0017] The text metadata and the target image are input into a trained semantic reasoning model to obtain a second confidence level of the multimedia content; the second confidence level is used to indicate the probability that the multimedia content belongs to the preset content type.

[0018] Based on the first confidence level and the second confidence level, the target processing action for the multimedia content is determined.

[0019] Fourthly, this application also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, performs the following steps:

[0020] Extract the text feature vector of the text metadata of the multimedia content, and extract the image feature vector of the target image included in the multimedia content;

[0021] The text feature vector and the image feature vector are fused to obtain a fused feature vector. A first confidence level of the multimedia content is determined based on the fused feature vector. The first confidence level is used to indicate the probability that the multimedia content belongs to a preset content type.

[0022] The text metadata and the target image are input into a trained semantic reasoning model to obtain a second confidence level of the multimedia content; the second confidence level is used to indicate the probability that the multimedia content belongs to the preset content type.

[0023] Based on the first confidence level and the second confidence level, the target processing action for the multimedia content is determined.

[0024] Fifthly, this application also provides a computer program product, including a computer program that, when executed by a processor, performs the following steps:

[0025] Extract the text feature vector of the text metadata of the multimedia content, and extract the image feature vector of the target image included in the multimedia content;

[0026] The text feature vector and the image feature vector are fused to obtain a fused feature vector. A first confidence level of the multimedia content is determined based on the fused feature vector. The first confidence level is used to indicate the probability that the multimedia content belongs to a preset content type.

[0027] The text metadata and the target image are input into a trained semantic reasoning model to obtain a second confidence level of the multimedia content; the second confidence level is used to indicate the probability that the multimedia content belongs to the preset content type.

[0028] Based on the first confidence level and the second confidence level, the target processing action for the multimedia content is determined.

[0029] The aforementioned multimedia content processing methods, apparatuses, computer devices, storage media, and computer program products, by fusing the text feature vectors of the text metadata of the multimedia content and the image feature vectors of the target images in the multimedia content, can obtain a more comprehensive fused feature vector. Based on this fused feature vector, a first confidence level representing the multimedia content can be determined more accurately. Furthermore, by processing the text metadata and target images of the multimedia content using a trained semantic reasoning model, a second confidence level representing the multimedia content can be determined more accurately. Combining the first and second confidence levels, the target processing action corresponding to the multimedia content can be determined more accurately, allowing for more targeted processing of the multimedia content and improving processing accuracy. Moreover, the entire process employs multimodal analysis, avoiding the shortcomings of single-modal analysis, which struggles to capture cross-modal associations and is prone to errors leading to lower processing accuracy, further enhancing the overall accuracy of multimedia content processing. Attached Figure Description

[0030] To more clearly illustrate the technical solutions in the embodiments of this application or related technologies, the drawings used in the description of the embodiments of this application or related technologies will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0031] Figure 1 This is a flowchart illustrating a multimedia content processing method in one embodiment;

[0032] Figure 2 This is a flowchart illustrating a multimedia content processing method in another embodiment;

[0033] Figure 3 This is a flowchart illustrating the functional characteristics of a system in one embodiment;

[0034] Figure 4 This is a flowchart illustrating a unified coding framework for multimodal features in one embodiment;

[0035] Figure 5 This is a flowchart illustrating the dynamic contextual reasoning process of LLM (Large Language Model) in one embodiment.

[0036] Figure 6 This is a flowchart illustrating a multimedia content processing method in yet another embodiment;

[0037] Figure 7This is a structural block diagram of a multimedia content processing device in one embodiment;

[0038] Figure 8 This is an internal structural diagram of a computer device in one embodiment. Detailed Implementation

[0039] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0040] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of the relevant data must comply with relevant regulations.

[0041] In one exemplary embodiment, such as Figure 1 As shown, a multimedia content processing method is provided. This embodiment illustrates the method by applying it to a server. It is understood that this method can also be applied to a terminal, or to a system including both a terminal and a server, and is implemented through interaction between the terminal and the server. The terminal can be, but is not limited to, various personal computers, laptops, smartphones, and tablets; the server can be a standalone server or a server cluster consisting of multiple servers, such as a cloud server (computer). In this embodiment, the method includes the following steps:

[0042] Step S101: Extract the text feature vector of the text metadata of the multimedia content, and extract the image feature vector of the target image included in the multimedia content.

[0043] Multimedia content includes audio, video, text, and images.

[0044] The text metadata, also known as work metadata, is in JSON (JavaScript Object Notation, a text data exchange format) format and includes the song title, description text, and ASR (Automatic Speech Recognition) text of the multimedia content.

[0045] Among them, the text feature vector is used to represent the representation vector of metadata. In practical scenarios, the text feature vector refers to the CLS (Classification Token Vector) vector.

[0046] The target image, also known as multimedia data, includes video keyframes (1 frame per second) and carousel images (up to 5 images) of multimedia content.

[0047] The image feature vector is used to represent the representation vector of the target image. In practical scenarios, the image feature vector refers to a feature vector with a dimension of 768.

[0048] For example, in response to a content processing request for multimedia content, the server obtains the text metadata of the multimedia content and the target image included in the multimedia content. Next, the server extracts the song title, description text, and ASR text from the metadata, combines them to obtain a combined text, and inputs the combined text into the RoBERTa (Robustly Optimized BERT Pretraining Approach, a pre-trained language model) model. RoBERTa performs feature extraction processing on the combined text to obtain the corresponding feature vector, which serves as the text feature vector of the text metadata. Then, the server determines the video keyframes and carousel images in the target image (which may contain dynamic or static images), and inputs them into the ViT-H-14 (Vision Transformer-Huge-14, an image feature extraction model) model. The ViT-H-14 model performs feature extraction processing on the video keyframes and carousel images respectively, obtaining the feature vectors of the video keyframes and the carousel images. The feature vectors of the video keyframes and the carousel images are then fused to obtain the image feature vector of the target image.

[0049] For example, see reference. Figure 2 The data preprocessing layer parses the metadata of the work and preprocesses the multimedia data. The RoBERTa model in the multimodal coding layer extracts text features from the parsed metadata to obtain a CLS vector, which serves as the text feature vector of the text metadata. The ViT-H-14 model in the multimodal coding layer extracts image features from the preprocessed multimedia data to obtain a 768-dimensional vector, which serves as the image feature vector of the target image.

[0050] Step S102: The text feature vector and the image feature vector are fused to obtain a fused feature vector, and the first confidence level of the multimedia content is determined based on the fused feature vector.

[0051] Among them, the fused feature vector refers to the feature vector obtained by fusing text feature vectors and image feature vectors.

[0052] The first confidence level indicates the probability that the multimedia content belongs to a preset content type. The preset content type refers to illegal content.

[0053] For example, the server performs dimension unification processing on the text feature vector and the image feature vector respectively, obtaining text feature vectors and image feature vectors with the same dimension. Then, the server performs fusion processing on the text feature vectors and image feature vectors with the same dimension, obtaining a fused feature vector. Next, the server inputs the fused feature vector into a fully connected classification layer, and adjusts the fused feature vector through the weight matrix and bias vector of the fully connected classification layer, obtaining a two-dimensional adjusted fused feature vector (e.g., [probability of violation, probability of compliance]). Then, the server normalizes the adjusted fused feature vector through the Softmax function, obtaining a normalized fused feature vector. Next, the server extracts the first value in the fused feature vector from the normalized fused feature vector as the probability that the multimedia content belongs to a preset content type, and uses the probability that the multimedia content belongs to the preset content type as the first confidence level of the multimedia content.

[0054] In step S103, the text metadata and target image are input into the trained semantic reasoning model to obtain the second confidence level of the multimedia content.

[0055] Semantic reasoning models refer to network models that can obtain the second confidence level of multimedia content by utilizing the metadata and target images of multimedia content, such as the LLM model.

[0056] The second confidence level is used to indicate the probability that the multimedia content belongs to a preset content type.

[0057] For example, the server performs denoising processing on the text metadata and the target image to obtain denoised metadata and denoised target image; then, the server inputs the denoised metadata and denoised target image into a trained semantic reasoning model to obtain the probability that the multimedia content belongs to a preset content type, which serves as the second confidence level of the multimedia content.

[0058] Step S104: Determine the target processing action for the multimedia content based on the first confidence level and the second confidence level.

[0059] The target processing actions include delisting, release, traffic restriction, and manual review.

[0060] For example, the server queries the correspondence between the first confidence level, the second confidence level and the processing action to obtain the processing action corresponding to the first confidence level and the second confidence level, which is used as the target processing action corresponding to the multimedia content; then, the server processes the multimedia content according to the target processing action corresponding to the multimedia content.

[0061] In an exemplary embodiment, step S102 above, which fuses the text feature vector and the image feature vector to obtain a fused feature vector, specifically includes the following: projecting the text feature vector and the image feature vector respectively to obtain projected text feature vector and projected image feature vector; determining the modal gating weights corresponding to the projected text feature vector and the projected image feature vector; determining the first weight corresponding to the projected text feature vector and the second weight corresponding to the projected image feature vector based on the modal gating weights; and fusing the text feature vector and the image feature vector according to the first weight and the second weight to obtain the fused feature vector.

[0062] Among them, the projected text feature vector refers to the text feature vector after dimensional unification or spatial alignment.

[0063] Among them, the projected image feature vector refers to the image feature vector after dimensional unification or spatial alignment.

[0064] Modal gating weights refer to the weight parameters used to determine the modal importance of text and images.

[0065] The first weight refers to the fusion weight corresponding to the text feature vector.

[0066] The second weight refers to the fusion weight corresponding to the image feature vector.

[0067] For example, the server inputs text feature vectors and image feature vectors into a dynamic dimensionality projection network. The image feature vectors are processed by deformable convolution through the dynamic dimensionality projection network to obtain convolved image feature vectors. Then, based on the weight matrix and bias vector of the convolved image feature vectors, spatially perceptual projection is performed on the convolved image feature vectors to obtain projected image feature vectors. The text feature vectors are then compressed using multi-head projection (multi-head attention processing) through the dynamic dimensionality projection network to obtain projected text feature vectors. Next, the server determines the modality gating weights corresponding to the projected text feature vectors and projected image feature vectors based on the projected text feature vectors and projected image feature vectors. Then, the server performs an inverse complementary transformation on the modality gating weights to obtain processed modality gating weights, and uses these processed modality gating weights as the first weights corresponding to the projected text feature vectors and as the second weights corresponding to the projected image feature vectors. Finally, the server sums the text feature vectors and image feature vectors according to the first and second weights to obtain a fused feature vector.

[0068] For example, the feature vector of the projected image can be calculated using the following formula:

[0069] Equation (1)

[0070] in, This refers to the feature vector of the projected image. This refers to the feature vector of the image after convolution. It refers to the weight matrix of the feature vectors of the image after convolution. It refers to the bias vector of the feature vector of the image after convolution.

[0071] For example, the projected text feature vector can be calculated using the following formula:

[0072] Equation (2)

[0073] in, This refers to the text feature vector after projection. This refers to the multi-head attention parameters corresponding to the text feature vectors. This refers to the text feature vector.

[0074] For example, the fused feature vector can be calculated using the following formula:

[0075] Equation (3)

[0076] in, This refers to the fusion of feature vectors, where ⊙ represents element-wise multiplication, and α refers to the modal gating weight.

[0077] In this embodiment, the text feature vector and image feature vector are projected separately to unify the feature space. Then, the weights corresponding to the feature vectors are dynamically determined by modal gating weights and then fused. This can eliminate the fusion barrier caused by feature differences between different modalities, so that the fused feature vector can more accurately capture the core semantic and visual information of multimedia content, which is conducive to improving the accuracy of the determination of the fused feature vector.

[0078] In an exemplary embodiment, the modal gating weights corresponding to the projected text feature vector and the projected image feature vector are determined, specifically including the following: concatenating the projected text feature vector and the projected image feature vector to obtain a concatenated feature vector; and performing nonlinear transformation and activation processing on the concatenated feature vector to obtain the modal gating weights.

[0079] Among them, the concatenated feature vector refers to the feature vector obtained by concatenating the projected text feature vector and the projected image feature vector.

[0080] For example, the server concatenates the projected text feature vector and the projected image feature vector according to a preset concatenation order to obtain a concatenated feature vector; then, the server performs a nonlinear transformation on the concatenated feature vector using an MLP (Multilayer Perceptron) to obtain a processed concatenated feature vector; finally, the server activates the processed concatenated feature vector using a Sigmoid activation function to obtain modal gating weights.

[0081] For example, the modal gating weights can be calculated using the following formula:

[0082] Equation (4)

[0083] Where σ is the Sigmoid activation function.

[0084] In this embodiment, the splicing operation can completely preserve the original information of the two modalities and avoid feature loss; the nonlinear transformation can overcome the limitations of linear processing and capture the complex interaction between text semantics and image vision; the activation processing can map the transformation results to a reasonable range and generate gate weights that accurately reflect the importance of the two modalities, providing a reliable basis for subsequent dynamic weighted fusion.

[0085] In an exemplary embodiment, projecting an image feature vector to obtain a projected image feature vector specifically includes the following steps: determining the location encoding vector corresponding to the image feature vector based on the timestamp corresponding to the image feature vector; fusing the image feature vector and the location encoding vector to obtain an enhanced image feature vector; applying an attention mechanism to the enhanced image feature vector to obtain a processed image feature vector; performing global pooling on the processed image feature vector to obtain a global image feature vector; and projecting the global image feature vector to obtain a projected image feature vector.

[0086] The timestamp is used to represent the time stamp (such as millisecond-level timestamp, frame number, etc.) in which the image corresponding to the image feature vector was generated or acquired in a time sequence (such as video frames, continuously captured image streams). For example, the timestamp of the 25th frame in the sequence is 25.

[0087] The position encoding vector is used to represent the representation vector of the position information of the image in the time sequence corresponding to the image feature vector.

[0088] Among them, the enhanced image feature vector refers to the feature vector obtained by fusing the image feature vector and the location coding vector.

[0089] Here, the processed image feature vector refers to the enhanced image feature vector after attention mechanism processing.

[0090] The global image feature vector refers to the processed image feature vector after global pooling.

[0091] For example, the server determines the timestamp corresponding to the image feature vector based on the image acquisition time. Next, the server performs positional encoding (e.g., sine / cosine hybrid encoding) on ​​the timestamp corresponding to the image feature vector, transforming the timestamp into a feature vector of the same dimension as the image feature vector, which serves as the positional encoding vector corresponding to the image feature vector. Then, the server sums the image feature vector and the positional encoding vector to obtain the enhanced image feature vector. Next, the server applies an attention mechanism to the enhanced image feature vector through a pre-constructed temporal self-attention layer to obtain the processed image feature vector. Finally, the server performs global pooling on the processed image feature vector to obtain the global image feature vector. For example, the server performs... Mean pooling and max pooling are used to obtain the image feature vectors after mean pooling and max pooling, respectively. These two feature vectors are then concatenated to obtain the global image feature vector. Next, the server inputs the text feature vector and the global image feature vector into a dynamic dimensionality projection network. The global image feature vector undergoes deformable convolution through the dynamic dimensionality projection network to obtain the convolved image feature vector. Based on the weight matrix and bias vector of the convolved image feature vector, spatially perceptual projection is performed to obtain the projected image feature vector. Finally, the text feature vector is compressed using multi-head projection (multi-head attention processing) through the dynamic dimensionality projection network to obtain the projected text feature vector.

[0092] For example, the enhanced image feature vector can be calculated using the following formula:

[0093] Equation (5)

[0094] Where t refers to the timestamp, and PositionEncoding(t) refers to the position encoding vector.

[0095] For example, the feature vector of the processed image can be calculated using the following formula:

[0096] Equation (6)

[0097] in, It refers to the processed image feature vector, where Q, K, and V are obtained by linear transformation of the enhanced image feature vector, and d is the feature dimension of the enhanced image feature vector.

[0098] In this embodiment, by injecting time-stamp-related location encoding into the image feature vector, temporal dimension information is first added to enhance the spatiotemporal representation capability of the features. Then, the attention mechanism is used to capture the correlation between cross-frame image features. The resulting global image feature vector more comprehensively and accurately reflects the overall information of the image sequence, which is beneficial to improving the accuracy of global image feature vector determination.

[0099] In an exemplary embodiment, step S103 above, which inputs the text metadata and the target image into the trained semantic reasoning model to obtain the second confidence of the multimedia content, specifically includes the following: if the first confidence is less than the preset confidence, the text metadata and the target image are input into the trained semantic reasoning model to obtain the second confidence of the multimedia content.

[0100] Here, pre-set reliability refers to a pre-defined confidence threshold. It should be noted that the pre-set reliability depends on the specific circumstances.

[0101] For example, the server determines the first confidence level based on a preset confidence level; if the first confidence level is less than the preset confidence level, the server inputs the text metadata and the target image into the trained semantic reasoning model to obtain the second confidence level of the multimedia content; if the first confidence level is greater than or equal to the preset confidence level, the server generates a takedown instruction for the multimedia content and takes down the multimedia content according to the takedown instruction.

[0102] For example, such as Figure 3 As shown, if the first confidence level is less than the preset confidence level, a multimodal report is generated through LLM review for quick manual confirmation; if the first confidence level is greater than or equal to the preset confidence level (i.e., it is a high-confidence violation), the multimedia content is taken down through the first-level automatic handling mechanism.

[0103] It should be noted that the flowchart of the functional characteristics of this system is as follows: Figure 3 As shown, integrated into the multimedia content review platform, it performs secondary precise judgment on suspected violations recalled by machine review algorithms, covering the following scenarios: 1. Multimodal combination violation detection: including normal lyrics paired with carousel images, and contradictions between homophonic lyrics and ASR text. 2. Implicit semantic understanding: metaphorical lyrics and partial image mosaics to circumvent review. 3. Dynamic policy adaptation: real-time adjustment of review rules according to regulatory requirements (such as filtering sensitive words for specific holidays).

[0104] Features:

[0105] 1. Visualization of multimodal evidence chains: Highlighting of violating elements, such as marking sensitive words in red in the lyrics text, and selecting violating areas in the carousel; semantic association graphs, such as displaying the relationships between lyrics, images, and audio.

[0106] 2. Tiered review workflow: Level 1 Automatic handling: Directly remove high-confidence violation content; Level 2 LLM review: Generate multimodal review reports for medium-risk content for manual confirmation.

[0107] 3. Dynamic updates to the knowledge base: Real-time synchronization of regulatory policies and emerging internet slang into the LLM context.

[0108] In this embodiment, by setting a pre-set confidence level as a decision threshold, multimedia content with different first confidence levels is treated differently, reducing the erroneous removal of compliant content due to misjudgment of a single confidence level. Ultimately, a balance is achieved between the efficiency and accuracy of risk prevention and control, which is conducive to improving the overall reliability and practicality of multimedia content review.

[0109] In an exemplary embodiment, step S103 above, which inputs the text metadata and the target image into a trained semantic reasoning model to obtain a second confidence level of the multimedia content, specifically includes the following: determining the prompt words corresponding to the multimedia content based on the text metadata and the target image; inputting the prompt words into the trained semantic reasoning model to obtain the probability level of the multimedia content belonging to the preset content type; and obtaining the second confidence level of the multimedia content based on the probability level of the multimedia content belonging to the preset content type.

[0110] Among them, prompt words are used to represent natural language instructions corresponding to multimedia content, also known as multimodal prompts.

[0111] Among them, the probability level is used to represent the confidence level of the category to which multimedia content belongs, such as high-risk content and medium-risk content.

[0112] For example, the server constructs prompt words corresponding to the text metadata and the target image based on the text metadata and the target image, which are used as prompt words for the multimedia content; then, the server inputs the prompt words into a trained semantic reasoning model to obtain the probability level of the multimedia content belonging to a preset content type output by the trained semantic reasoning model; then, the server uses the probability level of the multimedia content belonging to the preset content type as the second confidence level of the multimedia content.

[0113] In this embodiment, by integrating the structured information of text metadata with the visual information of the target image into a prompt word, and using this prompt word as a bridge, multi-source information can be transformed into a unified instruction that the model can understand. This avoids the separation between metadata and image information, and makes the final second confidence score take into account both the rigor of the structured data and the intuitiveness of the visual information, which is conducive to improving the accuracy of the second confidence score determination.

[0114] In an exemplary embodiment, determining the prompt words corresponding to the multimedia content based on text metadata and target images specifically includes the following: determining the target semantic relevance of each cross-modal pair among multiple cross-modal pairs based on text metadata and target images; selecting cross-modal pairs with a target semantic relevance greater than a preset semantic relevance as target cross-modal pairs from multiple cross-modal pairs; and determining the prompt words corresponding to the multimedia content based at least on the target cross-modal pairs.

[0115] Among them, the target semantic relevance is used to represent the degree of correlation between text metadata and the visual semantic information of the target image.

[0116] Each cross-modal pair includes one text metadata and one target image. For example, assuming there are 3 text metadata (e.g., text metadata A, text metadata B, and text metadata C) and 2 target images (e.g., target image a and target image b), there are a total of 6 cross-modal pairs (i.e., text metadata A + target image a, text metadata B + target image a, text metadata C + target image a, text metadata A + target image b, text metadata B + target image b, and text metadata C + target image b).

[0117] The preset semantic relevance refers to a pre-set threshold for semantic relevance. It should be noted that the preset semantic relevance depends on the specific circumstances.

[0118] Among them, the target cross-modal pair refers to the cross-modal pair whose target semantic relevance is greater than the preset semantic relevance.

[0119] For example, the server combines the sub-text metadata in the text metadata and the sub-target image in the target image to obtain multiple cross-modal pairs; then, the server inputs each of the multiple cross-modal pairs into a trained semantic relevance prediction model to obtain the target semantic relevance of each cross-modal pair; then, the server selects cross-modal pairs from the multiple cross-modal pairs whose target semantic relevance is greater than a preset semantic relevance, and uses these cross-modal pairs as target cross-modal pairs; then, the server constructs prompt words corresponding to the target cross-modal pairs as prompt words corresponding to the multimedia content.

[0120] In this embodiment, by calculating the semantic correlation degree of the cross-modal pairs formed by the text metadata and the target image, highly correlated target cross-modal pairs are selected to generate prompt words. This can filter out invalid information that does not match the semantics of the text metadata and the image, avoid irrelevant information from interfering with the generation of prompt words, and help improve the generation quality of prompt words.

[0121] In an exemplary embodiment, the target semantic relevance of each of the multiple cross-modal pairs is determined based on text metadata and the target image. Specifically, this includes: constructing a text-guided image attention matrix and an image-guided text attention matrix based on text metadata and the target image; and determining the target semantic relevance of each cross-modal pair based on a first semantic relevance and a second semantic relevance.

[0122] The text-guided image attention matrix includes the first semantic relevance of each cross-modal pair.

[0123] The first semantic relevance is used to represent the semantic attention weight of the text metadata to the target image.

[0124] The image-guided text attention matrix includes a second semantic relevance for each cross-modal pair.

[0125] The second semantic relevance is used to represent the semantic attention weight of the target image to the text metadata.

[0126] For example, the server determines the semantic attention weight of the text metadata to the target image, and constructs a text-guided image attention matrix based on the semantic attention weight of the text metadata to the target image; then, the server determines the semantic attention weight of the target image to the text metadata, and constructs an image-guided text attention matrix based on the semantic attention weight of the target image to the text metadata; then, the server determines the maximum semantic relevance from the first semantic relevance and the second semantic relevance of each cross-modal pair, and uses it as the target semantic relevance of each cross-modal pair.

[0127] Furthermore, the server can also determine the semantic attention weights of the text feature vectors to the processed image feature vectors, and construct a text-guided image attention matrix based on the semantic attention weights of the text feature vectors to the processed image feature vectors; then, the server determines the semantic attention weights of the processed image feature vectors to the text feature vectors, and constructs an image-guided text attention matrix based on the semantic attention weights of the processed image feature vectors to the text feature vectors.

[0128] For example, the text-guided image attention matrix can be calculated using the following formula:

[0129] Equation (7)

[0130] in, This refers to the matrix elements in the text-guided image attention matrix. and Linear projection functions for text and image features, respectively, T i I represents the i-th text unit.j Let N be the j-th image region, N be the total number of target images, and k be the image index variable.

[0131] in, and The linear projection functions for text and image features are given below, with the corresponding formulas as follows:

[0132] Equation (8)

[0133] Equation (9)

[0134] Among them, W text and W img The weight matrix projects the features onto a common semantic space of the same dimension; b text and b img It refers to the bias vector.

[0135] For example, the image-guided text attention matrix can be calculated using the following formula:

[0136] Equation (10)

[0137] in, This refers to the matrix elements in the text-guided image attention matrix. and Linear projection functions for text and image features, respectively, T i I represents the i-th text unit. j Let N be the j-th image region, N be the total number of text metadata, and k be the text metadata variable of the image.

[0138] For example, see reference. Figure 4 By using a cross-modal semantic alignment network, multi-granular text parsing is performed on text metadata, and salient region detection is performed on target images. Based on the processed text metadata and processed target images, text-guided image attention matrix and image-guided text attention matrix are constructed.

[0139] In this embodiment, by constructing a bidirectional attention matrix between text-guided images and image-guided text, the deep semantic relationship between text metadata and target images is accurately captured. Then, the target semantic relationship is determined by combining the first semantic relationship degree and the second semantic relationship degree. This helps to break the information fragmentation limitation of single-modal review and provide a reliable data foundation for subsequent dynamic reasoning.

[0140] In an exemplary embodiment, the prompt corresponding to the multimedia content is determined at least according to the target cross-modal pair, which specifically includes the following: constructing a text-guided image attention matrix and an image-guided text attention matrix according to the text metadata and the target image; determining a target transcription text from the multiple transcription texts based on the text deviation between the multiple transcription texts corresponding to the multimedia content and the target text corresponding to the multimedia content; constructing the prompt corresponding to the multimedia content according to the target cross-modal pair, the target transcription text, the text-guided image attention matrix, the image-guided text attention matrix, and the text deviation.

[0141] The transcription text refers to the ASR text corresponding to the multimedia content, which is specifically obtained by converting the audio data in the multimedia content through ASR technology.

[0142] It should be noted that there are multiple transcription texts corresponding to the multimedia content, and the reasons are as follows: In the prior art, there are obvious defects in audio review relying on ASR results to match the sensitive word library. ASR recognition is easily affected by accents, noises, homophonic pronunciations, etc. and generates errors (such as misconverting "qingliu" to "qinglou"), and such errors will directly lead to the problem of misjudgment in subsequent reviews. Therefore, generating multiple transcription texts is precisely to cover various possibilities of ASR recognition to improve the determination accuracy of the target transcription text.

[0143] The target text refers to the official lyrics text corresponding to the multimedia content.

[0144] The text deviation is used to represent the semantic difference degree between the transcription text and the target text.

[0145] The target transcription text is used to represent the transcription text with a text deviation greater than the preset text deviation.

[0146] For example, the server constructs a text-guided image attention matrix and an image-guided text attention matrix based on the text metadata and the target image. Then, the server inputs the multimedia content into multiple ASR models with different architectures to obtain multiple transcribed texts corresponding to the multimedia content. Next, the server extracts the semantic vectors of the transcribed text and the target text using a Sentence-BERT (Sentence Embedding Model) model, respectively. Then, the server determines the similarity between the transcribed text and the target text based on the semantic vectors of the transcribed text and the target text. Next, the server performs an inverse complementary transformation on the similarity between the transcribed text and the target text to obtain the text deviation between the transcribed text and the target text, and determines the target transcribed text from the transcribed text based on the text deviation between the multiple transcribed texts corresponding to the multimedia content and the target text corresponding to the multimedia content. Next, the server constructs prompt words corresponding to the target cross-modal pair, the target transcribed text, the text-guided image attention matrix, the image-guided text attention matrix, and the text deviation, as prompt words corresponding to the multimedia content.

[0147] In this embodiment, a bidirectional attention matrix is ​​used to accurately capture the deep cross-modal semantic relationship between text metadata and target images. The text deviation is used to filter out accurate target transcribed text. Then, the target cross-modal pairs, bidirectional attention information and deviation are fused to construct prompt words, which breaks the limitation of single-modal information fragmentation and effectively avoids missed detection and misjudgment of multimodal combination violations and veiled violations.

[0148] In an exemplary embodiment, prompt words corresponding to multimedia content are constructed based on the target cross-modal pair, target transcribed text, text-guided image attention matrix, image-guided text attention matrix, and text deviation. Specifically, this includes: querying a preset knowledge graph to obtain target retrieval results corresponding to the target cross-modal pair and target transcribed text based on the target cross-modal pair and target transcribed text; and constructing prompt words corresponding to multimedia content based on the target cross-modal pair, target transcribed text, text-guided image attention matrix, image-guided text attention matrix, text deviation, and target retrieval results.

[0149] The pre-defined knowledge graph is used to represent the correspondence between cross-modal pairs, transcribed texts, and search results. The target search results refer to the search results in the pre-defined knowledge graph that correspond to the target cross-modal pair and the target transcribed text.

[0150] For example, the server constructs an initial knowledge graph using a pre-defined review specification database, a network slang evolution database, and a multimodal violation case database. It then updates the initial knowledge graph through a dual-channel mechanism (including active synchronization, i.e., real-time acquisition of updated review specifications and triggering graph reconstruction, and passive learning, i.e., optimization of graph node weights through comparative learning based on manual review feedback data), resulting in a pre-defined knowledge graph. Next, the server queries the pre-defined knowledge graph based on the target cross-modal pair and the target transcribed text to obtain target retrieval results corresponding to the target cross-modal pair and the target transcribed text. Then, based on the target cross-modal pair, the target transcribed text, the text-guided image attention matrix, the image-guided text attention matrix, the text deviation, and the target retrieval results, the server constructs prompt words corresponding to the target cross-modal pair, the target transcribed text, the text-guided image attention matrix, the image-guided text attention matrix, the text deviation, and the target retrieval results, which serve as prompt words for the multimedia content.

[0151] For example, a multimodal dynamic prompt generation framework such as Figure 5 As shown, its core process is as follows:

[0152] Step 1, Multimodal Feature Association Extraction: Obtain cross-modal association features from the multimodal coding layer, including text-image attention matrix (identifying the semantic association strength between lyric fragments and image regions), audio-text contradiction score (calculating the semantic deviation between ASR results and lyric text), extract key association nodes, filter cross-modal pairs with text-image attention values ​​exceeding threshold τ1 (such as the text "swimsuit" and the exposed area image), and identify fragments with audio-text contradiction scores exceeding threshold τ2 (such as ASR transcribing as "brothel" and lyric text displaying "clear stream").

[0153] The "Audio-Text Contradiction Score" aims to quantify the semantic differences between the ASR transcription results and the official lyrics text. Its calculation steps are as follows: 1. Input: ASR transcription text Sasr and official lyrics text Slyric. 2. Semantic Encoding: Using the Sentence-BERT sentence embedding model, Sasr and Slyric are encoded into fixed-dimensional semantic vectors Vasr and Vlyric, respectively. 3. Similarity Calculation: The cosine similarity between the two semantic vectors is calculated. 4. Contradiction Score: The contradiction score is defined as ContradictionScore = The higher the score, the greater the semantic difference between the ASR results and the original lyrics.

[0154] Step 2, Knowledge Graph Context Retrieval: Based on the key features extracted in Step 1, construct hybrid query conditions. Perform hybrid retrieval through Elasticsearch's multi-level index engine (an indexing engine) to return relevant background knowledge.

[0155] Step 3, Multimodal Cue Synthesis: Design a hierarchical cue template structure to achieve structured injection of multimodal information. Its data comes from the outputs of multiple upstream modules: 1. Text Risk Items: from keyword hit results and confidence scores of the text moderation model (RoBERTa). 2. Image Risk Items: from NSFW scores and region detection results of the image moderation model (ViT). 3. Cross-modal Associations: from the most significant connection pairs and their similarity in the cross-modal attention matrix (i.e., the text-image attention matrix). 4. Audio Contradiction Items: from audio-text contradiction scores. 5. Background Knowledge: from the results of knowledge graph retrieval (e.g., mapping "brothel" to "pure stream").

[0156] Generation process: The system has predefined natural language generation templates. The numerical values ​​and identifiers (such as region descriptions and similarities) in the above data are filled into the corresponding placeholders in the templates to automatically generate a coherent text description containing quantified evidence, i.e., a multimodal evidence chain.

[0157] It should be noted that this solution overcomes the limitations of traditional Large Language Models (LLMs) in content moderation scenarios due to their static and fragmented applications, proposing a dynamic contextual reasoning framework for multimodal moderation. By constructing a knowledge-driven context generation engine and designing domain-adaptive model optimization strategies, the system can accurately identify subtly prohibited content by combining real-time policy requirements with multimodal semantic associations. To support dynamic context generation, this solution constructs a knowledge graph covering multi-dimensional moderation knowledge, including the following core elements:

[0158] Policy and Regulation Subgraph: Structured storage of review standards, achieving semantic expression of clauses through entity extraction and relation modeling. Internet Slang Evolution Library: A dynamically updated mapping table of internet slang based on historical review data and social media corpora.

[0159] Real-time updates to the knowledge graph are achieved through a dual-channel mechanism of active synchronization and passive learning: Active synchronization: By connecting to the API (Application Programming Interface) of regulatory agencies, policy updates are obtained in real time, triggering graph reconstruction (such as automatic parsing and node insertion for newly added sensitive word lists). Passive learning: Based on feedback data from manual review, the weights of graph nodes are optimized through comparative learning (such as automatically increasing the associated risk level when the misjudgment rate of a certain type of homophone exceeds a threshold). In this embodiment, by relying on the knowledge graph to supplement external prior knowledge not included in the multimedia content itself, and combining the target cross-modal pairs and target transcribed text queries with the preset knowledge graph to obtain target retrieval results, the generated prompt words are supported by both the content's own information and external professional knowledge, avoiding the one-sidedness of analysis caused by information silos, and providing more comprehensive and accurate input data for subsequent semantic reasoning models.

[0160] It should be noted that the trained semantic reasoning model can be obtained through the following method: First, a training dataset (including 200,000 historical review records and 100,000 adversarial examples) is constructed to train the semantic reasoning model. The model is then trained using this dataset to obtain the standard cross-entropy loss, format loss, and knowledge reference loss values. These values ​​are then weighted and summed to obtain the loss value of the semantic reasoning model. Next, the model is iteratively trained according to the loss value to obtain the trained semantic reasoning model. Then, based on the training data of the trained semantic reasoning model (50,000 review cases with multimodal annotations), the trained semantic reasoning model is fine-tuned using multimodal alignment to obtain the fine-tuned semantic reasoning model. Finally, this fine-tuned semantic reasoning model is used as the completed semantic reasoning model.

[0161] It should be noted that, in one exemplary embodiment, a hierarchical progressive cross-modal distillation architecture is proposed to realize knowledge transfer from the teacher model to the student model. The specific process is as follows: Obtain the teacher model and the corresponding student model; input the sample training data into the teacher model and the student model respectively to obtain the output results of the teacher model and the student model; based on the difference between the output results of the teacher model and the student model, obtain the cross-entropy loss of the student model's main task, the mean squared error loss of intramodal feature imitation, and the KL (Kullback-Leibler) loss of the cross-modal association matrix. The loss is calculated by summing the divergence loss (Kourbeck-Leibler divergence) and the loss from the downstream auxiliary task. These loss values ​​are then fused to obtain a fused loss value. Based on this fused loss value, the student model is iteratively trained to obtain a trained student model. Using the trained student model, text feature vectors of the multimedia content's metadata and image feature vectors of the target images within the multimedia content are extracted. The text and image feature vectors are then fused using the trained student model to obtain a fused feature vector. Based on this fused feature vector, the first confidence level of the multimedia content is determined.

[0162] In one exemplary embodiment, such as Figure 6 As shown, another multimedia content processing method is provided. Taking the application of this method to a server as an example, the method includes the following steps:

[0163] Step S601: Extract the text feature vector of the text metadata of the multimedia content, and extract the image feature vector of the target image included in the multimedia content. Step S602: Perform projection processing on the text feature vector and image feature vector respectively to obtain projected text feature vector and projected image feature vector. Step S603: Concatenate the projected text feature vector and projected image feature vector to obtain a concatenated feature vector; perform nonlinear transformation and activation processing on the concatenated feature vector to obtain modal gating weights. Step S604: Determine the first weight corresponding to the projected text feature vector and the second weight corresponding to the projected image feature vector based on the modal gating weights. Step S605: Perform fusion processing on the text feature vector and image feature vector according to the first weight and the second weight to obtain a fused feature vector. Step S606: Determine the first confidence level of the multimedia content based on the fused feature vector; the first confidence level is used to represent the probability that the multimedia content belongs to a preset content type. Step S607: Construct a text-guided image attention matrix and an image-guided text attention matrix based on the text metadata and the target image. The text-guided image attention matrix includes the first semantic relevance of each cross-modal pair; the image-guided text attention matrix includes the second semantic relevance of each cross-modal pair; each cross-modal pair includes one piece of text metadata and one target image. Step S608: Determine the target semantic relevance of each cross-modal pair based on the first and second semantic relevance. Step S609: Select cross-modal pairs with a target semantic relevance greater than a preset semantic relevance from multiple cross-modal pairs as target cross-modal pairs. Step S610: Construct a text-guided image attention matrix and an image-guided text attention matrix based on the text metadata and the target image; determine the target transcribed text from multiple transcribed texts based on the text deviation between the multiple transcribed texts corresponding to the multimedia content and the target text corresponding to the multimedia content. Step S611: Based on the target cross-modal pair and the target transcribed text, query the preset knowledge graph to obtain the target retrieval results corresponding to the target cross-modal pair and the target transcribed text; the preset knowledge graph is used to represent the correspondence between the cross-modal pair, the transcribed text, and the retrieval results. Step S612: Based on the target cross-modal pair, the target transcribed text, the text-guided image attention matrix, the image-guided text attention matrix, the text deviation, and the target retrieval results, construct prompt words corresponding to the multimedia content. Step S613: Input the prompt words into the trained semantic reasoning model to obtain the probability level of the multimedia content belonging to the preset content type. Step S614: Based on the probability level of the multimedia content belonging to the preset content type, obtain the second confidence level of the multimedia content; the second confidence level is used to represent the probability that the multimedia content belongs to the preset content type. Step S615: Based on the first confidence level and the second confidence level, determine the target processing action for the multimedia content.

[0164] The above-mentioned multimedia content processing method can achieve the following beneficial effects: By fusing the text feature vector of the text metadata of the multimedia content and the image feature vector of the target image in the multimedia content, a more comprehensive fused feature vector can be obtained. Based on this fused feature vector, the first confidence level used to represent the multimedia content can be determined more accurately. By processing the text metadata and target image of the multimedia content through the trained semantic reasoning model, the second confidence level used to represent the multimedia content can be determined more accurately. Furthermore, by combining the first and second confidence levels, the target processing action corresponding to the multimedia content can be determined more accurately, thus enabling more targeted processing of the multimedia content and improving the processing accuracy. Moreover, the entire process adopts a multimodal analysis approach, avoiding the shortcomings of single-modal analysis, which is difficult to capture cross-modal associations and prone to errors, resulting in lower processing accuracy of multimedia content. This further improves the processing accuracy of multimedia content.

[0165] It should be understood that although the steps in the flowcharts of the above embodiments are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the above embodiments may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages of other steps.

[0166] Based on the same inventive concept, this application also provides a multimedia content processing apparatus for implementing the multimedia content processing method described above. The solution provided by this apparatus is similar to the implementation scheme described in the above method; therefore, the specific limitations in one or more multimedia content processing apparatus embodiments provided below can be found in the limitations of the multimedia content processing method described above, and will not be repeated here.

[0167] In one exemplary embodiment, such as Figure 7 As shown, a multimedia content processing device is provided, including: a vector extraction module 701, a first determination module 702, a second determination module 703, and a content processing module 704, wherein:

[0168] The vector extraction module 701 is used to extract the text feature vector of the text metadata of the multimedia content, and to extract the image feature vector of the target image included in the multimedia content.

[0169] The first determining module 702 is used to perform fusion processing on the text feature vector and the image feature vector to obtain a fused feature vector, and to determine a first confidence level of the multimedia content based on the fused feature vector; the first confidence level is used to indicate the probability that the multimedia content belongs to a preset content type.

[0170] The second determining module 703 is used to input the text metadata and the target image into a trained semantic reasoning model to obtain a second confidence level of the multimedia content; the second confidence level is used to indicate the probability that the multimedia content belongs to the preset content type.

[0171] The content processing module 704 is used to determine the target processing action for the multimedia content based on the first confidence level and the second confidence level.

[0172] In an exemplary embodiment, the first determining module 702 is further configured to perform projection processing on the text feature vector and the image feature vector respectively to obtain projected text feature vector and projected image feature vector; determine modal gating weights corresponding to the projected text feature vector and the projected image feature vector; determine a first weight corresponding to the projected text feature vector and a second weight corresponding to the projected image feature vector based on the modal gating weights; and perform fusion processing on the text feature vector and the image feature vector according to the first weight and the second weight to obtain the fused feature vector.

[0173] In an exemplary embodiment, the first determining module 702 is further configured to concatenate the projected text feature vector and the projected image feature vector to obtain a concatenated feature vector; and to perform nonlinear transformation and activation processing on the concatenated feature vector to obtain the modal gating weights.

[0174] In an exemplary embodiment, the first determining module 702 is further configured to: determine the position encoding vector corresponding to the image feature vector based on the timestamp corresponding to the image feature vector; perform fusion processing on the image feature vector and the position encoding vector to obtain an enhanced image feature vector; perform attention mechanism processing on the enhanced image feature vector to obtain a processed image feature vector; perform global pooling processing on the processed image feature vector to obtain a global image feature vector; and perform projection processing on the global image feature vector to obtain a projected image feature vector.

[0175] In an exemplary embodiment, the second determining module 703 is further configured to input the text metadata and the target image into a trained semantic reasoning model to obtain a second confidence level of the multimedia content if the first confidence level is less than a preset confidence level.

[0176] In an exemplary embodiment, the second determining module 703 is further configured to determine the prompt word corresponding to the multimedia content based on the text metadata and the target image; input the prompt word into a trained semantic reasoning model to obtain the probability level of the multimedia content belonging to the preset content type; and obtain a second confidence level of the multimedia content based on the probability level of the multimedia content belonging to the preset content type.

[0177] In an exemplary embodiment, the second determining module 703 is further configured to determine the target semantic relevance of each of the plurality of cross-modal pairs based on the text metadata and the target image; each cross-modal pair includes a text metadata and a target image; from the plurality of cross-modal pairs, select cross-modal pairs whose target semantic relevance is greater than a preset semantic relevance as target cross-modal pairs; and at least based on the target cross-modal pairs, determine the prompt words corresponding to the multimedia content.

[0178] In an exemplary embodiment, the second determining module 703 is further configured to construct a text-guided image attention matrix and an image-guided text attention matrix based on the text metadata and the target image; the text-guided image attention matrix includes a first semantic correlation degree for each of the cross-modal pairs; the image-guided text attention matrix includes a second semantic correlation degree for each of the cross-modal pairs; and determine a target semantic correlation degree for each of the cross-modal pairs based on the first semantic correlation degree and the second semantic correlation degree for each of the cross-modal pairs.

[0179] In an exemplary embodiment, the second determining module 703 is further configured to: construct a text-guided image attention matrix and an image-guided text attention matrix based on the text metadata and the target image; determine the target transcribed text from the plurality of transcribed texts based on the text deviation between the plurality of transcribed texts corresponding to the multimedia content and the target text corresponding to the multimedia content; and construct the prompt words corresponding to the multimedia content based on the target cross-modal pair, the target transcribed text, the text-guided image attention matrix, the image-guided text attention matrix, and the text deviation.

[0180] In an exemplary embodiment, the second determining module 703 is further configured to query a preset knowledge graph to obtain target retrieval results corresponding to the target cross-modal pair and the target transcribed text based on the target cross-modal pair and the target transcribed text; the preset knowledge graph is used to represent the correspondence between the cross-modal pair, the transcribed text and the retrieval results; and to construct prompt words corresponding to the multimedia content based on the target cross-modal pair, the target transcribed text, the text-guided image attention matrix, the image-guided text attention matrix, the text deviation degree and the target retrieval results.

[0181] Each module in the aforementioned multimedia content processing device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device, or stored in the memory of a computer device as software, so that the processor can call and execute the operations corresponding to each module.

[0182] In one exemplary embodiment, a computer device is provided, which may be a server, and its internal structure diagram may be as follows: Figure 8 As shown, this computer device includes a processor, memory, input / output interfaces (I / O), and a communication interface. The processor, memory, and I / O interfaces are connected via a system bus, and the communication interface is also connected to the system bus via the I / O interfaces. The processor provides computational and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and a database. The internal memory provides the environment for the operating system and computer programs stored in the non-volatile storage media. The database stores metadata, target images, and other data. The I / O interfaces are used for exchanging information between the processor and external devices. The communication interface is used for communicating with external terminals via a network connection. When the computer program is executed by the processor, it implements a multimedia content processing method.

[0183] Those skilled in the art will understand that Figure 8 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0184] In one exemplary embodiment, a computer device is also provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps in the above-described method embodiments.

[0185] In one exemplary embodiment, a computer-readable storage medium is provided having a computer program stored thereon that, when executed by a processor, implements the steps in the above-described method embodiments.

[0186] In one exemplary embodiment, a computer program product is provided, including a computer program that, when executed by a processor, implements the steps in the above-described method embodiments.

[0187] Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments described above. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, etc., and are not limited to these.

[0188] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0189] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of this patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. A multimedia content processing method, characterized in that, The method includes: Extract the text feature vector of the text metadata of the multimedia content, and extract the image feature vector of the target image included in the multimedia content; The text feature vector and the image feature vector are fused to obtain a fused feature vector. A first confidence level of the multimedia content is determined based on the fused feature vector. The first confidence level is used to indicate the probability that the multimedia content belongs to a preset content type. The text metadata and the target image are input into a trained semantic reasoning model to obtain a second confidence level of the multimedia content; the second confidence level is used to indicate the probability that the multimedia content belongs to the preset content type. Based on the first confidence level and the second confidence level, the target processing action for the multimedia content is determined.

2. The method according to claim 1, characterized in that, The process of fusing the text feature vector and the image feature vector to obtain a fused feature vector includes: The text feature vector and the image feature vector are respectively projected to obtain the projected text feature vector and the projected image feature vector. Determine the modal gating weights corresponding to the projected text feature vector and the projected image feature vector; The first weight corresponding to the projected text feature vector and the second weight corresponding to the projected image feature vector are determined based on the modal gating weights. The text feature vector and the image feature vector are fused according to the first weight and the second weight to obtain the fused feature vector.

3. The method according to claim 2, characterized in that, The determination of the modal gating weights corresponding to the projected text feature vector and the projected image feature vector includes: The projected text feature vector and the projected image feature vector are concatenated to obtain a concatenated feature vector. The spliced ​​feature vectors are subjected to nonlinear transformation and activation processing to obtain the modal gating weights.

4. The method according to claim 2, characterized in that, The image feature vector is projected to obtain the projected image feature vector, including: Based on the timestamp corresponding to the image feature vector, the location encoding vector corresponding to the image feature vector is determined; the image feature vector and the location encoding vector are fused to obtain the enhanced image feature vector; The enhanced image feature vector is processed by an attention mechanism to obtain a processed image feature vector; the processed image feature vector is then subjected to global pooling to obtain a global image feature vector. The global image feature vector is projected to obtain the projected image feature vector.

5. The method according to claim 1, characterized in that, The step of inputting the text metadata and the target image into a trained semantic reasoning model to obtain the second confidence level of the multimedia content includes: If the first confidence level is less than the preset confidence level, the text metadata and the target image are input into the trained semantic reasoning model to obtain the second confidence level of the multimedia content.

6. The method according to claim 1, characterized in that, The step of inputting the text metadata and the target image into a trained semantic reasoning model to obtain the second confidence level of the multimedia content includes: Based on the text metadata and the target image, determine the prompt words corresponding to the multimedia content; The prompt words are input into a trained semantic reasoning model to obtain the probability level of the multimedia content belonging to the preset content type; Based on the probability level of the multimedia content belonging to the preset content type, a second confidence level of the multimedia content is obtained.

7. The method according to claim 6, characterized in that, The step of determining the prompt words corresponding to the multimedia content based on the text metadata and the target image includes: Based on the text metadata and the target image, determine the target semantic relevance of each of the multiple cross-modal pairs; each cross-modal pair includes a text metadata and a target image; From the plurality of cross-modal pairs, cross-modal pairs with a target semantic relevance greater than a preset semantic relevance are selected as target cross-modal pairs; Based at least on the target cross-modal pair, determine the prompt words corresponding to the multimedia content.

8. The method according to claim 7, characterized in that, The step of determining the target semantic relevance of each of the multiple cross-modal pairs based on the text metadata and the target image includes: Based on the text metadata and the target image, a text-guided image attention matrix and an image-guided text attention matrix are constructed; the text-guided image attention matrix includes a first semantic relevance for each of the cross-modal pairs; the image-guided text attention matrix includes a second semantic relevance for each of the cross-modal pairs. The target semantic correlation degree of each cross-modal pair is determined based on the first semantic correlation degree and the second semantic correlation degree of each cross-modal pair.

9. The method according to claim 7, characterized in that, The step of determining the cue words corresponding to the multimedia content based at least on the target cross-modal pair includes: Based on the text metadata and the target image, construct a text-guided image attention matrix and an image-guided text attention matrix; Based on the text deviation between the multiple transcribed texts corresponding to the multimedia content and the target text corresponding to the multimedia content, the target transcribed text is determined from the multiple transcribed texts; Based on the target cross-modal pair, the target transcribed text, the text-guided image attention matrix, the image-guided text attention matrix, and the text deviation, the prompt words corresponding to the multimedia content are constructed.

10. The method according to claim 9, characterized in that, The step of constructing prompt words corresponding to the multimedia content based on the target cross-modal pair, the target transcribed text, the text-guided image attention matrix, the image-guided text attention matrix, and the text deviation includes: Based on the target cross-modal pair and the target transcribed text, a preset knowledge graph is queried to obtain target retrieval results corresponding to the target cross-modal pair and the target transcribed text; the preset knowledge graph is used to represent the correspondence between the cross-modal pair, the transcribed text and the retrieval results; Based on the target cross-modal pair, the target transcribed text, the text-guided image attention matrix, the image-guided text attention matrix, the text deviation, and the target retrieval results, the prompt words corresponding to the multimedia content are constructed.

11. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the multimedia content processing method according to any one of claims 1 to 10.

12. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the multimedia content processing method according to any one of claims 1 to 10.

13. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the multimedia content processing method according to any one of claims 1 to 10.

Images