A media content processing method, apparatus, equipment, medium, and program product.
By performing visual feature processing and pseudo-text generation on media content, and combining it with a hybrid tower structure content retrieval model, the problem of low retrieval accuracy in existing technologies has been solved, achieving efficient and accurate media content retrieval.
Patent Information
- Application Number
- CN202510622027.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-05-14
- Publication Date
- 2026-11-17
AI Technical Summary
Existing text-to-media content retrieval algorithms suffer from low retrieval accuracy, especially due to the poor retrieval quality caused by the coarse-grained visual representation of media content.
By processing the visual features of media content to generate pseudo-text information and performing feature interaction processing, more expressive visual representation information is generated, which is then combined with a content retrieval model with a hybrid tower structure for retrieval.
It improves the accuracy and efficiency of media content retrieval, enabling more accurate recall of relevant media content and reducing the need for complex calculations during the online phase.
Smart Images

Figure CN123115444A_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of computer technology, and more particularly to the field of artificial intelligence, specifically to a media content processing method, a media content processing device, a computer equipment, a computer-readable storage medium, and a computer program product. Background Technology
[0002] Media content retrieval refers to the process by which a user inputs a query text, and then retrieves and searches for related media content from a vast amount of media content based on that query text.
[0003] Currently, existing text-to-media content retrieval algorithms suffer from problems such as low retrieval accuracy. For example, in the text-to-media content retrieval process, simply retaining the dot product operation for basic text-to-media content retrieval, and utilizing coarse-grained visual representations of media content, leads to poor content retrieval quality and reduced retrieval accuracy. Therefore, how to efficiently achieve text-to-media content retrieval is an important research topic. Summary of the Invention
[0004] This application provides a media content processing method, apparatus, device, medium, and program product that can ensure the accuracy of content retrieval while improving the efficiency of content retrieval.
[0005] On one hand, embodiments of this application provide a media content processing method, the method comprising:
[0006] The process involves acquiring media content to be processed and performing visual feature processing on the media content to obtain visual feature information, which is used to characterize the visual semantics of the media content.
[0007] Based on visual feature information, pseudo-text prediction processing is performed on media content to generate pseudo-text information; the pseudo-text information is used to characterize the textual semantics of the pseudo-query text used when the media content is retrieved.
[0008] Feature interaction processing is performed on pseudo-text information and visual feature information to obtain visual representation information of media content; feature interaction processing is used to: enhance the text semantics of pseudo-query text representation in the visual semantics represented by visual feature information; visual representation information is used for content retrieval of media content.
[0009] On the other hand, embodiments of this application provide a media content processing apparatus, which includes:
[0010] The acquisition unit is used to acquire the media content to be processed and to perform visual feature processing on the media content to obtain visual feature information, which is used to characterize the visual semantics of the media content.
[0011] The processing unit is used to perform pseudo-text prediction processing on media content based on visual feature information to generate pseudo-text information; the pseudo-text information is used to characterize the text semantics of the pseudo-query text used when the media content is retrieved.
[0012] The processing unit is also used to perform feature interaction processing on pseudo-text information and visual feature information to obtain visual representation information of media content; the feature interaction processing is used to: enhance the text semantics of the pseudo-query text representation in the visual semantics represented by the visual feature information; the visual representation information is used for content retrieval of media content.
[0013] In one implementation, the processing unit, used to perform visual feature processing on the media content and obtain visual feature information, specifically performs the following:
[0014] Feature extraction is performed on media content from multiple content dimensions to obtain initial feature information for each content dimension; the initial feature information represents the visual semantics of the media content from the corresponding content dimension.
[0015] The initial feature information under each content dimension is linearly projected into the cross-modal feature space to obtain the visual sub-feature information of each initial feature information in the cross-modal feature space; the cross-modal feature space includes feature information of at least two modalities, and one initial feature information corresponds to one visual feature information;
[0016] Visual feature information is obtained by concatenating the visual sub-features of each initial feature information in the cross-modal feature space.
[0017] In one implementation, the media content is video; the processing unit, used to extract features from the media content from multiple content dimensions, specifically performs the following when obtaining initial feature information for each content dimension:
[0018] The media video is converted from a high-dimensional to a low-dimensional initial feature vector; the initial feature vector includes the initial sub-feature vector corresponding to each video frame in the media video;
[0019] Frame-level labeling information is added to each initial sub-feature vector, and the initial sub-feature vectors after adding frame-level labeling information are concatenated to obtain the initial concatenated vector;
[0020] Add video-level labeling information to the initial concatenated vector to obtain the initial labeled feature vector;
[0021] Feature extraction is performed on the initial labeled feature vector from multiple content dimensions to obtain the initial feature information for each content dimension.
[0022] In one implementation, the media content is a media video, and the visual sub-feature information includes: video-level feature information corresponding to the media video, frame-level feature information corresponding to each video frame in the media video, and image block-level feature information corresponding to each image block in each video frame.
[0023] The processing unit is used to perform feature concatenation on the visual sub-features of each initial feature information in the cross-modal feature space to obtain visual feature information. Specifically, it is used for:
[0024] Feature filtering is performed on the image block-level feature information corresponding to each image block in each video frame to obtain the filtered key image block-level feature information;
[0025] Visual feature information is obtained by concatenating the video-level feature information, the frame-level feature information corresponding to each video frame, and the key image block-level feature information.
[0026] In one implementation, the processing unit, when performing feature filtering on the image block-level feature information corresponding to each image block in each video frame to obtain the filtered key image block-level feature information, specifically performs the following:
[0027] Based on the video-level feature information corresponding to the media video, attention distribution operation is performed on the image block-level feature information corresponding to each image block in each video frame to obtain the information matrix; the information matrix includes the matching degree between each image block-level feature information and the video-level feature information;
[0028] At least one image block-level feature information that meets the matching degree requirement is selected from the information matrix and used as the key image block-level feature information after selection.
[0029] In one implementation, the processing unit is used to perform pseudo-text prediction processing on the media content based on visual feature information. Specifically, when generating pseudo-text information, it is used for:
[0030] Based on causal constraints, attention operations are performed on visual feature information to obtain key visual features corresponding to the visual feature information; causal constraints include generating pseudo-text information describing media content according to content coherence.
[0031] By mapping key visual features to the text query space, pseudo-text information corresponding to media content can be obtained.
[0032] In one implementation, the processing unit, used to perform feature interaction processing on pseudo-text information and visual feature information to obtain visual representation information of the media content, specifically performs the following:
[0033] Guided by pseudo-text information, feature weights are assigned to visual feature information to obtain visual representation information of media content; the visual semantics represented by the visual representation information match the text semantics described by the pseudo-text information.
[0034] In one implementation, the visual feature information is obtained by feature stitching together multiple visual sub-feature information, including: video-level feature information corresponding to the media video, and frame-level feature information corresponding to each video frame in the media video.
[0035] The processing unit, used for guiding visual feature information based on pseudo-text information, performs feature interaction processing on visual feature information under multiple content dimensions to obtain visual representation information of media content. Specifically, it is used for:
[0036] Using pseudo-text information as query parameters for the attention mechanism; and,
[0037] The video-level feature information corresponding to the media video and the frame-level feature information corresponding to each video frame in the media video are concatenated, and the concatenated feature information is used as the key and value parameters of the attention mechanism.
[0038] Attention operations are performed on query parameters, key parameters, and value parameters to obtain visual representation information of media content.
[0039] In one implementation, the method is executed by invoking a trained content retrieval model; the processing unit is also used for:
[0040] Retrieve the content database, which includes multiple media content items that can be retrieved;
[0041] The content retrieval model is invoked to perform feature representation on each media content, thereby obtaining the visual representation information of each media content; wherein, feature representation includes visual feature processing, pseudo-text prediction processing, and feature interaction processing;
[0042] The visual representation information of each media content is cached.
[0043] In one implementation, the processing unit is further used for:
[0044] Receive content retrieval requests, which carry query text, which is text describing the media content to be queried;
[0045] In response to a content retrieval request, text features are extracted from the query text to obtain the textual semantic information of the query text;
[0046] Calculate the similarity between the semantic information of the text and the visual representation information of each media content;
[0047] Media content that meets at least one similarity threshold will be used as the retrieved target media content.
[0048] In one implementation, the content retrieval model is obtained by training an initial content retrieval model; the model training process includes:
[0049] Obtain the training dataset, which includes text content pairs. Each text content pair includes one sample media content and at least one sample query text.
[0050] An initial content retrieval model is used to perform visual feature processing on the sample media content to obtain predicted visual feature information;
[0051] An initial content retrieval model is used to perform pseudo-text prediction processing on the sample media content based on the predicted visual feature information, thereby generating the predicted pseudo-text information corresponding to the sample media content.
[0052] An initial content retrieval model is used to perform feature interaction processing on predicted pseudo-text information and predicted visual feature information to obtain the predicted visual representation information corresponding to the sample media content.
[0053] Based on the sample query text corresponding to the sample media content, predict pseudo-text information and predict visual representation information to optimize the initial content retrieval model and obtain a trained content retrieval model.
[0054] In one implementation, the processing unit, used to optimize the initial content retrieval model based on the sample query text corresponding to the sample media content, predicts pseudo-text information and the predicted visual representation information, to obtain a trained content retrieval model, specifically is used for:
[0055] Following the direction of reducing the gap between the sample query text corresponding to the sample media content and the predicted pseudo-text information, the initial content retrieval model is optimized in the first stage to obtain the intermediate content retrieval model.
[0056] Following the direction of reducing the gap between the sample query text corresponding to the sample media content and the predicted visual representation information, the intermediate content retrieval model is optimized in the second stage to obtain the trained content retrieval model.
[0057] In another aspect, embodiments of this application provide a computer device, the computer device comprising:
[0058] A processor, adapted to execute computer programs;
[0059] A computer-readable storage medium storing a computer program that, when executed by a processor, implements the aforementioned media content processing method.
[0060] In another aspect, embodiments of this application provide a computer-readable storage medium storing a computer program adapted to be loaded by a processor and executed by the aforementioned media content processing method.
[0061] In another aspect, embodiments of this application provide a computer program product, which includes a computer program that, when executed by a processor, implements the aforementioned media content processing method.
[0062] In this embodiment, after obtaining the media content to be processed, visual feature processing is performed on the media content to obtain visual feature information. This visual feature information represents the visual semantics (or content semantics) of the media content from the perspective of the media content itself. Further, pseudo-text prediction processing is performed on the media content based on the visual feature information to generate pseudo-text information of the media content. This pseudo-text information represents the textual semantics of the pseudo-query text used when the media content is retrieved, and this textual semantics typically describes a more granular content semantics of the media content. Thus, feature interaction processing is performed on the pseudo-text information and the visual feature information of the media content to obtain visual representation information of the media content. This visual representation information has a more refined semantic expression capability than the visual feature information; specifically, the represented visual semantics matches better with the textual semantics of the pseudo-query text that the user might input. Therefore, in the process of content retrieval based on query text, high-precision content retrieval can be achieved based on the visual representation information that integrates pseudo-text information and visual feature information. As described above, in the process of generating visual representation information of media content, the embodiments of this application can integrate pseudo-text information of pseudo-query text that may be used during media content retrieval. This enhances (or highlights) the textual semantics represented by the pseudo-text information within the visual semantics represented by the visual feature information of the media content itself, thereby generating visual representation information that is more consistent with the pseudo-query text. Consequently, in the actual content retrieval process, media content that is more relevant to the query text can be retrieved based on the visual representation information, thus improving the accuracy and precision of content retrieval. Attached Figure Description
[0063] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0064] Figure 1 This is a schematic diagram of a video frame in a media video.
[0065] Figure 2aThis is a schematic diagram of an existing single-tower content retrieval model.
[0066] Figure 2b This is a schematic diagram of an existing dual-tower content retrieval model.
[0067] Figure 2c This is a schematic diagram of a content retrieval model with a hybrid tower structure provided in an exemplary embodiment of this application;
[0068] Figure 3 This is a schematic diagram of the architecture of a media content processing system provided in an exemplary embodiment of this application;
[0069] Figure 4 This is a schematic flowchart of a media content processing method provided in an exemplary embodiment of this application;
[0070] Figure 5 This is a schematic diagram illustrating a method for generating visual representation information of media content based on a content retrieval model, as provided in an exemplary embodiment of this application.
[0071] Figure 6 This is a schematic diagram illustrating an exemplary embodiment of this application of adding frame-level and video-level tagging information to an initial feature vector;
[0072] Figure 7 This is a schematic diagram illustrating the generation of pseudo-text information using a pseudo-query generator, provided in an exemplary embodiment of this application.
[0073] Figure 8 This is a schematic diagram illustrating feature filtering of image block-level feature information using an information selector, provided by an exemplary embodiment of this application.
[0074] Figure 9 This is a flowchart illustrating another media content processing method provided in an exemplary embodiment of this application;
[0075] Figure 10 This is a schematic diagram of a model training process provided in an exemplary embodiment of this application;
[0076] Figure 11 This is a schematic diagram of experimental data provided in an exemplary embodiment of this application;
[0077] Figure 12 This is a schematic diagram of the structure of a media content processing apparatus provided in an exemplary embodiment of this application;
[0078] Figure 13 This is a schematic diagram of the structure of a computer device provided in an exemplary embodiment of this application. Detailed Implementation
[0079] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of this application.
[0080] This application proposes a media content processing scheme, specifically a text-to-media content retrieval scheme based on a content retrieval model. For ease of understanding, the technical terms and related concepts involved in the media content processing scheme are briefly introduced below:
[0081] I. Text-to-media content retrieval.
[0082] Media content refers to various forms of carriers or presentations that carry information; media content may include, but is not limited to, images, videos, and animations, and is used to store, present, process, or transmit information. In this embodiment, media video is used as an example for explanation; media video, often simply referred to as video, is a file composed of at least two video frames (or image frames) sequentially linked together. That is, a video frame is the smallest or most basic unit of video; when playing media video, multiple video frames are continuously output in the order of their playback time. When the continuous video frame changes exceed 24 frames per second, based on the principle of visual persistence, the human eye perceives a smooth and continuous visual effect from each video frame.
[0083] It is worth noting that adjacent video frames in a series of video frames output sequentially according to their playback time are continuous or coherent in terms of image content; multiple consecutively output video frames often express a coherent visual semantic through continuous image content. For example... Figure 1 As shown, suppose the video contains N consecutive video frames, where N is a positive integer, such as N≥4. The N video frames include consecutive video frames i→i+1→i+2→i+3. The image content of video frame i is a jumping boy, the image content of video frame i+1 is the boy throwing a ball, the image content of video frame i+2 is the ball leaving the boy and falling, and the image content of video frame i+3 is the ball falling. Then, when playing video frames i→i+1→i+2→i+3 consecutively, the coherent visual semantics can be roughly as follows: a boy jumps and throws a ball, and the ball falls after being thrown.
[0084] Furthermore, text-to-media content retrieval refers to the process of retrieving and querying media content related to a query text (or search text) from a vast amount of media content. The query text can be understood as a piece of text describing the media content, including keywords representing key elements of the media content; the aim is to retrieve corresponding media content based on the text described in the query text. For example, Figure 1 The query text corresponding to the media video shown might be "a boy playing ball".
[0085] II. Content Retrieval Model.
[0086] Content retrieval models are used by text-to-media content retrieval algorithms to implement AI (Artificial Intelligence) models that retrieve media content based on query text. Artificial intelligence utilizes digital computers or computers-controlled machines to simulate, extend, and expand human intelligence, perceiving the environment, acquiring knowledge, and using that knowledge to obtain optimal results—a set of theories, methods, technologies, and application systems. Machine learning models within the field of artificial intelligence are network models obtained through training using machine learning. Machine learning is a multidisciplinary field involving probability theory, statistics, approximation theory, convex analysis, algorithm complexity theory, and many other disciplines. Machine learning typically includes techniques such as artificial neural networks, belief networks, reinforcement learning, transfer learning, inductive learning, formulaic learning, and deep learning (DL).
[0087] This application's embodiments mainly relate to content retrieval models in machine learning technology; currently, the content retrieval models used in text-to-media content retrieval algorithms are classified according to their model structure into: single-tower content retrieval models and dual-tower content retrieval models. Among them:
[0088] ① A schematic diagram of the content retrieval model with a single-tower structure can be found in [reference needed]. Figure 2a The single-tower content retrieval model includes a text encoder, a video encoder (or image encoder), a feature interaction module, and a dot product operation module. The text encoder extracts text features from the input query text; the video encoder extracts visual features from the input media content (specifically, video content); the feature interaction module performs feature interaction between the extracted text features and the extracted video features; and the dot product operation module calculates the similarity between the interacted text and visual features to determine if the media content matches the query text. See also... Figure 2aAs shown in the single-tower structure, each text-to-media content retrieval process requires feature interaction between the query text and the media content through the feature interaction module. The feature interaction module has high complexity and requires a large amount of computing resources and storage space, which significantly reduces the retrieval efficiency of text-to-media content.
[0089] ② A schematic diagram of the dual-tower content retrieval model can be found in [reference needed]. Figure 2b The dual-tower content retrieval model includes a text encoder, a video encoder (or image encoder), and a dot product operation module. The functions of each module in the dual-tower content retrieval model are the same as those of their corresponding modules in the single-tower content retrieval model, and will not be elaborated upon here. (Comparison) Figure 2b and Figure 2a It is evident that the dual-tower structure, compared to the single-tower structure, lacks a feature interaction module but retains a dot product operation module. The dual-tower structure relies solely on the dot product operation of the text features extracted from the query text by the text encoder and the video features extracted from the media content by the video encoder to achieve content retrieval. However, the feature extraction capabilities of the text encoder and the video encoder are limited, thereby restricting the expressive power of the text features and video features, and consequently reducing the retrieval accuracy from text to media content.
[0090] To improve the retrieval accuracy of text-to-media content retrieval, this application presents a novel content retrieval model structure—the hybrid tower structure—combining single-tower and dual-tower structures. For example, one model structure of the hybrid tower structure content retrieval model can be found here. Figure 2c As shown in (1), the hybrid tower structure includes: a text encoder, a video encoder, a pseudo-query generator, a feature interaction module, and a dot product operation module. The functions of the text encoder, video encoder, and dot product operation module can be found in [reference needed]. Figure 2a As shown in the description, the pseudo-query generator is used to generate pseudo-text information that may describe the media content based on the media content. This pseudo-text information represents the textual semantics of the pseudo-query text. Figure 2c The hybrid tower structure content retrieval model in (1) significantly improves retrieval accuracy compared to the dual-tower and single-tower structures by introducing a pseudo-query generator and a feature interaction module. Furthermore, one model structure of the hybrid tower structure content retrieval model can be found in [reference needed]. Figure 2c (2) shown; Figure 2c As shown in (1) and Figure 2c The difference shown in (2) is: Figure 2c In section (1), only the part of the video encoder that extracts video features from the media content is executed offline. Figure 2c In (2), the entire execution process of visual representation information of media content is executed offline; therefore, Figure 2cThe content retrieval model of the hybrid tower structure in (2) not only has Figure 2c The hybrid tower structure in (1) has high retrieval accuracy and greatly improves the retrieval efficiency of online text-to-media content retrieval compared to the feature interaction between online text and media content in the single tower structure.
[0091] Based on this, the media content processing scheme provided in this application is specifically a text-to-media content retrieval scheme based on a content retrieval model with a hybrid tower structure. This scheme can generate pseudo-text information based on the media content, representing the pseudo-query text that may be used when the content is retrieved. Furthermore, it performs feature interaction between the pseudo-text information and the visual feature information of the media content itself to obtain more expressive visual representation information. This allows for efficient retrieval of media content more relevant to the user's actual query text, improving the accuracy of content retrieval. Further, it supports feature interaction between the pseudo-text information and the visual feature information of the media content itself, as well as the generation of visual representation information of the media content, in the offline stage, thus eliminating the need for complex feature interaction in the online stage and improving content retrieval efficiency.
[0092] The general process of enhancing the visual representation information of media content based on pseudo-text information may include:
[0093] The process involves acquiring media content that requires feature representation. First, visual feature processing is performed to extract the visual features inherent to the content, representing its visual semantics. For example, if the content is video, the visual features represent the video semantics. Then, based on these visual features, pseudo-text prediction is performed to generate pseudo-text information representing possible pseudo-query texts. This pseudo-text information represents the textual semantics of the pseudo-query text. Finally, feature interaction processing is applied to the pseudo-text information and visual feature information to obtain the visual representation of the media content. This feature interaction process enhances or highlights the textual semantics represented by the pseudo-text information within the visual semantics represented by the media content's visual features, ensuring that the visual representation better expresses the visual semantics related to the pseudo-query text. Thus, when retrieving content based on the user's actual query text (representing the user's genuine query needs, as opposed to the generated pseudo-text information), the more expressive visual representation information can be used, improving the accuracy of content retrieval.
[0094] Therefore, on the one hand, the embodiments of this application support the integration of pseudo-text information from pseudo-query text that may be used during media content retrieval during the process of feature representation of media content. This enhances (or highlights) the textual semantics represented by the pseudo-text information within the visual semantics represented by the visual feature information of the media content itself, thereby generating visual representation information that better matches the pseudo-query text. Consequently, in the actual content retrieval process, media content more relevant to the query text can be retrieved based on the visual representation information, improving the accuracy and precision of content retrieval. On the other hand, the process of generating pseudo-text information from media content and the feature interaction between pseudo-text information and visual feature information to generate visual representation information can be performed offline. This avoids the complex calculations of real-time pseudo-text information generation and feature interaction during the online content retrieval stage, thereby improving the retrieval efficiency of content retrieval.
[0095] The media content processing solution provided in this application, as a text-to-media content retrieval solution, can achieve high-precision content retrieval while improving content retrieval efficiency. This makes the media content processing solution provided in this application applicable to various content retrieval scenarios for text-to-media content retrieval. Content retrieval scenarios may include, but are not limited to: ① Video retrieval scenarios, where videos may include, but are not limited to: long videos, short videos, and movies / TV series, etc.; In video retrieval scenarios, it supports generating pseudo-text for each candidate media video in the video library during the offline stage, and performing feature interaction between pseudo-text information and visual feature information of candidate media videos, etc., to cache the visual representation information of the generated candidate media videos. In the actual video retrieval process, content retrieval is performed based on the real video query text input by the user and the mixed visual representation information of candidate media videos, thereby improving the accuracy of video retrieval. ② Image retrieval scenario: Supports offline generation of pseudo-text for each candidate media image in the image library, and feature interaction between pseudo-text information and visual feature information of the candidate media image. This caches the visual representation information of the generated candidate media images. During actual video retrieval, content retrieval is performed based on the user-inputted real image query text and the mixed visual representation information of the candidate media images, improving the accuracy of video retrieval. It should be noted that the aforementioned video library is a database storing candidate media videos, and the image library is a database storing candidate media images. In some cases, the video library and image library can be integrated into the same database, i.e., this database stores both media videos and media images, thus enabling retrieval of both media images and media videos based on this database. This application embodiment does not limit the content format of the media content stored in the database.
[0096] Furthermore, the media content processing solution provided in this application embodiment can be applied to any content retrieval product that can realize text-to-media content retrieval. Content retrieval products may include, but are not limited to, video applications, image applications, and social applications; an application refers to a computer program designed to perform one or more specific tasks. Specifically: ① A video application is an application capable of playing video frames. Video applications may include, but are not limited to, applications capable of playing short videos (videos are often short, such as a few seconds or minutes), applications capable of playing long videos (such as movies or TV series, which often have longer playback times), and applications with movie / TV series playback permissions, etc. ② An image application is an application capable of creating, updating, and retrieving images; users can upload, modify, retrieve, or download images in this image application. ③ A social application is an application that enables instant messaging and social interaction based on the internet; in some social scenarios within a social application, users may have a need for media content retrieval, such as searching / finding historical media videos or images in social conversation records based on text retrieval / search / query. It is worth noting that the applications mentioned above can be classified according to their operating method into: ① clients that require downloading and installing an installation package to run in the terminal; or ② mini-programs that can be used without downloading and installing, which are usually sub-programs of the client; ③ web (World Wide Web) applications that are opened through a browser, etc.
[0097] It should be understood that the above description is merely an exemplary content retrieval scenario and content retrieval product provided in the embodiments of this application, and does not limit the content retrieval scenario and content retrieval product of the media content processing solution provided in the embodiments of this application. Computer devices deployed with the media content processing solution provided in the embodiments of this application can provide highly accurate and efficient text-to-media content retrieval services in various content retrieval scenarios and content retrieval products, demonstrating high value and practicality, and helping to improve users' content retrieval experience and satisfaction.
[0098] To facilitate understanding of the resource processing scheme provided in the embodiments of this application, the following is combined with... Figure 3 The media content processing system shown introduces the media content processing solution. Please refer to [link / reference]. Figure 3 , Figure 3 This is a schematic diagram of the architecture of a media content processing system provided in an exemplary embodiment of this application; the media content processing system includes terminals and servers. This application embodiment does not limit the number or type of terminals and servers included in the media content processing system, but this is only described herein. For example, Figure 3The media content processing system shown includes a terminal 301 and a server 302; the following describes... Figure 3 The devices in the system shown are described below, including:
[0099] Terminal 301 is a user-held terminal device that runs a content retrieval product (such as a video application) to facilitate text-to-media content retrieval by the user. The terminal device may include, but is not limited to, smartphones (such as smartphones running Android or iOS), tablets, portable personal computers, mobile internet devices (MIDs), in-vehicle devices, head-mounted devices, smart TVs, or smart home devices. This application does not limit the type of terminal; this is only a description of the specific type.
[0100] Server 302 is the backend server corresponding to terminal 301, specifically the backend server corresponding to the content retrieval product running on terminal 301. The content retrieval product can perform text-to-media content retrieval through server 302. Thus, server 302 interacts with the content retrieval product deployed on terminal 301 to provide computing and application service support for the content retrieval product deployed on terminal 301. The server can be a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms. Furthermore, server 302 is associated with a content database, which is a database used to store media content, i.e., the content database includes multiple media content items that are allowed to be retrieved. This content database can be deployed on a backend device independent of server 302, or it can be deployed within server 302, or it can be a distributed database; there are no limitations on this. In addition, server 302 is also used to train the content retrieval model and deploy the trained content retrieval model.
[0101] The various devices included in the media content processing system can communicate directly or indirectly through wired or wireless means. This application embodiment does not limit the communication method between devices; for example, the communication method between devices may include, but is not limited to, HTTP requests, remote procedure calls, sockets, and content sharing.
[0102] The media content processing solution provided in this application embodiment can be derived from... Figure 3 The media content processing system shown executes the action either on the terminal or the server, or jointly on both the terminal and the server; the following is in conjunction with... Figure 3 The system shown uses the joint execution of terminal 301 and server 302 as an example to introduce the process of text-to-media content retrieval in a content retrieval scenario; where:
[0103] (1) Model training phase of the content retrieval model: Server 302 obtains the training dataset and uses the training dataset to train the initial content retrieval model, resulting in a trained content retrieval model. The trained content retrieval model is deployed on server 302.
[0104] (2) Offline generation stage of visual representation information of media content: Server 302 retrieves each media content from the content database and calls the trained content retrieval model to perform feature representation on each media content to obtain the visual representation information of each media content. The feature representation for any media content includes: visual feature processing, pseudo-text prediction processing, and feature interaction processing. The detailed process may include: Server 302 retrieves the media content to be processed from the content database and performs visual feature processing on the media content to extract the visual feature information of the media content itself. Then, based on the visual feature information of the media content, Server 302 performs pseudo-text prediction processing on the media content to generate pseudo-text information of possible pseudo-query texts corresponding to the media content. This pseudo-text information represents the textual semantics of the pseudo-query text. Finally, Server 302 performs feature interaction processing on the pseudo-text information and visual feature information to obtain the visual representation information of the media content. Server 302 performs the above feature representation on each media content in the content database to obtain the visual representation information of each media content and caches the visual representation information of each media content for direct use later.
[0105] (3) Online retrieval stage of text-to-media content retrieval: Terminal 301 receives the actual query text input by the user, which describes the main content information of the media content to be retrieved; then, terminal 301 generates a content retrieval request based on the query text and transmits the content retrieval request to server 302. In response to the content retrieval request, server 302 performs a similarity calculation on the textual semantics of the query text and the visual representation information of each cached media content, and selects the media content with high similarity as the retrieved target media content. Server 302 returns the target media content to terminal 301 so that the user can view the target media content related to the query text in a timely manner.
[0106] Therefore, on the one hand, during the offline phase, server 302 invokes the content retrieval model to perform feature representation on each media content in the content database and caches the visual representation information of each candidate media content. Compared to the feature interaction between text features and video features online under a single-tower structure, this greatly reduces the computational resources required for online retrieval, thereby improving retrieval efficiency. On the other hand, during the process of server 302 invoking the content retrieval model to perform feature representation on media content, it pre-integrates the pseudo-text information of possible pseudo-query texts corresponding to the media content into the visual feature information of the media content. This ensures that the visual representation information of the media content is as relevant as possible to the possible query texts during content retrieval, thereby improving the retrieval accuracy during online content retrieval.
[0107] Based on the above brief introduction of the media content processing scheme and media content processing system provided in the embodiments of this application, the following points should also be noted:
[0108] ①The embodiments of this application mentioned above Figure 3 The system architecture shown is for the purpose of more clearly illustrating the technical solutions of the embodiments of this application and does not constitute a limitation on the technical solutions provided in the embodiments of this application. Those skilled in the art will understand that, with the evolution of system architecture and the emergence of new business scenarios, the technical solutions provided in the embodiments of this application are also applicable to similar technical problems. That is to say, Figure 3 The schematic diagram of the media content processing system shown is an exemplary schematic diagram; in actual applications, the number and types of devices included in the media content processing system may vary, and the schematic diagram of the media content processing system in this application embodiment is not limited.
[0109] ② In this application embodiment, the collection and processing of relevant data should strictly comply with the requirements of relevant laws and regulations. Obtaining personal information requires the knowledge or consent of the individual (or a legal basis for information acquisition), and subsequent data use and processing should be carried out within the scope of laws, regulations, and the authorization of the personal information subject. For example, when this application embodiment is applied to specific products or technologies, such as when terminal 301 sends the user's actual query text to server 302 in the form of a content retrieval request, the user's permission or consent is required, and the collection, use, and processing of relevant data must comply with the relevant laws, regulations, and standards of the relevant regions.
[0110] Based on the media content processing scheme described above, this application proposes a more detailed media content processing method. The media content processing method proposed in this application will be described in detail below with reference to the accompanying drawings.
[0111] Figure 4 The illustration shows a flowchart of a media content processing method provided in an exemplary embodiment of this application; the media content processing method can be executed by a computer device in the aforementioned system; the media content processing method may include, but is not limited to, steps S401-S403:
[0112] S401: Obtain the media content to be processed, and perform visual feature processing on the media content to obtain visual feature information.
[0113] The media content to be processed is retrieved from a content database on a computer device, and its corresponding visual representation information needs to be generated offline. After training the content retrieval model and deploying it on the computer device, the computer device retrieves each piece of media content from the content database and performs feature representation to generate corresponding visual representation information; in this embodiment of the application... Figure 4 The method embodiment shown uses the example of a computer device calling a content retrieval model to perform feature representation on a single media content in order to generate its corresponding visual representation information. This is explained here.
[0114] In specific implementation, after the computer device acquires the media content to be processed, it performs visual feature processing on the media content to obtain visual feature information used to characterize the visual semantics of the media content. To improve the accuracy of the visual feature information in expressing the visual semantics of the media content, this application embodiment supports visual feature processing of the media content from multiple granularities or multiple content dimensions to obtain initial sub-feature information under multiple content dimensions. The visual feature information is then jointly represented by these multiple initial sub-feature information, thereby improving the semantic expressive power of the visual feature information. The general process of performing visual feature processing on the media content from multiple content dimensions may include the following steps s11-s13:
[0115] s11: Extract features from media content from multiple content dimensions to obtain initial feature information for each content dimension.
[0116] Depending on the content format of the media content, the type and number of content dimensions for feature extraction also vary. Optionally, when the media content is a media image, the multiple content dimensions corresponding to the media image can include, but are not limited to, image patch level and image level. Specifically: ① An image patch, also known as a plot, is a region cropped from a media image; feature extraction from the patch allows for the extraction of local features (such as image edges, individual elements, etc.) within the media image, helping to extract more detailed local features. ② Image level refers to treating the media image as a whole and performing feature extraction on the entire media image, enabling the extraction of global features and aiding in understanding the global feature semantics of the media image. Optionally, when the media content is a media video, the multiple content dimensions of the media image can include, but are not limited to, video level, frame level, and image patch level. Among them, video level refers to the entire media video as the object of feature extraction, which can realize the global video semantics of the media video; frame level is similar to the concept of image level mentioned above, which refers to the entire video frame as the object of feature extraction, aiming to analyze the frame semantics of each video frame in the media video; the relevant content of image block level can be found in the aforementioned description, and will not be repeated here.
[0117] It should be noted that media content can be in other forms besides media videos and media images; when the media content is in other forms, the content dimensions corresponding to the media content may not be limited to the above-mentioned content dimensions, and this application embodiment does not limit this.
[0118] For ease of explanation, the following uses media video as an example to introduce the process of extracting features from media video from multiple content dimensions to obtain initial feature information for each content dimension. Specifically, after the computer device acquires the media video, it converts the high-dimensional media video into a low-dimensional initial feature vector. Specifically, each video frame included in the media video is converted into a low-dimensional initial feature vector. The purpose is to retain the main image features of each video frame while reducing data complexity, facilitating better subsequent image processing. In other words, the initial feature vector includes the initial sub-feature vector corresponding to each video frame in the media video, such as each initial sub-feature vector being a 1*49 vector.
[0119] Then, to achieve feature extraction from media videos across multiple content dimensions, the computer device adds frame-level labeling information to each initial sub-feature vector. The initial sub-feature vectors with added frame-level labeling information are treated as a whole for frame-level feature extraction. The initial sub-feature vectors with added frame-level labeling information are then concatenated to obtain an initial concatenated vector. For example, if the number of initial sub-feature vectors is 12 (i.e., the media video includes 12 video frames), then each initial sub-feature vector with added frame-level labeling information will be represented as a 1*50 vector. Concatenating these 12 initial sub-feature vectors with added frame-level labeling information yields an initial concatenated vector of size 12*(1+49). Furthermore, after adding frame-level labels to each initial sub-feature, the computer device also adds video-level labeling information to the initial concatenated vector corresponding to the entire media video, obtaining an initial labeled feature vector. The purpose of adding video-level labeling information is to indicate the need for video-level feature extraction when the video-level labeling information is detected.
[0120] Finally, the computer device extracts features from the initial marker feature vector from multiple content dimensions to obtain initial feature information for each content dimension. Specifically, when the computer device calls the content retrieval model to perform visual feature processing on the initial marker feature vector, it performs frame-level feature extraction on the initial sub-feature information related to the frame-level marker information when frame-level marker information is identified, and it also performs image patch-level feature extraction on the initial sub-feature information. For example, before adding frame-level marker information to the initial sub-feature vector, frame-level feature extraction is performed on the adjacent and summed initial sub-feature vectors of the frame-level marker information to obtain initial feature information in the frame dimension—frame-level initial feature information. The number of frame-level feature information is the same as the number of video frames included in the media video; one video frame corresponds to one frame-level feature information, which is used to characterize the frame semantics of the corresponding video frame. The computer device also performs image patch-level feature extraction on the adjacent and summed initial sub-feature vectors of the frame-level marker information to obtain initial feature information in the image patch dimension—image patch-level initial feature information, which is used to characterize the image patch semantics of image patches in the video frame. Similarly, when a computer device recognizes video-level tag information, it performs video-level feature extraction on the initial splicing vector related to the video-level tag information to obtain initial feature information in the video-level dimension—video-level initial feature information. This video-level feature information is used to characterize the video-level semantics of the entire media video.
[0121] s12: Linearly project the initial feature information under each content dimension to the cross-modal feature space to obtain the visual sub-feature information of each initial feature information in the cross-modal feature space.
[0122] To facilitate the subsequent feature interaction between textual and visual features, this embodiment, after calculating the initial feature information of the media content in each content dimension, projects the initial feature information onto a cross-modal feature space using linear projection, obtaining visual sub-feature information for each initial feature information in the cross-modal feature space, with one initial feature information corresponding to one visual feature information. Based on this, the initial feature information in the frame dimension—frame-level initial feature information—is linearly projected onto the cross-modal feature space to obtain visual sub-feature information in the frame-level dimension—frame-level feature information; similarly, the initial feature information in the image patch dimension—image patch-level initial feature information—is linearly projected onto the cross-modal feature space to obtain visual sub-feature information in the frame-level dimension, represented as image patch-level feature information; similarly, the initial feature information in the video-level dimension—video-level initial feature information—is linearly projected onto the cross-modal feature space to obtain visual sub-feature information in the video-level dimension—video-level feature information. In this context, the cross-modal feature space refers to a feature space that accommodates feature information from multiple modalities. That is, the cross-modal feature space includes feature information from at least two modalities. By mapping data from different perceptual modalities (such as images, text, speech, video, etc.) into a shared feature space, the feature information from these different modalities can be compared, correlated, and analyzed in the shared cross-modal feature space.
[0123] s13: Perform feature concatenation on the visual sub-features of each initial feature information in the cross-modal feature space to obtain visual feature information.
[0124] After obtaining the initial feature information of media content under multiple content dimensions and the visual sub-feature information in the cross-modal feature space, the embodiments of this application support the direct feature splicing of visual sub-feature information under multiple content dimensions. The aim is to splice visual sub-feature information under different content dimensions and comprehensively utilize various content dimensions to jointly express a more complete, comprehensive and richer visual semantics of media content.
[0125] Furthermore, as mentioned above, computer devices can acquire visual sub-feature information in the cross-modal feature space, including: video-level feature information corresponding to the media video, frame-level feature information corresponding to each video frame in the media video, and image block-level feature information corresponding to each image block in each video frame. Considering that the number of image block-level feature information in the image block dimension of media content is relatively large, such as the sum of the number of image blocks divided in each video frame of the media video; if there are 12 video frames, and each video frame is divided into 3 image blocks, then each image block corresponds to one image block-level feature information, and the number of image block-level feature information is 12*3=36.
[0126] A large amount of image block-level feature information contains redundancy and noise. To reduce computational complexity, this application embodiment supports feature filtering of image block-level feature information before feature stitching of visual sub-feature information under multiple content dimensions. This aims to select key image block-level feature information for feature stitching, which not only preserves important image block-level feature information but also greatly avoids information redundancy and reduces computational complexity. The feature filtering process generally includes: a computer device performing feature filtering on the image block-level feature information corresponding to each image block in each video frame to obtain the filtered key image block-level feature information. Then, the video-level feature information, the frame-level feature information corresponding to each video frame, and the key image block-level feature information are stitched together to obtain the visual feature information.
[0127] In summary, based on the aforementioned steps s11-s13, visual feature processing of media content can be performed from multiple granularities or multiple content dimensions, aiming to extract global and local features of media content and improve the comprehensiveness, richness, and completeness of the visual feature information of media content.
[0128] S402: Based on visual feature information, perform pseudo-text prediction processing on media content to generate pseudo-text information.
[0129] After obtaining the multi-granularity visual feature information of the media content in step S401, this embodiment of the application designs a pseudo-query generator in the content retrieval model. This generator receives the multi-granularity visual feature information as input and generates (or reconstructs) fine-grained and discriminative pseudo-text information (or pseudo-text query representation). The aim is to generate pseudo-text information that corresponds to the pseudo-query text predicted for use by the user when the media content is retrieved. This pseudo-text information characterizes the textual semantics of the predicted pseudo-query text when the media content is retrieved, thereby enhancing the expressive power of the media content's visual representation information by subsequently integrating the pseudo-text information into the visual feature information of the media content. It should be noted that the pseudo-query generator generates pseudo-text information that characterizes the textual semantics of the pseudo-query text; it does not generate the pseudo-query text itself.
[0130] The pseudo-query generator is based on a causal attention mechanism to predict and generate pseudo-text information. This causal attention mechanism ensures that the content retrieval model, when processing visual feature information of a sequence, only considers features prior to the current position, thus avoiding the leakage of feature information after the current position and maintaining the orderliness and coherence of the generation process. Specifically, the pseudo-query generator performs attention operations on visual feature information based on causal constraints to obtain the key visual features corresponding to the visual feature information. These causal constraints are the conditions specified by the causal attention mechanism, including but not limited to generating pseudo-text information describing media content according to content coherence. Specifically, the causal attention mechanism needs to ensure that the content retrieval model, when predicting the next word in the pseudo-text information, only considers already generated words, thus maintaining the orderliness and coherence of the pseudo-text information generation process. Then, the key visual features in the visual feature space are mapped to the text query space to obtain the pseudo-text information corresponding to the media content. The visual feature space can be understood as the feature space used to represent and process visual features, and correspondingly, the text query space can be understood as the feature space used to represent and process text features.
[0131] S403: Perform feature interaction processing on pseudo-text information and visual feature information to obtain visual representation information of media content.
[0132] Feature interaction processing is mainly used to enhance the text semantics of pseudo-text information representation in the visual semantics represented by visual feature information, so that the visual representation information after feature interaction processing can not only represent the visual semantics of media content, but also be related to the text semantics represented by pseudo-text information.
[0133] In this embodiment, the computer device performs feature interaction processing on pseudo-text information and visual feature information. Specifically, guided by the pseudo-text information, feature weights are assigned to the visual feature information. The feature weight assignment aims to use the pseudo-text information as guidance to perform attention pooling on a series of visual feature information. This allows the device to learn to highlight video frames in the media video that are related to the textual semantics represented by the pseudo-text information, and to suppress video frames in the media video that are unrelated to the textual semantics represented by the pseudo-text information, ensuring that the visual semantics represented by the obtained visual representation information matches the textual semantics described by the pseudo-text information. For example, the textual semantics represented by the pseudo-text information is... Figure 1 The semantics expressed by video frames i to i+3 in the multi-frame video are shown. When performing feature interaction processing, video frames i to i+3 will be highlighted according to the given pseudo-text information, while video frames other than video frames i to i+3 in the media video will be suppressed.
[0134] It should be noted that steps S401-S403 are executed by the computer device calling the content retrieval model of the hybrid tower structure; to better understand the specific implementation process of the content retrieval model executing steps S401-S403, the following will be combined with... Figure 5 The content retrieval model structure shown provides a detailed explanation of the specific implementation process described in steps S401-S403. For example... Figure 5 As shown, the content retrieval model includes a video encoder, a pseudo-query generator, and a fine-grained interaction module. The following describes the specific implementation process of steps S401-S403 of each module in the content retrieval module for media video, using media content as the example.
[0135] (1) A video encoder can be represented as φ f (·), which can be the video encoder in CLIP-Vip (Contrastive Language-Image Pre-Training, a multimodal machine learning model). This video encoder is a VisionTransformer (ViT) structure. The VisionTransformer structure can segment an image (or video frame) into fixed-size image patches and flatten the image patches from a high dimension to a low-dimensional initial feature vector. Considering that the CLIP-Vip ViT structure does not have frame-level feature extraction capabilities, this application embodiment supports introducing frame-level frame-level token information ([CLS]token) into the CLIP-Vip ViT structure, thereby enabling the CLIP-Vip ViT structure to output frame-level initial feature information in the frame dimension. In practical applications, the video encoder φ v (·) Receive the media video to be processed and divide the media video into a video segment composed of m video frames, v = {f1, f2, ..., fm}. m}, where m is a positive integer. Then, the video segment is input into the ViT structure. The ViT structure converts each frame of the video segment into a low-dimensional initial feature vector, and adds frame-level and video-level labeling information, resulting in an initial labeled feature vector. For example, a diagram illustrating the addition of frame-level and video-level labeling information to the initial feature vector can be found in [reference needed]. Figure 6 ;like Figure 6As shown, each video frame corresponds to an initial feature vector, and a frame-level marker [CLS] token is added before each initial feature vector. Additionally, video-level marker information (e.g., 4 [CLS] tokens) is added before the initial spliced vector corresponding to m video frames.
[0136] Furthermore, feature extraction is performed on the initial token feature vector from the video-level, frame-level, and image patch dimensions to obtain the initial feature information under each content dimension, which is the video-level initial feature information (video-level[CLS] token), denoted as x'. v The initial frame-level feature information (frame-level[CLS] token) is denoted as x' f Let x' be the initial feature information at the image patch level. p Furthermore, the video-level initial feature information, frame-level initial feature information, and image patch-level initial feature information are mapped to the cross-modal feature space (or cross-modal shared space) through the visual projection layer in CLIP-ViP, resulting in the video-level feature information x corresponding to the video-level initial feature information. v The frame-level feature information x corresponding to the initial frame-level feature information f and image block-level initial feature information x p The aforementioned video encoder φ v The series of operations performed (·) can be represented by the following formula:
[0137]
[0138] in, This indicates that video-level feature information belongs to The number 4 represents the number of [CLS] tokens for video-level tagging information. Represents frame-level feature information Where m is the number of video frames, and d is the embedding dimension in the cross-modal feature space. Indicates that the image block-level feature information belongs to Where n is the number of image blocks (patch tokens) in each video frame.
[0139] (2) The pseudo-query generator can be represented as φ g(·), the pseudo-query generator includes at least a causal attention-based query generator; in this case, the pseudo-query generator is a query generator, which includes a transformer based on the causal attention mechanism. The query generator receives visual feature information after feature concatenation, which is represented as [x v ,x f ,x p [and generates fine-grained and discriminative (or distinguishable) pseudo-text information based on visual feature information.] p Among them, pseudo-text information t p Having discriminative or distinguishable properties means that pseudo-text information t p The represented textual semantics are related to the visual semantics of some video frames in m-frame video frames, aiming to distinguish or identify some video frames from m-frame video frames. Among them, the pseudo-textual information t p The pseudo-text prediction and generation process can be represented as:
[0140] t p =φ g ([x v ,x f ,x p ])
[0141] in, This represents the generated pseudo-text information, or pseudo-query features; It represents the visual feature information after stitching, including video-level feature information, frame-level feature information and image block-level feature information.
[0142] Furthermore, when performing feature filtering on a large amount of image block-level feature information, the pseudo-query generator also includes an Informationativeness Token Selection (ITS). Specifically, on the one hand, inspired by the sparsity of the attention mechanism in Vision Transformer (ViT) (i.e., only a few tokens provide effective semantic information, and background tokens may even affect performance), this application embodiment designs an importance evaluation mechanism based on attention weights to construct the information selector; on the other hand, in order to filter out as many key image block-level feature information as possible from a large amount of image block-level feature information that is more relevant to the video-level feature information of the media content, this application embodiment also supports the introduction of video-level tagging information (which corresponds to video-level feature information representing the entire media video) to achieve feature filtering on a large amount of image block-level feature information, aiming to filter out key image block-level feature information that is more relevant to the overall visual semantics of the media video from a large amount of image block-level feature information.
[0143] For example, the pseudo-query generator includes an information selector and a query generator, and the process of generating pseudo-text information through the information selector and query generator can be found in [reference needed]. Figure 7 .like Figure 7 As shown, the information selector performs feature filtering on a large amount of image patch-level feature information to obtain key image patch-level feature information. After concatenating the video-level feature information, the frame-level feature information of each video frame, and the key image patch-level feature information, the concatenated visual feature information is fed into the query generator through linear projection. This allows the query generator to perform pseudo-text prediction processing on the media content based on the visual feature information according to the causal attention mechanism, generating pseudo-text information, which is then represented as:
[0144] t p =φ g ([x v ,x f ,x ip ])
[0145] x ip These are the key image block-level feature information after filtering.
[0146] In detail, the information selector uses an importance evaluation mechanism based on attention weights and video-level labeling information to filter the image patch-level feature information corresponding to each image patch in each video frame. The process of obtaining the filtered key image patch-level feature information can be found in [link to relevant documentation]. Figure 8 .like Figure 8 As shown:
[0147] 1) Input the video-level tagging information (specifically, video-level feature information) and the image patch-level feature information of each image block in each video frame (this image patch-level feature information is the image patch-level tagging information, which can be represented as a patch token (or patch level token)) into the information selector; The information selector calculates the attention distribution of the video-level tagging information on all frame-level tagging information to obtain an information matrix for measuring the importance of each image patch-level feature information; Specifically, based on the video-level feature information corresponding to the media video, the attention distribution operation is performed on the image patch-level feature information corresponding to each image block in each video frame to obtain the information matrix. The information matrix includes the matching degree between each image patch-level feature information and the video-level feature information. A high matching degree indicates that the image patch semantics represented by the image patch-level feature information and the video semantics represented by the video-level feature information are more relevant.
[0148] As mentioned above, the number of video-level markers can be multiple (e.g., four). This application embodiment supports selecting at least one video-level marker to calculate the attention distribution over all frame-level markers. Taking the calculation of the attention distribution over all frame-level markers using the first video-level marker from the four video-level markers to obtain the information matrix as an example, this process can be represented as:
[0149]
[0150] in, and Let h be the linear transformation matrix of the h-th attention head in the multi-head attention mechanism of the information selector; let the linear transformation matrix be... x as image block-level feature information p The weights are calculated. As a key parameter in the attention mechanism; linear transformation matrix As video-level feature information x v The weights are calculated. As the query parameter in the attention mechanism. `softmax()` is the activation function, and the calculated S... h ∈[0,1] m×n This represents the informativeness matrix of each image block-level feature information; it is obtained by maximizing S during multi-head attention. h The value is used to obtain the importance score of each image block-level feature information, thereby generating an information matrix in which each score corresponds to the importance of an image block-level feature information.
[0151] 2) The information selector, based on the scores of each image-level feature in the information matrix (higher scores indicate a higher matching degree between image block-level and video-level feature information), filters at least one image block-level feature that meets the matching degree requirement from the information matrix. This filtered key image block-level feature serves as the input to the subsequent query generator, thereby improving the discriminative and representational capabilities of pseudo-text information. The specific content of the matching degree requirement is not limited in this application embodiment. For example, the matching degree requirement may be to filter the top K image block-level feature information with the highest scores in the information matrix, or to filter at least one image block-level feature with a score greater than a score threshold in the information matrix. The filtered key image block-level feature is denoted as... The visual feature information obtained by concatenating video-level feature information, frame-level feature information, and key image block-level feature information is as follows:
[0152]
[0153] (3) The feature interaction module is represented as φf (·); In this embodiment of the application, a single-layer attention mechanism (XPool Attention) without residual connections is supported to construct a feature interaction module. Its main function is to use pseudo-text information as a guide to perform attention pooling on visual feature information (specifically, a series of visual sub-feature information corresponding to each video frame in the media video). It can learn to query and highlight video frames related to pseudo-text information based on given pseudo-text information, and suppress video frames unrelated to pseudo-text information.
[0154] In its specific implementation, the feature interaction module, based on a single-layer attention mechanism, achieves refined feature interaction between pseudo-text information and visual characteristic information. The process includes using pseudo-text information as the query parameter t of the attention mechanism. p The query parameter Q serves as the attention mechanism; and the video-level feature information corresponding to the media video and the frame-level feature information corresponding to each video frame in the media video are concatenated, and the concatenated feature information is then used as the query parameter Q. The key and value parameters (represented as K&V) serve as the attention mechanism. Then, attention operations are performed on the query parameters, key parameters, and value parameters to obtain the visual representation information of the media content. In detail, the feature interaction module takes pseudo-text information and visual feature information as input, and outputs the final visual representation information enhanced by "pseudo-text information-visual feature information interaction modeling" using the following formula:
[0155]
[0156] Where [·,·] represents the concatenation operation; KV:[x v ,x f ] refers to K&V:[x v ,x f ] indicates that the overall representation of the concatenated key and value parameters is [x v ,x f LN stands for Layer Normalization; FC stands for Fully Connected Network.
[0157] In summary, on the one hand, the embodiments of this application support pseudo-text prediction processing of media content based on visual feature information of media content, generating pseudo-text information of the media content. This pseudo-text information represents the textual semantics of the pseudo-query text used when the media content is retrieved, and this textual semantics typically describes a more granular content semantics of the media content. Thus, by performing feature interaction processing on the pseudo-text information and the visual feature information of the media content, visual representation information of the media content is obtained. This visual representation information has a more refined semantic expression capability than the visual feature information; specifically, the represented visual semantics matches better with the textual semantics of the pseudo-query text that the user might input. Therefore, in the process of content retrieval based on query text, high-precision content retrieval can be achieved based on visual representation information that integrates pseudo-text information and visual feature information. On the other hand, the process of generating pseudo-text information and performing feature interaction between pseudo-text information and visual feature information to generate visual representation information can be performed offline, thereby avoiding complex calculations such as real-time pseudo-text information generation and feature interaction during the online content retrieval stage, and thus improving the retrieval efficiency of content retrieval.
[0158] Figure 4 The embodiments primarily describe the offline stage of calling the trained content retrieval model to represent the features of media content in the content database. As can be seen from the foregoing description, the embodiments of this application also involve a model training stage for the content retrieval model and an online content retrieval stage based on the user's actual query text. Based on this, the following will combine... Figure 9 The illustrated embodiments provide a complete description of the media content processing method of this application. Figure 9 The illustration shows a schematic flowchart of a media content processing method provided in an exemplary embodiment of this application; the media content processing method can be executed by a computer device in the aforementioned system; the media content processing method may include, but is not limited to, steps S901-S904:
[0159] S901: Obtain the training dataset and use it to train the initial content retrieval model, thus obtaining the trained content retrieval model.
[0160] The content retrieval model is obtained by training the initial content retrieval model; in other words, the model before training is called the initial content retrieval model, and the model after training is called the content retrieval model. The training dataset is the data source used to train the initial content retrieval model; this training dataset consists of N... v One sample media content and N t The query text consists of N sample queries. v One sample media content and N tEach sample query text is composed of text content pairs, where each text content pair includes one sample media content and at least one sample query text. For example, the training dataset can be represented as D = {v...} i ,t j}, v i N represents v The i-th sample media content in N sample media content, N t This represents the j-th sample query text in at least one sample query text corresponding to the i-th sample media content.
[0161] Specifically, a diagram illustrating how computer equipment trains an initial content retrieval model using a training dataset can be found in [reference needed]. Figure 10 ;like Figure 10 As shown, the initial content retrieval model includes a video encoder, a pseudo-query generator, a feature interaction module, and a text encoder, denoted as φ. t (·). In this embodiment, the text encoder can be the pseudo-standard CLIP text encoder, mainly used to extract text features of the query text. In the online content retrieval stage, given the input query text t, the text encoder maps it to a vector representation in the cross-modal feature space (i.e., text semantic information, or text embedding features), denoted as: t = φ t (t), where, This represents the extracted text semantic information, where t is the actual query text input by the user during the online content retrieval stage, or the sample query text input during the model training stage (used as sample labels for model training).
[0162] The process of training the initial content retrieval model can include: ① using the initial content retrieval model to perform visual feature processing on the sample media content to obtain predicted visual feature information; ② using the initial content retrieval model to perform pseudo-text prediction processing on the sample media content based on the predicted visual feature information to generate predicted pseudo-text information corresponding to the sample media content; ③ using the initial content retrieval model to perform feature interaction processing on the predicted pseudo-text information and the predicted visual feature information to obtain predicted visual representation information corresponding to the sample media content. The specific implementation processes of the visual feature processing, pseudo-text prediction processing, and feature interaction processing shown in steps ①②③ above can be found in the aforementioned... Figure 4The specific implementation processes for visual feature processing, pseudo-text prediction processing, and feature interaction processing in the illustrated embodiments are not detailed here. ④ Based on the sample query text corresponding to the sample media content, predict pseudo-text information and predict visual representation information to optimize the initial content retrieval model, thereby obtaining a trained content retrieval model. Specifically, the sample query text corresponding to the sample media content is input into the text encoder in the initial content retrieval model. The text encoder extracts text features from the sample query text to obtain the predicted text semantics of the sample query text. Then, based on the predicted text semantics, predicted pseudo-text information, and predicted visual representation information of the sample query text, the initial content retrieval model is optimized to obtain a trained content retrieval model.
[0163] See Figure 10 As can be seen from the model training process shown, the optimization process for the initial content retrieval model in this embodiment includes two stages:
[0164] The first optimization stage involves pre-training the pseudo-query generator using only the reconstruction loss function, while keeping the video encoder and feature interaction frozen. Specifically, real sample query text corresponding to the sample media content is used as supervision to constrain the predicted pseudo-text information reconstructed or predicted from the media content, thereby optimizing the pseudo-query generator. The reconstruction loss function can be expressed as:
[0165]
[0166] `simcos()` represents the cosine similarity between two features, specifically the cosine similarity between the text features of the sample query text and the predicted pseudo-text information. In the first optimization stage, the optimization proceeds in the direction of reducing the gap between the sample query text corresponding to the sample media content and the predicted pseudo-text information, i.e., reducing... The initial content retrieval model is optimized in the first stage (i.e., the model parameters of the pseudo-query generator are adjusted) to obtain the intermediate content retrieval model after the first stage optimization.
[0167] The second optimization stage involves fine-tuning the initial content retrieval model by simultaneously using both reconstruction and contrastive loss functions. Specifically, real sample query text corresponding to the sample media content is used as supervision to constrain the predicted visual representation reconstructed or predicted from the media content, thereby optimizing the entire initial content retrieval model. The contrastive loss function can be expressed as:
[0168]
[0169] Where B is the batch size of training data input to the initial content retrieval model during one round of model training, i.e., the number of sample media content - sample query text pairs. s(t,v)=t·v / ‖t‖·‖v‖ represents cosine similarity, used to measure the similarity between sample media content and sample query text; τ is a learnable temperature scaling factor.
[0170] The total loss function used in the second optimization stage is expressed as follows:
[0171]
[0172] α represents the scaling weight of the reconstruction loss function. During the second optimization phase, the optimization proceeds in the direction of reducing the gap between the sample query text corresponding to the sample media content and the predicted visual representation information, i.e., by reducing... In the direction of optimization, the intermediate content retrieval model is optimized in the second stage (i.e., the model parameters of each module in the initial content retrieval model are adjusted) to obtain the trained content retrieval model.
[0173] As described above, the embodiments of this application support a two-stage model optimization during the model training process for the initial content retrieval model. Specifically, the first optimization stage only optimizes the parameters of the pseudo-query generator in the initial content retrieval model to improve the prediction ability of the pseudo-text information of the pseudo-query generator. The second optimization stage optimizes the parameters of all modules in the initial content retrieval model to comprehensively improve the feature representation ability of the initial content retrieval model for media content.
[0174] S902: Obtain the content database and call the content retrieval model to perform feature representation on each media content, obtain the visual representation information of each media content, and cache the visual representation information of each media content.
[0175] After training the content retrieval model based on step S901, this embodiment supports, in the offline stage, invoking the content retrieval model to pre-represent each retrievalable media content in the content database using feature representations, and caching the video representation information of the media content. This allows for direct retrieval of the cached visual representation information of the media content during subsequent online content retrieval, avoiding the complex feature representation process during online content retrieval and improving the retrieval efficiency. The feature representation of each media content in the offline stage includes: visual feature processing, pseudo-text prediction processing, and feature interaction processing. For the specific implementation process of visual feature processing, pseudo-text prediction processing, and feature interaction processing, please refer to the foregoing. Figure 4 The specific implementation process described in steps S401-S403 of the embodiment will not be repeated here.
[0176] S903: Receives a content retrieval request and, in response to the content retrieval request, extracts text features from the query text to obtain the text semantic information of the query text.
[0177] S904: Calculate the similarity between the text semantic information and the visual representation information of each media content, and take at least one media content whose similarity meets the similarity threshold as the retrieved target media content.
[0178] In steps S903-S904, during the online content retrieval phase, users can upload actual query text. This query text describes the media content they want to query (i.e., the content the user wants to search for). The query text elucidates the content information of the media content; for example, if the media content to be queried is a video, then the query text might describe the coherent content of several consecutive video frames in that video. Figure 1 The video shown has a query text describing the coherent content depicted from video frame i to video frame i+3. The terminal generates a content retrieval request based on this query text, which carries the query text. If the terminal does not have a content retrieval model deployed, it sends the request to the server. If the terminal has a content retrieval model deployed, it directly responds to the request and performs a content retrieval. When the server receives the content retrieval request, it invokes the text encoder in the content retrieval model to extract text features from the query text, obtaining the text semantic information. Then, the server calculates the similarity between the text semantic information and the visual representation information of each media content, obtaining the similarity result. For example, the text semantic information and visual representation information are represented as embedding vectors. The similarity result is calculated by calculating the vector distance between the embedding vectors; the closer the vector distance, the higher the similarity between the text semantic information and the visual representation information of the media content. Finally, at least one media content whose similarity meets the similarity threshold is used as the retrieved target media content, which means at least one media content whose similarity result is greater than the similarity threshold is used as the target media content.
[0179] This application's embodiments compare feature-based relearning methods and CLIP-based end-to-end methods. Experimental data on content retrieval across different databases can be found in [link to relevant documentation]. Figure 11 .like Figure 11As shown: ① The feature relearning methods compared in this application's embodiments include: SEA, W2VV++, MMT, and LAFF; the single-tower CLIP end-to-end methods compared in this application's embodiments include: TS2-Net, X-CLIP, XPool, DRL, ProST, and UCoFiA; the dual-tower CLIP end-to-end methods compared in this application's embodiments include: CLIP4clip, Center CLIP, Cap4Video, PromptSwitch, STAN, TeachCLIP, and CLIP-ViP. ② The datasets used for the comparative experiments in this application's embodiments include: MSRVTT-1k, MSRVTT-3k, MSVD, and VATEX. ③ The metrics used for the comparative experiments in this application's embodiments include: FLOPs (Floating Point Operations Per Second), R@1, R@5, and SumR; among them, FLOPs is an important metric for measuring the number of floating point operations performed per second when a computer model calls a content retrieval model to perform a task. R@1 and R@5 are important metrics for measuring model performance. They represent the proportion of relevant items in the first and first five results of the returned results (i.e., the retrieved media content), respectively. R@1 focuses more on evaluating the accuracy of the media content recommended first, while R@5 evaluates the overall accuracy of the media content recommended in the top few results. The SumR metric represents the proportion of relevant items in all returned results. ④ Figure 11 The mean in the last column refers to the R@1 mean across all datasets. ⑤ In end-to-end methods (such as single-tower CLIP end-to-end methods and dual-tower CLIP end-to-end methods), only some end-to-end methods have results on multiple datasets. For example, only TS2-Net and X-CLIP in the single-tower CLIP end-to-end method have results on the dataset, and only CLIP4clip, TeachCLIP, and CLIP-ViP in the dual-tower CLIP end-to-end method have results on the dataset.
[0180] based on Figure 11The experimental results show that, in terms of content retrieval efficiency, the video representation information of media videos in the content database of this application embodiment is extracted offline and pre-stored, and can be directly retrieved and used during inference. This makes the FLOPs (i.e., the number of operations) required to calculate each media content-query text match during the inference stage significantly less than the FLOPs of existing content retrieval schemes, achieving an optimal efficiency of 0.5K FLOPs per match. In terms of content retrieval accuracy, compared with existing end-to-end methods, this application embodiment stands out with an average R@1 of 48.0, significantly outperforming its dual-tower CLIP end-to-end method. Specifically, the performance of the backbone model CLIP-ViP in the hybrid tower structure content retrieval model is improved, with an R@1 improvement of 2.6 on MSRV TT-1k, 1.6 on MSRV TT-3k, 2.8 on MSVD, and 1.9 on VATEX, thus surpassing the existing content retrieval method TeachCLIP. Furthermore, the hybrid tower structure content retrieval model provided in this application has extremely low computational cost (0.5K FLO Ps), further highlighting its efficiency and making it very suitable for real-time retrieval applications.
[0181] In summary, this application presents a hybrid tower content retrieval model based on pseudo-query generation and feature interaction for text-to-media content retrieval, and a media content processing method based on this hybrid tower content retrieval model. The hybrid tower content retrieval model and media content processing method provided in this application significantly improve both content retrieval accuracy and efficiency compared to existing content retrieval schemes. It can generate possible pseudo-text information based on media content, thereby performing fine-grained feature interaction between visual feature information and pseudo-text information of media content in a single-tower manner before the user's actual query text arrives, obtaining and storing more expressive visual representation information of the media content offline. After completing the pseudo-interaction, i.e., during online deployment, it can efficiently recall relevant target media content based on the user's actual query text and the dot product operation in the dual-tower structure, thus achieving comprehensive optimization in content retrieval effect and efficiency.
[0182] The methods of the embodiments of this application have been described in detail above. To facilitate better implementation of the above-described solutions of the embodiments of this application, the apparatus of the embodiments of this application is provided below. In the embodiments of this application, the terms "module" or "unit" refer to a computer program or part of a computer program with a predetermined function, which works together with other related parts to achieve a predetermined goal, and can be implemented wholly or partially using software, hardware (such as processing circuitry or memory), or a combination thereof. Similarly, a processor (or multiple processors or memory) can be used to implement one or more modules or units. Furthermore, each module or unit can be part of an overall module or unit that includes the functionality of that module or unit.
[0183] Figure 12 This application shows a schematic diagram of the structure of a media content processing apparatus according to an exemplary embodiment; the media content processing apparatus can be used to perform... Figure 4 and Figure 9 Some or all of the steps in the method embodiments shown. Please refer to [link / reference]. Figure 12 The device includes the following units:
[0184] The acquisition unit 1201 is used to acquire the media content to be processed and to perform visual feature processing on the media content to obtain visual feature information, which is used to characterize the visual semantics of the media content.
[0185] The processing unit 1202 is used to perform pseudo-text prediction processing on media content based on visual feature information to generate pseudo-text information; the pseudo-text information is used to characterize the text semantics of the pseudo-query text used for prediction when the media content is retrieved.
[0186] The processing unit 1202 is also used to perform feature interaction processing on pseudo-text information and visual feature information to obtain visual representation information of media content; the feature interaction processing is used to: enhance the text semantics of the pseudo-query text representation in the visual semantics represented by the visual feature information; the visual representation information is used for content retrieval of media content.
[0187] In one implementation, the processing unit 1202 is used to perform visual feature processing on the media content, and when obtaining visual feature information, it is specifically used for:
[0188] Feature extraction is performed on media content from multiple content dimensions to obtain initial feature information for each content dimension; the initial feature information represents the visual semantics of the media content from the corresponding content dimension.
[0189] The initial feature information under each content dimension is linearly projected into the cross-modal feature space to obtain the visual sub-feature information of each initial feature information in the cross-modal feature space; the cross-modal feature space includes feature information of at least two modalities, and one initial feature information corresponds to one visual feature information;
[0190] Visual feature information is obtained by concatenating the visual sub-features of each initial feature information in the cross-modal feature space.
[0191] In one implementation, the media content is video; the processing unit 1202, used to extract features from the media content from multiple content dimensions, specifically performs the following when obtaining initial feature information for each content dimension:
[0192] The media video is converted from a high-dimensional to a low-dimensional initial feature vector; the initial feature vector includes the initial sub-feature vector corresponding to each video frame in the media video;
[0193] Frame-level labeling information is added to each initial sub-feature vector, and the initial sub-feature vectors after adding frame-level labeling information are concatenated to obtain the initial concatenated vector;
[0194] Add video-level labeling information to the initial concatenated vector to obtain the initial labeled feature vector;
[0195] Feature extraction is performed on the initial labeled feature vector from multiple content dimensions to obtain the initial feature information for each content dimension.
[0196] In one implementation, the media content is a media video, and the visual sub-feature information includes: video-level feature information corresponding to the media video, frame-level feature information corresponding to each video frame in the media video, and image block-level feature information corresponding to each image block in each video frame.
[0197] Processing unit 1202 is used to perform feature concatenation on the visual sub-features of each initial feature information in the cross-modal feature space to obtain visual feature information. Specifically, it is used for:
[0198] Feature filtering is performed on the image block-level feature information corresponding to each image block in each video frame to obtain the filtered key image block-level feature information;
[0199] Visual feature information is obtained by concatenating the video-level feature information, the frame-level feature information corresponding to each video frame, and the key image block-level feature information.
[0200] In one implementation, the processing unit 1202 is used to perform feature filtering on the image block-level feature information corresponding to each image block in each video frame, and to obtain the filtered key image block-level feature information, specifically for:
[0201] Based on the video-level feature information corresponding to the media video, attention distribution operation is performed on the image block-level feature information corresponding to each image block in each video frame to obtain the information matrix; the information matrix includes the matching degree between each image block-level feature information and the video-level feature information;
[0202] At least one image block-level feature information that meets the matching degree requirement is selected from the information matrix and used as the key image block-level feature information after selection.
[0203] In one implementation, the processing unit 1202 is used to perform pseudo-text prediction processing on the media content based on visual feature information. Specifically, when generating pseudo-text information, it is used for:
[0204] Based on causal constraints, attention operations are performed on visual feature information to obtain key visual features corresponding to the visual feature information; causal constraints include generating pseudo-text information describing media content according to content coherence.
[0205] By mapping key visual features to the text query space, pseudo-text information corresponding to media content can be obtained.
[0206] In one implementation, the processing unit 1202, when performing feature interaction processing on the pseudo-text information and visual feature information to obtain the visual representation information of the media content, specifically performs the following:
[0207] Guided by pseudo-text information, feature weights are assigned to visual feature information to obtain visual representation information of media content; the visual semantics represented by the visual representation information match the text semantics described by the pseudo-text information.
[0208] In one implementation, the visual feature information is obtained by feature stitching together multiple visual sub-feature information, including: video-level feature information corresponding to the media video, and frame-level feature information corresponding to each video frame in the media video.
[0209] Processing unit 1202, used for guiding visual feature information based on pseudo-text information, performs feature interaction processing on visual feature information under multiple content dimensions to obtain visual representation information of media content, specifically for:
[0210] Using pseudo-text information as query parameters for the attention mechanism; and,
[0211] The video-level feature information corresponding to the media video and the frame-level feature information corresponding to each video frame in the media video are concatenated, and the concatenated feature information is used as the key and value parameters of the attention mechanism.
[0212] Attention operations are performed on query parameters, key parameters, and value parameters to obtain visual representation information of media content.
[0213] In one implementation, the method is executed by invoking a trained content retrieval model; the processing unit 1202 is also used for:
[0214] Retrieve the content database, which includes multiple media content items that can be retrieved;
[0215] The content retrieval model is invoked to perform feature representation on each media content, thereby obtaining the visual representation information of each media content; wherein, feature representation includes visual feature processing, pseudo-text prediction processing, and feature interaction processing;
[0216] The visual representation information of each media content is cached.
[0217] In one implementation, the processing unit 1202 is further configured to:
[0218] Receive content retrieval requests, which carry query text, which is text describing the media content to be queried;
[0219] In response to a content retrieval request, text features are extracted from the query text to obtain the textual semantic information of the query text;
[0220] Calculate the similarity between the semantic information of the text and the visual representation information of each media content;
[0221] Media content that meets at least one similarity threshold will be used as the retrieved target media content.
[0222] In one implementation, the content retrieval model is obtained by training an initial content retrieval model; the model training process includes:
[0223] Obtain the training dataset, which includes text content pairs. Each text content pair includes one sample media content and at least one sample query text.
[0224] An initial content retrieval model is used to perform visual feature processing on the sample media content to obtain predicted visual feature information;
[0225] An initial content retrieval model is used to perform pseudo-text prediction processing on the sample media content based on the predicted visual feature information, thereby generating the predicted pseudo-text information corresponding to the sample media content.
[0226] An initial content retrieval model is used to perform feature interaction processing on predicted pseudo-text information and predicted visual feature information to obtain the predicted visual representation information corresponding to the sample media content.
[0227] Based on the sample query text corresponding to the sample media content, predict pseudo-text information and predict visual representation information to optimize the initial content retrieval model and obtain a trained content retrieval model.
[0228] In one implementation, the processing unit 1202, when optimizing the initial content retrieval model based on the sample query text corresponding to the sample media content, predicts pseudo-text information and the predicted visual representation information to obtain a trained content retrieval model, specifically performs the following:
[0229] Following the direction of reducing the gap between the sample query text corresponding to the sample media content and the predicted pseudo-text information, the initial content retrieval model is optimized in the first stage to obtain the intermediate content retrieval model.
[0230] Following the direction of reducing the gap between the sample query text corresponding to the sample media content and the predicted visual representation information, the intermediate content retrieval model is optimized in the second stage to obtain the trained content retrieval model.
[0231] According to one embodiment of this application, Figure 12 The various units in the media content processing apparatus shown can be individually or entirely merged into one or more other units, or some of the units can be further divided into multiple functionally smaller units. This achieves the same operation without affecting the technical effects of the embodiments of this application. The above units are based on logical function division. In practical applications, the function of one unit can also be implemented by multiple units, or the function of multiple units can be implemented by one unit. In other embodiments of this application, the media content processing apparatus may also include other units. In practical applications, these functions can also be implemented with the assistance of other units, and can be implemented collaboratively by multiple units. According to another embodiment of this application, the media content processing apparatus can be executed by running on a general-purpose computing device, such as a computer, which includes processing elements and storage elements such as a central processing unit (CPU), random access storage medium (RAM), and read-only storage medium (ROM). Figure 4 and Figure 9 The computer program (including program code) for each step involved in the corresponding method shown, to construct such... Figure 12 The media content processing apparatus shown herein, and the media content processing method for implementing the embodiments of this application, are described. A computer program may be recorded on, for example, a computer-readable recording medium, loaded onto the aforementioned computing device via the computer-readable recording medium, and executed therein.
[0232] In the process of generating visual representation information for media content, this application embodiment can integrate pseudo-text information from pseudo-query text that may be used during media content retrieval. This enhances (or highlights) the textual semantics represented by the pseudo-text information within the visual semantics represented by the visual features of the media content itself, thereby generating visual representation information that better matches the pseudo-query text. Consequently, in the actual content retrieval process, media content that is more relevant to the query text can be retrieved based on the visual representation information, improving the accuracy and precision of content retrieval.
[0233] Figure 13 A schematic diagram of the structure of a computer device provided in an exemplary embodiment of this application is shown. Please refer to... Figure 13 The computer device includes a processor 1301, a communication interface 1302, and a computer-readable storage medium 1303. The processor 1301, communication interface 1302, and computer-readable storage medium 1303 can be connected via a bus or other means. The communication interface 1302 is used to receive and send data. The computer-readable storage medium 1303 can be stored in the computer device's memory and is used to store computer programs, including program instructions. The processor 1301 is used to execute the program instructions stored in the computer-readable storage medium 1303. The processor 1301 (or CPU (Central Processing Unit)) is the computing and control core of the computer device, suitable for implementing one or more instructions, specifically suitable for loading and executing one or more instructions to achieve corresponding method flows or corresponding functions.
[0234] This application embodiment also provides a computer-readable storage medium (Memory), which is a memory device in a computer device used to store programs and data. It is understood that the computer-readable storage medium here can include both the built-in storage medium in the computer device and extended storage media supported by the computer device. The computer-readable storage medium provides storage space that stores the processing system of the computer device. Furthermore, the storage space also stores one or more instructions suitable for loading and execution by the processor 1301, which may be one or more computer programs (including program code). It should be noted that the computer-readable storage medium here can be high-speed RAM memory or non-volatile memory, such as at least one disk storage device; optionally, it can also be at least one computer-readable storage medium located remotely from the aforementioned processor.
[0235] In one embodiment, the computer-readable storage medium stores one or more instructions; the processor 1301 loads and executes one or more instructions stored in the computer-readable storage medium to implement the corresponding steps in the above-described media content processing method embodiment; the one or more instructions in the computer-readable storage medium are loaded by the processor 1301 and executed as follows:
[0236] The process involves acquiring media content to be processed and performing visual feature processing on the media content to obtain visual feature information, which is used to characterize the visual semantics of the media content.
[0237] Based on visual feature information, pseudo-text prediction processing is performed on media content to generate pseudo-text information; the pseudo-text information is used to characterize the textual semantics of the pseudo-query text used when the media content is retrieved.
[0238] Feature interaction processing is performed on pseudo-text information and visual feature information to obtain visual representation information of media content; feature interaction processing is used to: enhance the text semantics of pseudo-query text representation in the visual semantics represented by visual feature information; visual representation information is used for content retrieval of media content.
[0239] In one implementation, when one or more instructions in the computer-readable storage medium are loaded by the processor 1301 and executed to perform visual feature processing on the media content to obtain visual feature information, the following steps are specifically performed:
[0240] Feature extraction is performed on media content from multiple content dimensions to obtain initial feature information for each content dimension; the initial feature information represents the visual semantics of the media content from the corresponding content dimension.
[0241] The initial feature information under each content dimension is linearly projected into the cross-modal feature space to obtain the visual sub-feature information of each initial feature information in the cross-modal feature space; the cross-modal feature space includes feature information of at least two modalities, and one initial feature information corresponds to one visual feature information;
[0242] Visual feature information is obtained by concatenating the visual sub-features of each initial feature information in the cross-modal feature space.
[0243] In one implementation, the media content is video; when one or more instructions in the computer-readable storage medium are loaded by the processor 1301 and executed to extract features from the media content from multiple content dimensions to obtain initial feature information for each content dimension, the following steps are specifically performed:
[0244] The media video is converted from a high-dimensional to a low-dimensional initial feature vector; the initial feature vector includes the initial sub-feature vector corresponding to each video frame in the media video;
[0245] Frame-level labeling information is added to each initial sub-feature vector, and the initial sub-feature vectors after adding frame-level labeling information are concatenated to obtain the initial concatenated vector;
[0246] Add video-level labeling information to the initial concatenated vector to obtain the initial labeled feature vector;
[0247] Feature extraction is performed on the initial labeled feature vector from multiple content dimensions to obtain the initial feature information for each content dimension.
[0248] In one implementation, the media content is a media video, and the visual sub-feature information includes: video-level feature information corresponding to the media video, frame-level feature information corresponding to each video frame in the media video, and image block-level feature information corresponding to each image block in each video frame.
[0249] When one or more instructions in a computer-readable storage medium are loaded by processor 1301 and executed to perform feature concatenation of visual sub-features in the cross-modal feature space for each initial feature information to obtain visual feature information, the following steps are specifically performed:
[0250] Feature filtering is performed on the image block-level feature information corresponding to each image block in each video frame to obtain the filtered key image block-level feature information;
[0251] Visual feature information is obtained by concatenating the video-level feature information, the frame-level feature information corresponding to each video frame, and the key image block-level feature information.
[0252] In one implementation, when one or more instructions in the computer-readable storage medium are loaded by the processor 1301 and executed to perform feature filtering on the image block-level feature information corresponding to each image block in each video frame to obtain the filtered key image block-level feature information, the following steps are specifically performed:
[0253] Based on the video-level feature information corresponding to the media video, attention distribution operation is performed on the image block-level feature information corresponding to each image block in each video frame to obtain the information matrix; the information matrix includes the matching degree between each image block-level feature information and the video-level feature information;
[0254] At least one image block-level feature information that meets the matching degree requirement is selected from the information matrix and used as the key image block-level feature information after selection.
[0255] In one implementation, when one or more instructions in the computer-readable storage medium are loaded by the processor 1301 and executed to perform pseudo-text prediction processing on the media content based on visual feature information to generate pseudo-text information, the following steps are specifically performed:
[0256] Based on causal constraints, attention operations are performed on visual feature information to obtain key visual features corresponding to the visual feature information; causal constraints include generating pseudo-text information describing media content according to content coherence.
[0257] By mapping key visual features to the text query space, pseudo-text information corresponding to media content can be obtained.
[0258] In one implementation, when one or more instructions in the computer-readable storage medium are loaded by the processor 1301 and executed to perform feature interaction processing on pseudo-text information and visual feature information to obtain visual representation information of the media content, the following steps are specifically performed:
[0259] Guided by pseudo-text information, feature weights are assigned to visual feature information to obtain visual representation information of media content; the visual semantics represented by the visual representation information match the text semantics described by the pseudo-text information.
[0260] In one implementation, the visual feature information is obtained by feature stitching together multiple visual sub-feature information, including: video-level feature information corresponding to the media video, and frame-level feature information corresponding to each video frame in the media video.
[0261] When one or more instructions in a computer-readable storage medium are loaded by processor 1301 and executed based on pseudo-text information, and feature interaction processing is performed on visual feature information under multiple content dimensions to obtain visual representation information of media content, the following steps are specifically executed:
[0262] Using pseudo-text information as query parameters for the attention mechanism; and,
[0263] The video-level feature information corresponding to the media video and the frame-level feature information corresponding to each video frame in the media video are concatenated, and the concatenated feature information is used as the key and value parameters of the attention mechanism.
[0264] Attention operations are performed on query parameters, key parameters, and value parameters to obtain visual representation information of media content.
[0265] In one implementation, the method is executed by invoking a trained content retrieval model; one or more instructions in a computer-readable storage medium are loaded by processor 1301 and the following steps are also executed:
[0266] Retrieve the content database, which includes multiple media content items that can be retrieved;
[0267] The content retrieval model is invoked to perform feature representation on each media content, thereby obtaining the visual representation information of each media content; wherein, feature representation includes visual feature processing, pseudo-text prediction processing, and feature interaction processing;
[0268] The visual representation information of each media content is cached.
[0269] In one implementation, one or more instructions in a computer-readable storage medium are loaded by processor 1301 and the following steps are also performed:
[0270] Receive content retrieval requests, which carry query text, which is text describing the media content to be queried;
[0271] In response to a content retrieval request, text features are extracted from the query text to obtain the textual semantic information of the query text;
[0272] Calculate the similarity between the semantic information of the text and the visual representation information of each media content;
[0273] Media content that meets at least one similarity threshold will be used as the retrieved target media content.
[0274] In one implementation, the content retrieval model is obtained by training an initial content retrieval model; the model training process includes:
[0275] Obtain the training dataset, which includes text content pairs. Each text content pair includes one sample media content and at least one sample query text.
[0276] An initial content retrieval model is used to perform visual feature processing on the sample media content to obtain predicted visual feature information;
[0277] An initial content retrieval model is used to perform pseudo-text prediction processing on the sample media content based on the predicted visual feature information, thereby generating the predicted pseudo-text information corresponding to the sample media content.
[0278] An initial content retrieval model is used to perform feature interaction processing on predicted pseudo-text information and predicted visual feature information to obtain the predicted visual representation information corresponding to the sample media content.
[0279] Based on the sample query text corresponding to the sample media content, predict pseudo-text information and predict visual representation information to optimize the initial content retrieval model and obtain a trained content retrieval model.
[0280] In one implementation, when one or more instructions in a computer-readable storage medium are loaded by processor 1301 and executed to predict pseudo-text information and the predicted visual representation information based on sample query text corresponding to sample media content, and to optimize the initial content retrieval model to obtain a trained content retrieval model, the following steps are specifically performed:
[0281] Following the direction of reducing the gap between the sample query text corresponding to the sample media content and the predicted pseudo-text information, the initial content retrieval model is optimized in the first stage to obtain the intermediate content retrieval model.
[0282] Following the direction of reducing the gap between the sample query text corresponding to the sample media content and the predicted visual representation information, the intermediate content retrieval model is optimized in the second stage to obtain the trained content retrieval model.
[0283] Based on the same inventive concept, the principle and beneficial effects of the computer device provided in the embodiments of this application in solving the problem are similar to the principle and beneficial effects of the media content processing method in the embodiments of this application in solving the problem. Please refer to the principle and beneficial effects of the implementation of the method. For the sake of brevity, they will not be repeated here.
[0284] This application also provides a computer program product, which includes a computer program that, when executed by a processor, implements the above-described media content processing method.
[0285] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed in this application can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.
[0286] In the above embodiments, implementation can be achieved entirely or partially through software, hardware, firmware, or any combination thereof. When implemented using software, it can be implemented entirely or partially as a computer program product. A computer program product includes one or more computer programs. When a computer program is loaded and executed on a computer device, the computer program performs the processes or functions described above in the embodiments of this application. The computer device can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer program can be stored in or transmitted through a computer-readable storage medium. The computer program can be transmitted from one website, computer device, server, or data center to another website, computer device, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.). The computer-readable storage medium can be any available medium accessible to the computer device or a data storage device such as a server or data center that integrates one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid-state disk (SSD)).
[0287] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.
Claims
1. A media content processing method, characterized in that, include: The media content to be processed is acquired, and the media content is subjected to visual feature processing to obtain visual feature information, which is used to characterize the visual semantics of the media content. Based on the visual feature information, pseudo-text prediction processing is performed on the media content to generate pseudo-text information; the pseudo-text information is used to characterize the textual semantics of the pseudo-query text used for prediction when the media content is retrieved. The pseudo-text information and the visual feature information are subjected to feature interaction processing to obtain the visual representation information of the media content; The feature interaction processing is used to: enhance the text semantics represented by the pseudo-text information in the visual semantics represented by the visual feature information; the visual representation information is used to perform content retrieval of the media content.
2. The method as described in claim 1, characterized in that, The step of performing visual feature processing on the media content to obtain visual feature information includes: Feature extraction is performed on the media content from multiple content dimensions to obtain initial feature information for each content dimension; the initial feature information represents the visual semantics of the media content from the corresponding content dimension. The initial feature information under each of the content dimensions is linearly projected into the cross-modal feature space to obtain the visual sub-feature information of each initial feature information in the cross-modal feature space; the cross-modal feature space includes feature information of at least two modalities, and one initial feature information corresponds to one visual feature information; Visual feature information is obtained by concatenating the visual sub-feature information of each initial feature information in the cross-modal feature space.
3. The method as described in claim 2, characterized in that, The media content is media video; the feature extraction of the media content from multiple content dimensions to obtain initial feature information for each content dimension includes: The media video is converted from a high-dimensional to a low-dimensional initial feature vector; the initial feature vector includes the initial sub-feature vector corresponding to each video frame in the media video; Frame-level labeling information is added to each of the initial sub-feature vectors, and the initial sub-feature vectors after adding the frame-level labeling information are concatenated to obtain an initial concatenated vector; Add video-level labeling information to the initial spliced vector to obtain the initial labeling feature vector; Feature extraction is performed on the initial label feature vector from multiple content dimensions to obtain initial feature information for each content dimension.
4. The method as described in claim 2, characterized in that, The media content is a media video, and the visual sub-feature information includes: video-level feature information corresponding to the media video, frame-level feature information corresponding to each video frame in the media video, and image block-level feature information corresponding to each image block in each video frame. The step of concatenating the visual sub-features of each of the initial feature information in the cross-modal feature space to obtain visual feature information includes: Feature filtering is performed on the image block-level feature information corresponding to each image block in each video frame to obtain the filtered key image block-level feature information; The video-level feature information, the frame-level feature information corresponding to each video frame, and the key image block-level feature information are concatenated to obtain visual feature information.
5. The method as described in claim 4, characterized in that, The step of performing feature filtering on the image block-level feature information corresponding to each image block in each video frame to obtain the filtered key image block-level feature information includes: Based on the video-level feature information corresponding to the media video, attention distribution operation is performed on the image block-level feature information corresponding to each image block in each video frame to obtain an information matrix; the information matrix includes the matching degree between each image block-level feature information and the video-level feature information; At least one image block-level feature information that meets the matching degree requirement is selected from the information matrix and used as the selected key image block-level feature information.
6. The method as described in claim 1, characterized in that, The step of performing pseudo-text prediction processing on the media content based on the visual feature information to generate pseudo-text information includes: Based on causal constraints, attention operations are performed on the visual feature information to obtain the key visual features corresponding to the visual feature information; the causal constraints include generating pseudo-text information describing the media content according to content coherence. The key visual features are mapped to the text query space to obtain the pseudo-text information corresponding to the media content.
7. The method as described in claim 1, characterized in that, The step of performing feature interaction processing on the pseudo-text information and the visual feature information to obtain the visual representation information of the media content includes: Guided by the pseudo-text information, feature weights are assigned to the visual feature information to obtain the visual representation information of the media content; the visual semantics represented by the visual representation information and the text semantics described by the pseudo-text information are matched.
8. The method as described in claim 7, characterized in that, The visual feature information is obtained by splicing together multiple visual sub-feature information. The visual sub-feature information includes: video-level feature information corresponding to the media video, and frame-level feature information corresponding to each video frame in the media video. The guidance based on the pseudo-text information involves performing feature interaction processing on the visual feature information under multiple content dimensions to obtain the visual representation information of the media content, including: The pseudo-text information is used as a query parameter for the attention mechanism; and, The video-level feature information corresponding to the media video and the frame-level feature information corresponding to each video frame in the media video are concatenated, and the concatenated feature information is used as the key parameter and value parameter of the attention mechanism. Attention operations are performed on the query parameters, key parameters, and value parameters to obtain the visual representation information of the media content.
9. The method according to any one of claims 1-8, characterized in that, The method is executed by calling a pre-trained content retrieval model; the method also includes: Obtain a content database, which includes multiple media contents that can be retrieved; The content retrieval model is invoked to perform feature representation on each media content to obtain visual representation information for each media content; wherein, the feature representation includes visual feature processing, pseudo-text prediction processing, and feature interaction processing; The visual representation information of each of the media contents is cached.
10. The method as described in claim 9, characterized in that, The method further includes: Receive a content retrieval request, wherein the content retrieval request carries query text, and the query text is text describing the media content to be queried; In response to the content retrieval request, text features are extracted from the query text to obtain the text semantic information of the query text; Calculate the similarity between the text semantic information and the visual representation information of each media content; Media content whose similarity meets at least one of the similarity thresholds is used as the retrieved target media content.
11. The method as described in claim 9, characterized in that, The content retrieval model is obtained by training an initial content retrieval model; the model training process includes: Obtain a training dataset, which includes text content pairs, each of which includes a sample media content and at least one sample query text; Using the initial content retrieval model, visual feature processing is performed on the sample media content to obtain predicted visual feature information; Using the initial content retrieval model, pseudo-text prediction processing is performed on the sample media content based on the predicted visual feature information to generate predicted pseudo-text information corresponding to the sample media content; Using the initial content retrieval model, feature interaction processing is performed on the predicted pseudo-text information and the predicted visual feature information to obtain the predicted visual representation information corresponding to the sample media content; Based on the sample query text corresponding to the sample media content, the predicted pseudo-text information, and the predicted visual representation information, the initial content retrieval model is optimized to obtain a trained content retrieval model.
12. The method as described in claim 11, characterized in that, The step of optimizing the initial content retrieval model based on the sample query text corresponding to the sample media content, the predicted pseudo-text information, and the predicted visual representation information to obtain a trained content retrieval model includes: The initial content retrieval model is optimized in the first stage according to the direction of reducing the gap between the sample query text corresponding to the sample media content and the predicted pseudo-text information, so as to obtain the intermediate content retrieval model. The intermediate content retrieval model is optimized in the second stage according to the direction of reducing the gap between the sample query text corresponding to the sample media content and the predicted visual representation information, so as to obtain a trained content retrieval model.
13. A media content processing device, characterized in that, include: An acquisition unit is used to acquire media content to be processed and to perform visual feature processing on the media content to obtain visual feature information, which is used to characterize the visual semantics of the media content. The processing unit is configured to perform pseudo-text prediction processing on the media content based on the visual feature information to generate pseudo-text information; the pseudo-text information is used to characterize the text semantics of the pseudo-query text used for prediction when the media content is retrieved. The processing unit is further configured to perform feature interaction processing on the pseudo-text information and the visual feature information to obtain visual representation information of the media content; the feature interaction processing is used to: enhance the text semantics representing the pseudo-query text in the visual semantics represented by the visual feature information; the visual representation information is used to perform content retrieval of the media content.
14. A computer device, characterized in that, A processor, adapted to execute computer programs; A computer-readable storage medium storing a computer program, which, when executed by the processor, implements the media content processing method as described in any one of claims 1-12.
15. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program adapted to be loaded by a processor and executed as described in any one of claims 1-12.
16. A computer program product, characterized in that, The computer program product includes a computer program that, when executed by a processor, implements the media content processing method as described in any one of claims 1-12.
