Method for processing media resource and related device

Abstract

The embodiment of the application provides a kind of media resource processing method and related equipment, wherein the method comprises: in response to the data analysis operation of first media resource, display data analysis interface;In data analysis interface, the data analysis report of first media resource is output, and the recommended prompt information is contained in data analysis report, and the recommended prompt information is used to prompt the media resource set matched with the content theme of first media resource;In response to the trigger operation of recommended prompt information, display the media resource in media resource set.The embodiment of the application can realize the media resource recommendation of content theme granularity, and improve the recommendation accuracy.

Description

Technical Field

[0001] This application relates to the field of Internet technology, specifically to a method for processing media resources, a device for processing media resources, a computer device, a computer-readable storage medium, and a computer program product. Background Technology

[0002] With the development of internet technology, to enable users to quickly browse media resources (such as videos) that they are interested in or follow, media resource recommendations are often provided. For example, if a user browses a certain media resource, other media resources in that category can be recommended; or, if a user browses a media resource published by a certain author, other media resources published by that author can be recommended; or, media resources with high popularity (such as high views or high attention) can be recommended. As mentioned above, existing media resource recommendations are mainly based on granularity such as "work category," "author category," and "viewer preferences." This recommendation method, when applied to certain business recommendation scenarios (such as those with high requirements for content theme matching), may suffer from inaccurate recommendations. Summary of the Invention

[0003] This application provides a method and related equipment for processing media resources, which can realize media resource recommendation at the content theme level and improve recommendation accuracy.

[0004] On one hand, embodiments of this application provide a method for processing media resources, the method comprising:

[0005] In response to data analysis operations on the primary media resource, the data analysis interface is displayed;

[0006] The data analysis interface outputs a data analysis report for the first media resource. The data analysis report includes recommendation prompts, which are used to suggest a set of media resources that match the content theme of the first media resource.

[0007] In response to a trigger action on a recommendation message, display media resources from the media resource collection.

[0008] In response to a data analysis operation on a first media resource, this application displays a data analysis interface and outputs a data analysis report for the first media resource within the interface. Therefore, this application can perform data analysis on media resources and present the analysis results visually and intuitively through the data analysis interface. This allows the recipients of the media resources to clearly understand the specific data details of the media resources. Furthermore, the data analysis report includes recommendation prompts to suggest a set of media resources that match the content theme of the first media resource. In response to triggering the recommendation prompts, the media resources in the set are displayed. In other words, during the data analysis of a media resource, this application can also recommend media resources that match its content theme. This achieves a media resource recommendation service at the content theme level, improving the accuracy of media resource recommendations. It also effectively guides recipients of the media resources to compare and optimize them in a timely manner, thereby improving the quality of the media resources.

[0009] On the one hand, embodiments of this application provide another method for processing media resources, the method comprising:

[0010] The first feature of the first sample media resource is obtained by identifying and processing the content theme of the first sample media resource as a multimodal feature.

[0011] Construct positive and negative samples for the first feature. Positive samples contain the second feature, which is obtained by data augmentation of the first feature. Negative samples contain the third feature, which is a multimodal feature obtained by identifying the content theme of the media resource of the second sample. The media resource of the second sample is any media resource that is different from the media resource of the first sample.

[0012] The vector representation model is called to obtain the first representation vector corresponding to the first feature, the second representation vector corresponding to the positive sample, and the third representation vector corresponding to the negative sample.

[0013] Multimodal contrastive learning is performed on the first representation vector and the second representation vector to obtain a first similarity between the first representation vector and the second representation vector; and multimodal contrastive learning is performed on the first representation vector and the third representation vector to obtain a second similarity between the first representation vector and the second representation vector.

[0014] The vector representation model is trained according to the rule of increasing the first similarity and decreasing the second similarity;

[0015] In the scenario of data analysis of the first media resource, the trained vector representation model is used to generate representation vectors that characterize the content theme of the first media resource, so as to recommend a set of media resources similar to the content theme of the first media resource based on the representation vector of the first media resource.

[0016] In this embodiment, a first feature of the first sample media resource can be obtained. This first feature is a multimodal feature obtained by identifying the content theme of the first sample media resource. Therefore, this embodiment utilizes multimodal features as the data foundation for model training. Since multimodal features cover multiple modal types of data (such as text, images, and audio), this provides more comprehensive and accurate information for the model training process. Furthermore, multimodal features can reflect the content theme of the media resource, enabling the trained model to provide recommendation services for media resources at the content theme granularity. Positive and negative samples of the first feature are constructed. Positive samples include a second feature, which is obtained by data augmentation processing of the first feature. Negative samples include a third feature, which is a multimodal feature obtained by identifying the content theme of the second sample media resource. The second sample media resource is any media resource different from the first sample media resource. Therefore, this embodiment adopts a self-supervised training mode, constructing positive and negative samples based on the first feature of the first sample media resource during training. This reduces the cost of acquiring sample data and increases data utilization efficiency. The vector representation model is invoked to obtain the first representation vector corresponding to the first feature, the second representation vector corresponding to the positive sample, and the third representation vector corresponding to the negative sample. Multimodal contrastive learning is performed on the first and second representation vectors to obtain the first similarity between them. Furthermore, multimodal contrastive learning is performed on the first and third representation vectors to obtain the second similarity between them. The vector representation model is trained according to the rule of increasing the first similarity and decreasing the second similarity. In the scenario of data analysis of the first media resource, the trained vector representation model is used to generate representation vectors representing the content theme of the first media resource, so as to recommend a set of media resources similar to the content theme of the first media resource based on the representation vector of the first media resource. As can be seen, the embodiments of this application can train a vector representation model that can express the content theme of any media resource. Applying the trained vector representation model to a business recommendation scenario can generate a representation vector that represents the content theme of the media resource. Based on this representation vector, media resource recommendation is performed, which realizes media resource recommendation at the content theme granularity and effectively improves the recommendation accuracy in business recommendation scenarios (especially business recommendation scenarios with high requirements for content theme matching).

[0017] On one hand, embodiments of this application provide a media resource processing apparatus, the apparatus comprising:

[0018] The display unit is used to display the data analysis interface in response to data analysis operations on the first media resource;

[0019] The processing unit is used to output a data analysis report of the first media resource in the data analysis interface. The data analysis report includes recommendation prompts, which are used to suggest a set of media resources that match the content theme of the first media resource.

[0020] The display unit is also used to display media resources in the media resource set in response to the triggering operation of the recommendation prompt information.

[0021] In one embodiment, the data analysis report includes at least one data diagnostic indicator; the processing unit may be specifically used for:

[0022] The data analysis interface displays at least one data diagnostic indicator in separate areas; and,

[0023] Display recommended tips in the display area of ​​any data diagnostic indicator.

[0024] In one embodiment, the data analysis report includes at least one data optimization suggestion; the processing unit may be specifically used for:

[0025] Display at least one data optimization suggestion in a separate area within the data analysis interface; and,

[0026] Display recommended tips in the display area of ​​any data optimization suggestion.

[0027] In one embodiment, the processing unit outputs a data analysis report of the first media resource in the data analysis interface, and is further configured to:

[0028] Display the differences in indicators between the primary media resource and the media resources within the media resource set in the display area for one or more data diagnostic indicators; or,

[0029] Display the differences in metrics between the first media resource and the media resources in the media resource set in the display area of ​​one or more data optimization suggestions.

[0030] In one embodiment, the media resource set includes P media resources that match the content theme of the first media resource, where P is a positive integer; the display unit can be specifically used for:

[0031] In response to a trigger action on the recommended information, display P media resources from the media resource set; or,

[0032] In response to a trigger action on the recommended information, the second media resource in the media resource set is displayed. The second media resource refers to a media resource in the media resource set whose playback popularity exceeds a preset popularity threshold. Playback popularity is determined based on at least one of the following: number of views, number of followers, number of comments, and number of shares; or...

[0033] In response to a trigger action on a recommendation message, display a third-party media resource within the media resource set, where the value of that third-party media resource on a specified data diagnostic indicator exceeds a preset threshold; or...

[0034] In response to the triggering of the recommendation prompt, the fourth media resource in the media resource set is displayed. The proximity between the publishing object of the fourth media resource and the publishing object of the first media resource is greater than a preset threshold.

[0035] In one embodiment, the processing unit is further configured to:

[0036] The content themes of the primary media resources are identified and processed to obtain the multimodal features of the primary media resources;

[0037] The trained vector representation model is invoked to perform vector representation processing on the multimodal features of the first media resource, resulting in a representation vector of the first media resource. The representation vector of the first media resource is used to characterize the content theme of the first media resource.

[0038] Based on the representation vector of the first media resource, media resources that match the content theme of the first media resource are searched from the resource library. The similarity between the representation vector of the matching media resource and the representation vector of the first media resource is greater than a preset similarity threshold.

[0039] Add matching media resources to the media resource collection and generate recommendation prompts.

[0040] In one embodiment, media resources in the resource library are clustered into N resource clusters according to their release time, with each resource cluster corresponding to a release time period; N is a positive integer; the processing unit can be specifically used for:

[0041] Determine the first resource cluster with the latest release time from the resource library in reverse chronological order;

[0042] The representation vector of the first media resource is matched with the representation vectors of each media resource in the first resource cluster to obtain the first search result. The first search result includes media resources whose similarity between the representation vector matched from the first resource cluster and the representation vector of the first media resource is greater than a preset similarity threshold.

[0043] In one embodiment, the processing unit is further configured to:

[0044] If no media resource with a similarity greater than a preset similarity threshold is found between the representation vector of the first media resource and the representation vector of the first media resource from the first resource cluster, or if the number of media resources with a similarity greater than a preset similarity threshold between the representation vector of the first media resource and the representation vector of the first media resource from the first resource cluster does not reach the recommended number threshold Q, then the second resource cluster with the latest release time, excluding the first resource cluster, is determined from the resource library in reverse chronological order.

[0045] The representation vector of the first media resource is matched with the representation vectors of each media resource in the second resource cluster.

[0046] In one embodiment, the processing unit is further configured to:

[0047] The representation vector of the first media resource and the first media resource are clustered into the corresponding resource cluster in the resource library according to the release time.

[0048] On the one hand, embodiments of this application provide another media resource processing apparatus, which includes:

[0049] The acquisition unit is used to acquire the first feature of the first sample media resource. The first feature is a multimodal feature obtained by identifying and processing the content theme of the first sample media resource.

[0050] The processing unit is used to construct positive and negative samples of the first feature. The positive sample contains the second feature, which is obtained by data augmentation processing of the first feature. The negative sample contains the third feature, which is a multimodal feature obtained by identifying the content theme of the media resource of the second sample. The media resource of the second sample is any media resource that is different from the media resource of the first sample.

[0051] The processing unit is also used to call the vector representation model to obtain the first representation vector corresponding to the first feature, the second representation vector corresponding to the positive sample, and the third representation vector corresponding to the negative sample, respectively.

[0052] The processing unit is further configured to perform multimodal contrastive learning on the first representation vector and the second representation vector to obtain a first similarity between the first representation vector and the second representation vector; and to perform multimodal contrastive learning on the first representation vector and the third representation vector to obtain a second similarity between the first representation vector and the second representation vector.

[0053] The processing unit is also used to train the vector representation model according to the rule of increasing the first similarity and decreasing the second similarity;

[0054] In the scenario of data analysis of the first media resource, the trained vector representation model is used to generate representation vectors that characterize the content theme of the first media resource, so as to recommend a set of media resources similar to the content theme of the first media resource based on the representation vector of the first media resource.

[0055] In one embodiment, the multimodality includes a first mode and a second mode; the processing unit is specifically used for:

[0056] The content theme of the first sample media resource is processed by the identification of the first modality to obtain the first modality feature. The first modality feature is used to characterize the content theme of the first sample media resource from the dimension of the first modality.

[0057] The content theme of the first sample media resource is processed by the second modality recognition to obtain the second modality feature. The second modality feature is used to characterize the content theme of the first sample media resource from the dimension of the second modality.

[0058] The first modal feature and the second modal feature are spliced ​​together to form the first feature of the first sample media resource.

[0059] In one embodiment, the first sample media resource includes text, audio, and images; the first modality is a text modality, and the second modality is an image modality; the processing unit is specifically used for:

[0060] The text in the first sample media resource is processed by text recognition to obtain the first text information;

[0061] The audio in the first sample media resource is processed by audio recognition to obtain the second text information;

[0062] The images in the first sample media resources are processed to convert them into text, resulting in third text information.

[0063] Entity identification is performed on the first sample media resource to obtain entity information;

[0064] At least one type of information is extracted from the first text information, the second text information, the third text information, and the entity information, and feature extraction processing is performed to obtain the first modality feature.

[0065] In one embodiment, the first sample media resource includes an image; the first modality is a text modality, and the second modality is an image modality; the processing unit is specifically used for:

[0066] Image recognition processing is performed on the images in the first sample media resources to obtain the second modality features.

[0067] In one embodiment, multimodal contrastive learning is performed by invoking a multimodal model; the first features include image features and text features;

[0068] Data augmentation processing includes at least one of the following: synonym replacement of text features, random repetition of text features, random cropping of text features, position swapping of character features included in text features, flipping of image features, translation of image features, and noise reduction of image features.

[0069] On one hand, embodiments of this application provide a computer device, the computer device comprising:

[0070] A processor is used to execute computer programs;

[0071] A computer-readable storage medium storing a computer program, which, when executed by a processor, implements the processing method for the media resources described above.

[0072] On one hand, embodiments of this application provide a computer-readable storage medium storing a computer program that is loaded by a processor and executes the processing method for the media resources described above.

[0073] On one hand, embodiments of this application provide a computer program product, which includes a computer program or computer instructions, and when the computer program or computer instructions are executed by a processor, they implement the above-mentioned media resource processing method. Attached Figure Description

[0074] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0075] Figure 1 An architecture diagram of a media resource-based processing system provided as an exemplary embodiment of this application;

[0076] Figure 2 This is a schematic diagram of a data processing flow provided in an exemplary embodiment of this application;

[0077] Figure 3 This is a flowchart illustrating a method for processing media resources provided in an exemplary embodiment of this application;

[0078] Figure 4a A schematic diagram of a data analysis interface is provided as an exemplary embodiment of this application;

[0079] Figure 4bA schematic diagram of a data analysis interface is provided for another exemplary embodiment of this application;

[0080] Figure 4c A schematic diagram of a data analysis interface is provided as yet another exemplary embodiment of this application;

[0081] Figure 5a A schematic diagram of a data analysis report provided for an exemplary embodiment of this application;

[0082] Figure 5b A schematic diagram of a data analysis report provided for another exemplary embodiment of this application;

[0083] Figure 6 A schematic diagram illustrating a recommendation prompt information display provided as an exemplary embodiment of this application;

[0084] Figure 7 A schematic diagram of a media resource display provided as an exemplary embodiment of this application;

[0085] Figure 8 A schematic diagram illustrating a display media resource as an exemplary embodiment of this application;

[0086] Figure 9 This is a flowchart illustrating a method for processing media resources provided in another exemplary embodiment of this application;

[0087] Figure 10 A schematic diagram of the structure of a media resource processing apparatus provided as an exemplary embodiment of this application;

[0088] Figure 11 A schematic diagram of the structure of a media resource processing apparatus provided as an exemplary embodiment of this application;

[0089] Figure 12 This is a schematic diagram of the structure of a computer device provided for an exemplary embodiment of this application. Detailed Implementation

[0090] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of this application.

[0091] The relevant technical terms provided in the embodiments of this application are described below.

[0092] I. Media Resources

[0093] Media resources refer to various types of media materials and content, including but not limited to: text resources, image resources, audio and video resources, game resources, etc. In streaming scenarios, a media resource can be a feed within a feed stream (also known as a dynamic stream or information stream). A feed can be a video, a news article, a message, an image, etc. A feed stream is an information stream that continuously updates and presents content (i.e., feeds) to objects. For example, a Timeline is a way to display a feed stream; the Timeline displays content to objects in the chronological order of its updates.

[0094] This application's embodiments involve multimodal features reflecting the content theme of media resources. These multimodal features include modal features across multiple modalities, including but not limited to: text modality, image modality, audio modality, video modality, etc. Modal features under different modalities can characterize the content theme of media resources from the corresponding modal dimension. For example, text modal features are used to characterize the content theme of media resources from the text modal dimension; image modal features are used to characterize the content theme of media resources from the image modal dimension.

[0095] II. Vector Representation Model and Multimodal Model

[0096] Vector representation models are models that convert features (such as the multimodal features mentioned in this application) into vectors (typically arrays of numbers). Vector representation models can map high-dimensional features to low-dimensional dense vectors to capture semantic information. In embodiments of this application, vector representation models can perform cross-modal alignment processing on multimodal features, mapping multimodal features to the same vector space to obtain representation vectors for the multimodal features. Since multimodal features include modal features across multiple modalities, the representation vectors for multimodal features can include representation vectors corresponding to various modal features. In one implementation, the vector representation model can be an emb model (also known as an embedding model or embedding model).

[0097] A multimodal model (or multi-model large model) is a machine learning model capable of understanding and processing information from multiple modalities. Multimodal models can fuse and analyze information from multiple different modalities to perform more complex and intelligent tasks. In the embodiments of this application, the multimodal model can fuse representation vectors of multimodal features, which facilitates subsequent multimodal comparative learning.

[0098] III. Multimodal Comparative Learning

[0099] Multimodal contrastive learning is a self-supervised learning method that aims to learn a unified semantic representation across modalities by comparing the semantic relationships between different modalities (such as images, text, and audio). This brings different modalities describing the same semantic meaning closer together in the vector space, while pushing semantically unrelated modalities further apart. Multimodal contrastive learning eliminates the need for extensive manually labeled data during model training. In this embodiment, multimodal contrastive learning is performed on different representation vectors to determine their similarity, thereby training the vector representation model. The trained model can bring similar multimodal features closer together in the vector space and push dissimilar multimodal features further apart.

[0100] This application provides a media resource processing solution that enables data analysis of any media resource to generate a data analysis report, which is then visualized and presented intuitively through a data analysis interface. Furthermore, the data analysis report includes recommendation prompts suggesting a set of media resources that match the content theme of any given media resource. This achieves the goal of recommending media resources that match the content theme of any given media resource, thus enabling a media resource recommendation service at the content theme granularity and improving the accuracy of media resource recommendations. In one implementation, the scheme can include two parts: model training and model application. (1) Model training: This part uses the multimodal features of sample media resources as the data basis for model training. Since the multimodal features cover multiple types of data (such as text, images, audio, etc.), this provides more comprehensive and accurate information for the model training process. In addition, the multimodal features can reflect the content theme of the media resources, which enables the trained model to provide media resource recommendation services at the content theme granularity. Furthermore, this part adopts a self-supervised training mode, which constructs positive and negative samples based on the features of sample media resources during the training process. This can reduce the cost of obtaining sample data and improve data utilization efficiency. In addition, this part can train a vector representation model that can express the content theme of any media resource. (2) Model application: The trained vector representation model is applied to the media resource analysis scenario. During the data analysis of any media resource (such as the first media resource), a representation vector representing the content theme of the first media resource can be generated. Then, based on the representation vector, a set of media resources that match the content theme can be recommended for the first media resource, and recommendation prompts for the set of media resources can be generated. This not only realizes media resource recommendation at the content theme level and effectively improves the accuracy of media resource recommendation, but also effectively guides the media resource publishing objects to compare and optimize media resources in a timely manner through the output recommendation prompts, thereby improving the quality of media resources.

[0101] For example, the first media resource is a video of a child posted by an advertiser, and the video's content theme includes cute and adorable children accompanied by background music. Using the above method, data analysis can be performed on this video, generating a representation vector that characterizes its content theme. Based on this representation vector, a set of videos (i.e., a set of media resources) with similar content themes to the child-posting video can be recommended, and a data analysis report can be generated. This report may include recommendation prompts suggesting video sets that match the video's content theme. Thus, the advertiser can access the data analysis interface to view the data analysis results for their posted video and can trigger the recommendation prompts to view the recommended video sets as needed. As can be seen, the above solution enables a more comprehensive analysis of the content themes of media resources, recommending similar media resources to the target audience. For the target audience, this allows them to view the recommended media resources and gain inspiration and creative techniques, improving their content creation abilities and promoting continuous improvement and innovation, thus enhancing the quality of media resources. Furthermore, the target audience can make more informed creative decisions based on the recommended videos, learning from the recommended media resources to create better content for publication, reducing trial-and-error costs and increasing creative efficiency. In addition, by comparing the recommended media resources with their own published media resources, the target audience can analyze the shortcomings of their published media resources and make targeted improvements, thereby enhancing the overall quality of their media resources.

[0102] The media resource processing system provided in the embodiments of this application will be described next.

[0103] Please see Figure 1 This is an architectural diagram of a media resource processing system provided in an exemplary embodiment of this application. Figure 1The media resource processing system may include terminal device 101, terminal device 102, and so on, as well as server 103. This application does not limit the number of terminal devices; similarly, the number of servers can also be multiple. Terminal devices 101 and 102 can be smartphones, tablets, laptops, desktop computers, smart vehicles, smart wearable devices, etc. Server 103 can be an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms. Terminal devices 101 and 102 in the media resource processing system can be directly or indirectly connected to server 103 via wired or wireless communication, and terminal devices 101 and 102 can interact with each other through server 103.

[0104] Figure 1 The interaction process between the terminal device and the server shown can be combined Figure 2 Provide a detailed description. Figure 2 This is a schematic diagram of a data processing flow provided in an exemplary embodiment of this application. See also: Figure 2 The data processing flow between the terminal device and the server includes the following three parts: (1) model training; (2) online process; and (3) media resource analysis and recommendation. These three parts will be elaborated upon below:

[0105] (1) Model Training: In this part, we mainly use media resources in the business recommendation scenario to train a vector representation model, so that the vector representation model can represent the multimodal features that characterize the content topic of any media resource into vectors, thereby mapping the multimodal features to the same vector space. The model training includes the following steps:

[0106] ① Server 103 obtains the first sample media resource. Specifically, media resources can be randomly selected from the business recommendation scenario as the first sample media resource. The number of first sample media resources can be one or more, such as 1 million, 2 million, etc. The following explanation will focus on the first sample media resource.

[0107] ② Server 103 acquires the first feature of the first sample media resource. In one implementation, multimodal recognition processing can be performed on the content theme of the first sample media resource to obtain modal features under each modality. The modal features under any modality can be used to characterize the content theme of the first sample media resource from the dimension of the corresponding modality. Multiple modalities may include, but are not limited to: image modality and text modality; for example, text recognition processing can be performed on the text in the first sample media resource (such as titles, tags of the first sample media resource, etc.), audio recognition processing can be performed on the audio in the first sample media resource, and image-to-text conversion can be performed on the images in the first sample media resource, etc., to obtain modal features under the text modality (hereinafter referred to as first modal features). Image recognition processing can be performed on the images in the first sample media resource to obtain image modal features under the image modality (hereinafter referred to as second modal features).

[0108] ③ Server 103 constructs positive and negative samples for the first feature. Specifically, positive samples may include the second feature, and negative samples may include the third feature. A second feature similar to the first feature (similarity greater than a preset value) and a third feature dissimilar to the first feature (similarity less than a preset value) can be constructed. Constructing a second feature similar to the first feature can be achieved by performing data augmentation on the first feature to obtain the second feature, thus making the second feature similar to the first feature to a certain degree. Constructing a third feature dissimilar to the first feature can be achieved by randomly selecting second sample media resources from the business recommendation scenario and determining the multimodal features obtained by identifying the content theme of the second sample media resources as the third feature; by randomly selecting second sample media resources, the third feature is made dissimilar to the first feature to a certain extent. As can be seen, positive and negative samples of the first feature are constructed through the two methods described above, so that the first feature forms a positive sample pair with the positive sample and a negative sample pair with the negative sample. The labels of the positive and negative sample pairs come from the data itself. That is, the label of the first feature comes from the positive and negative samples it has constructed. In this way, there is no need for manual labeling, and self-supervised training of the model is achieved.

[0109] ④ Server 103 inputs the first feature, the second feature, and the third feature into the vector representation model for vector representation, and obtains the first representation vector corresponding to the first feature, the second representation vector corresponding to the positive sample, and the third representation vector corresponding to the negative sample.

[0110] ⑤ Server 103 performs multimodal comparison learning on the first representation vector and the second representation vector, and performs multimodal comparison learning on the first representation vector and the third representation vector. For example... Figure 2In this process, the first and second representation vectors can be input into the multimodal model for multimodal comparative learning, and the first and third representation vectors can be input into the multimodal model for multimodal comparative learning.

[0111] ⑥ Server 103 trains the vector representation model based on the multimodal contrastive learning results, so that the distance between the first representation vector of the first feature and the second representation vector of the positive sample in the same vector space is closer, and the distance between the first representation vector of the first feature and the third representation vector of the negative sample in the same vector space is farther.

[0112] In summary, this application utilizes multimodal features as the data foundation for model training. Since multimodal features cover various data types (such as text, images, and audio), this provides more comprehensive and accurate information for the model training process. Furthermore, multimodal features can reflect the content themes of media resources, enabling the trained model to provide media resource recommendation services at the content theme granularity. In one implementation, the model training in this application can be performed offline. Offline training refers to the process of training the model using historical data in a non-production environment. "Production environment" refers to a critical stage in the software development and deployment lifecycle where the application or system is actually running and providing services.

[0113] (2) Online Process: After obtaining the trained vector representation model, it can be deployed online. Online deployment refers to publishing the trained model to the production environment and enabling it to process requests in real time. In this embodiment, the trained vector representation model can be applied to data analysis scenarios of media resources. The application process includes the following steps:

[0114] A. Server 103 verifies the online and offline consistency of the vector representation model after training. Online and offline consistency refers to ensuring that the model deployed online maintains the same consistency as the model trained offline during model training and application. Illustratively, online and offline consistency can mean that, under the same input features, the same model's prediction results for offline data should be consistent with its prediction results for online data, or that the difference between the prediction results for offline data and the prediction results for online data should be less than a preset difference (e.g., the preset difference could be 0.00001, 0.00002, etc.). For example, after obtaining the trained vector representation model, the online and offline vector representation models can be used to process the multimodal features of the same media resource to obtain the representation vector of the media resource in the online state and the representation vector in the offline state. If the difference between the representation vectors in the two states is less than a preset value, then it can be determined that the trained vector representation model is consistent online and offline, and step B can be executed; if the difference between the representation vectors in the two states is greater than or equal to the preset value, it indicates that the trained vector representation model is inconsistent online and offline, and it may be necessary to retrain the vector representation model.

[0115] B. If the trained vector representation model maintains consistency online and offline, then server 103 obtains the published media resources, identifies the content themes of the published media resources, and obtains the multimodal features of the published media resources; it then calls the trained vector representation model to perform vector representation processing on the multimodal features of the published media resources, and obtains the representation vector of the published media resources.

[0116] C. Server 103 associates and stores the published media resources with their corresponding representation vectors in the resource library. It is evident that by utilizing the online-deployed trained vector representation model, the representation vectors of media resources published by different publishing entities in the business recommendation scenario can be collected into the resource library. This facilitates subsequent data analysis of any media resource, enabling analysis based on content theme (i.e., the content theme expressed by the representation vectors of each media resource) using the resource library, thereby achieving the recommendation of media resources with similar content themes.

[0117] (3) Media Resource Analysis and Recommendation: This section mainly explains how to perform data analysis on media resources published by any publishing object and recommend media resources with similar content themes. The interaction between terminal device 101 and server 103 is used as an example for illustration.

[0118] ① The terminal device 101 runs a client for media resource recommendation. In response to data analysis operations on the first media resource in the client (such as triggering data analysis options associated with the first media resource, clicking on the first media resource directly, etc.), the terminal device 101 can generate a data analysis request for the target media resource. This data analysis request is used to request the server 103 to perform data analysis on the first media resource and recommend media resources that match the first media resource.

[0119] ② The server 103 receives the data analysis request and performs data analysis on the first media resource. In one implementation, the data analysis of the first media resource can be performed according to at least one data diagnostic indicator (such as the playback rate of the media resource, the response rate of the media resource, etc.). During the data analysis process, the content theme of the first media resource can be identified and processed to obtain the multimodal features of the target media resource.

[0120] ③ Server 103 inputs the multimodal features of the first media resource into the trained vector representation model for vector representation processing to obtain the representation vector of the first media resource.

[0121] ④ Server 103 searches the resource library for recommended media resources that match the content theme of the first media resource based on the representation vector of the first media resource. Since the representation vector of a media resource is used to represent the content theme of the media resource, searching for media resources that match the content theme of the first media resource can be understood as: searching for media resources in the resource library whose representation vectors are more similar to the representation vector of the first media resource than a preset similarity threshold as media resources that match the content theme of the first media resource.

[0122] ⑤ Server 103 can add matching media resources to the media resource set, generate recommendation prompts, and generate a data analysis report based on the analysis results of the recommendation prompts and various data diagnostic indicators.

[0123] ⑥ The terminal device 101 receives the data analysis report returned by the server 103 and displays the data analysis report in the data analysis interface so that the publishing object can view the data analysis report and trigger recommendation prompts to view the recommended media resource set as needed.

[0124] As can be seen, in this embodiment, the trained vector representation model can generate representation vectors that characterize the content theme of the first media resource. The similarity between the representation vectors of each media resource in the resource library and the representation vector of the first media resource reflects the similarity between certain media resources in the resource library and the first media resource in terms of content theme. This allows for the recommendation of videos similar to the first media resource in terms of content theme, achieving media resource recommendation at the content theme level and effectively improving recommendation accuracy in business recommendation scenarios. Furthermore, this embodiment enables data analysis of media resources and presents the data analysis results visually and intuitively through a data analysis interface. This facilitates media resource publishers to clearly understand the specific data of the media resources and effectively guides them to compare and optimize media resources in a timely manner through recommendations, thereby improving the quality of media resources.

[0125] It should be understood that the above data processing interaction process is for illustrative purposes only and does not limit the specific execution process of the terminal device and the server. Optionally, the model training involved in this embodiment can be executed by the server 103, while the online process and the media resource analysis and recommendation process can be executed by the terminal device. For example, in the online process, the terminal device deploys the trained vector representation model. Specifically, the server 103 can export the trained vector representation model in ONNX (Open Neural Network Exchange) format and deploy it on the terminal device. The terminal device calls the trained vector representation model to perform vector representation of the multimodal features of the media resources and stores the media resources and the corresponding representation vectors in the resource library of the server 103. In the media resource analysis and recommendation process, the terminal device can analyze the media resources and obtain the resource library from the server 103 to recommend media resources. Optionally, the above model training, online process, and media resource analysis and recommendation process can all be executed by the terminal device. Among them, ONNX is an open file format designed for machine learning models and used to store trained models.

[0126] The processing method for media resources provided in the embodiments of this application will be described below.

[0127] Please see Figure 3 This is a flowchart illustrating a media resource processing method provided in an exemplary embodiment of this application. The media resource processing method can be executed by a computer device, which can be any of the aforementioned terminals. The media resource processing method may include the following steps S301-S303:

[0128] S301, In response to the data analysis operation on the first media resource, display the data analysis interface.

[0129] The first media resource can be any media resource published by the publishing object in the business recommendation scenario. Step S301 may include at least one of the following: ① This application embodiment provides corresponding data analysis options for any published media resource, and the publishing object can trigger the data analysis option of the first media resource to realize the data analysis operation of the first media. Please refer to Figure 4a This is a schematic diagram illustrating a data analysis interface display, provided as an exemplary embodiment of this application. Figure 4a In the first media resource, there is a data analysis option 41, and in the second media resource, there is a data analysis option 42. In response to a trigger operation (such as a click) on the data analysis option 41, the data analysis interface 402 can be displayed. ② A data analysis entry point is provided. The publishing object can trigger this entry point to perform data analysis operations on the first media resource, and through this entry point, it can view data analysis reports for all media resources published by the publishing object. For example, Figure 4b This is a schematic diagram illustrating a data analysis interface display, provided as another exemplary embodiment of this application. Figure 4b In the middle, a data analysis entry 43 is displayed. In response to a trigger operation on the data analysis entry 43, a data analysis interface 402 can be displayed. ③ The data analysis operation can also be an operation of clicking the first media resource multiple times. In response to the operation of clicking the first media resource multiple times, the data analysis results can be displayed. It should be understood that the embodiments of this application do not limit the display location of the data analysis options or the data analysis entry, such as displaying it on the media resource publishing interface, the object information interface corresponding to the publishing object, etc.

[0130] S302. Output a data analysis report of the first media resource in the data analysis interface. The data analysis report contains recommendation prompts, which are used to suggest a set of media resources that match the content theme of the first media resource.

[0131] The media resource set includes at least one media resource, and the content theme of each media resource in the set matches the content theme of the first media resource. In one implementation, such as... Figure 4a As shown, the data analysis interface 402 only outputs the data analysis report 44 for the first media resource, which includes recommendation prompts 45. In another implementation, the data analysis interface can output data analysis reports for multiple media resources, such as... Figure 4b As shown, the data analysis interface 402 outputs a data analysis report 46 for the first media resource, and the data analysis report 46 includes recommendation prompts 47; and outputs a data analysis report 48 for media resource 1, and the data analysis report 48 includes recommendation prompts 49.

[0132] In addition to recommendation information, the data analysis report may include at least one of the following: data diagnostic indicators and data optimization suggestions. Data diagnostic indicators vary depending on the media resource. For example, if the media resource is video or audio, the data diagnostic indicators may include, but are not limited to: basic information of the media resource (such as the title, resource tags, etc.), response rate for the media resource, playback rate in the first N seconds, completion rate (referring to the playback completion rate of the media resource), etc. Similarly, if the media resource is text, the data diagnostic indicators may include, but are not limited to: basic information of the media resource (such as the title, resource tags, etc.), response rate for the media resource, viewing time for the media resource, etc. Data optimization suggestions indicate the optimization angles for the primary media resource, such as optimizing the title of the media resource, increasing interaction with the viewers of the primary media resource, etc.

[0133] ① The data analysis report can include at least one data diagnostic indicator; specifically, outputting the data analysis report of the primary media resource in the data analysis interface can include: displaying at least one data diagnostic indicator in different areas of the data analysis interface; for example, such as Figure 4b In the data analysis interface 402, four data diagnostic indicators are displayed in separate areas (namely: basic information, response rate, playback rate in the first M seconds, and completion rate). Optionally, any one of the data diagnostic indicators can be triggered to display the corresponding data optimization suggestions.

[0134] ② The data analysis report can include at least one data optimization suggestion. Outputting the data analysis report for the primary media resource in the data analysis interface can include displaying at least one data optimization suggestion in different areas of the data analysis interface. For example, Figure 4c In the data analysis interface, Data Optimization Suggestion 1 and Data Optimization Suggestion 2 are displayed in separate areas. By outputting data optimization suggestions, the publishing entity can easily optimize its media resources and improve their quality.

[0135] ③ The data analysis report may include at least one data diagnostic indicator and at least one data optimization suggestion. In one implementation, a data diagnostic indicator may correspond to one or more data optimization suggestions; that is, the data optimization suggestion is generated based on the data diagnostic indicator. Outputting the data analysis report of the first media resource in the data analysis interface may include: displaying at least one data diagnostic indicator in the data analysis interface; and displaying the data optimization suggestions corresponding to each data diagnostic indicator by region. For example, Figure 4aIn the data analysis interface, basic information, response rate, first M seconds playback rate, completion rate and other data diagnostic indicators are displayed. In area 411, data optimization suggestions are displayed for the data diagnostic indicator "basic information"; in area 412, data optimization suggestions are displayed for the data diagnostic indicator "response rate"; and in area 413, data optimization suggestions are displayed for the data diagnostic indicator "first M seconds playback rate".

[0136] Optionally, the data analysis report output in the data analysis interface for the first media resource may also include: displaying the indicator differences between the first media resource and the media resources in the media resource set. These indicator differences may refer to the differences between the first media resource and the media resources in the media resource set under a certain data diagnostic indicator. For example, these indicator differences may include, but are not limited to: resource differences between the first media resource and the media resources in the media resource set in "Basic Information" (e.g., the first media resource lacks a title, while the media resources in the media resource set have prominent titles), differences in completion rates between the first media resource and the media resources in the media resource set under "Completion Rate," etc. Alternatively, these indicator differences may refer to the first media resource's indicator value under a certain data diagnostic indicator being lower or higher than L% (L is a positive number less than 100) of the media resources in the media resource set. For example, these indicator differences may refer to the first media resource's indicator value under a certain data diagnostic indicator being lower than 60% of the media resources in the media resource set.

[0137] The display of the differences in indicators between the first media resource and the media resources in the media resource set may include at least one of the following: (1) displaying the differences in indicators between the first media resource and the media resources in the media resource set in the display area of ​​one or more data diagnostic indicators. For example, such as Figure 5a The diagram shown is a schematic representation of a data analysis report provided in an exemplary embodiment of this application; Figure 5a In the data diagnostic indicator "Playback Rate in the First M Seconds", the indicator difference 51 between the first media resource and the media resources in the media resource set is displayed in display area 52, and the indicator difference between the first media resource and the media resources in the media resource set is displayed in display area 53 of the data diagnostic indicator "Completion Rate". (2) The indicator difference between the first media resource and the media resources in the media resource set is displayed in the display area of ​​one or more data optimization suggestions. For example, such as Figure 5b The diagram shown is a schematic representation of a data analysis report provided in another exemplary embodiment of this application; Figure 5b In the data optimization suggestion 2, the display area 53 shows the difference in indicators 54 between the first media resource and the media resources in the media resource set (i.e., less than 60% of similar media resources).

[0138] Furthermore, the display format of the recommendation prompts involved in the embodiments of this application is not limited in any way. For example, the recommendation prompts can be displayed as icons, text, controls, links, etc. In addition, the display area of ​​the recommendation prompts can be flexibly set. The display area of ​​the recommendation prompts may include, but is not limited to: ① displaying the recommendation prompts in the display area of ​​any data diagnostic indicator. For example, in... Figure 6 In the data diagnostic indicator "Top M Playback Rate", recommended prompt information 62 can be displayed in display area 61. This recommended prompt information is displayed in the form of a control. ② Recommended prompt information can be displayed in the display area of ​​any data optimization suggestion. For example... Figure 4c In the data optimization suggestion 2 display area, recommendation prompt information 400 can be displayed. ③ Recommendation prompt information can be displayed in the display area of ​​any data diagnostic indicator and in the display area of ​​any data optimization suggestion. ④ Recommendation prompt information can be displayed in target areas other than the display areas of any data diagnostic indicator and any data optimization suggestion. For example... Figure 4b In this embodiment, recommended prompts 47 can be displayed in the target area (such as the upper right corner) of the data analysis interface. It is evident that the recommended prompts in this embodiment can be displayed in different areas. For the publishing target, they can flexibly choose to trigger recommended prompts in any display area to quickly view the media resource set, thus improving the publishing target's experience.

[0139] S303. In response to the triggering operation of the recommendation prompt information, display the media resources in the media resource set.

[0140] The triggering operation can be a click or drag operation on the recommended prompt information. The media resource set includes at least one media resource. When the publishing object triggers the recommended prompt information, some or all of the media resources in the media resource set can be displayed. Let the media resource set contain P media resources that match the content theme of the first media resource, where P is a positive integer. In response to the triggering operation on the recommended prompt information, the media resources displayed in the media resource set can include any of the following:

[0141] (1) All media resources included in the media resource set are recommended. In response to a trigger action (such as selection) on the recommendation prompt, P media resources from the media resource set are displayed. For example, Figure 7 The diagram shown is a schematic representation of a media resource display provided in an exemplary embodiment of this application. Figure 7In response to the selection of recommended prompts, three (P=3) media resources in the media collection can be displayed. By displaying all media resources in the collection, the publisher can understand the content of multiple media resources, enabling a more comprehensive comparison and learning between their first published media resource and the various media resources in the collection, thereby improving the quality of their published media resources.

[0142] (2) In response to the triggering operation of the recommendation prompt information, randomly display media resources in the media resource set.

[0143] (3) Media resources are selected from the media resource set based on playback popularity for recommendation. In one implementation, in response to the triggering operation of the recommendation prompt information, a second media resource in the media resource set is displayed. This second media resource refers to a media resource in the media resource set whose playback popularity is higher than a preset popularity threshold. The number of second media resources is one or more. Recommending media resources based on playback popularity realizes the recommendation of media resources with high playback popularity and high content theme similarity to the media resources. This allows the publishing object to understand the content that the audience currently browsing the media resources wants to pay attention to, so that the publishing object can learn and create media resources with high playback popularity. Playback popularity is determined based on at least one of the following: number of views, number of followers, number of comments, and number of shares. ① Playback popularity is determined based on the number of views: the higher the number of views of a media resource, the higher the popularity of the media resource. In one implementation, the number of views of a media resource can be directly determined as the playback popularity of the media resource. ② Playback popularity is determined based on the number of comments: the higher the number of comments of a media resource, the higher the playback popularity of the media resource. In one implementation, the number of comments of a media resource can be directly determined as the playback popularity of the media resource. ③ Playback popularity is determined based on the number of views: the higher the number of views a media resource receives, the higher its playback popularity. In one implementation, the number of views can be directly used to determine the playback popularity of a media resource. ④ Playback popularity is determined based on the number of shares: the higher the number of shares a media resource receives, the higher its playback popularity. In one implementation, the number of shares can be directly used to determine the playback popularity of a media resource.

[0144] It should be understood that the playback popularity of media resources in this application embodiment can be determined based on at least two of the following: playback volume, number of views, number of comments, and number of shares. For example, if the playback popularity of media resources is determined based on playback volume and number of views, the weights of playback volume and number of views can be set separately, such as a weight of 0.7 for playback volume and a weight of 0.3 for number of views. The playback of media resources is determined by setting weights, that is, the playback volume and number of views of media resources are weighted and summed as the playback popularity of media resources.

[0145] In summary, the embodiments of this application provide multiple methods for determining playback popularity, allowing for flexible selection of the determination method according to needs. This enables the filtering of media resources with high playback popularity from the media resource set, thus achieving better media resource recommendation.

[0146] Optionally, when there are multiple secondary media resources, they can be displayed in descending order of popularity. Furthermore, the popularity of each secondary media resource can be displayed, allowing the recipient to view the popularity of each resource and choose the one with the higher popularity based on their needs.

[0147] (4) Select media resources from the media resource set for recommendation based on data diagnostic indicators. In one implementation, in response to a trigger operation on the recommendation prompt information, a third media resource in the media resource set is displayed. This third media resource has an indicator value on a specified data diagnostic indicator that is higher than a preset indicator threshold. The number of third media resources can be one or more. Specifically, the indicator values ​​of media resources in the media resource set on a specified data diagnostic indicator can be obtained, and media resources with indicator values ​​on the specified data diagnostic indicator that are higher than the preset indicator threshold can be selected from the media resource set as third media resources. For example, if the media resource is a video and the specified data diagnostic indicator is "first 3 seconds playback rate", then the third media resource can be a video in the media resource set with a first 3 seconds playback rate greater than the preset indicator threshold; or, the third media resource can be the media resource with the highest first 3 seconds playback rate in the media resource set. By displaying media resources with indicator values ​​on the specified data diagnostic indicator that are higher than the preset indicator threshold, the publishing target can understand the data status of each media resource on the specified data diagnostic indicator.

[0148] In this embodiment, data diagnostic indicators can be specified according to needs. For example, if it is necessary to view the "first 3 seconds playback rate" of a video, then the specified data diagnostic indicator can be directly set to "first 3 seconds playback rate". Alternatively, the data diagnostic indicator can be specified according to the display position of the recommendation prompt information. For example, Figure 8 This is a schematic diagram illustrating a display media resource as an exemplary embodiment of this application. Figure 8 In the middle, "Recommendation Tip 71" is displayed in the display area of ​​the data diagnostic indicator "Completion Rate", so the specified data diagnostic indicator can be "Completion Rate".

[0149] Optionally, when there are multiple third-party media resources, they can be displayed in descending order of their index values ​​on a specified data diagnostic metric. Furthermore, the playback popularity of each third-party media resource can be displayed, allowing the target audience to understand the playback popularity of each third-party media resource and choose the one with the higher playback popularity based on their needs. For example... Figure 8 The text can display two media resources in a media resource cluster: Media Resource 1 has a completion rate of 90%, and Media Resource 2 has a completion rate of 80%.

[0150] (5) Select media resources from the media resource set for recommendation based on the affinity between the publishing objects of the media resources. In one implementation, in response to a trigger operation of the recommendation prompt information, a fourth media resource in the media resource set is displayed, wherein the affinity between the publishing object of the fourth media resource and the publishing object of the first media resource is greater than a preset threshold. The number of fourth media resources can be one or more. In response to a trigger operation of the recommendation prompt information, the affinity between the publishing objects of each media resource in the media resource set and the publishing object of the first media resource can be obtained, and the media resources corresponding to the publishing objects whose affinity with the publishing objects of the first media resource is greater than a preset threshold are determined as the fourth media resources. Selecting media resources for recommendation based on the affinity between the publishing objects can increase the probability of the publishing objects viewing the media resources, thus improving the effectiveness of media recommendations.

[0151] The determination of the intimacy between the publishing objects of various media resources and the publishing object of the first media resource can include the following methods: Method 1: The higher the intimacy between publishing objects with social relationships, the higher the intimacy. In one implementation, if two publishing objects have social relationships, the intimacy between them can be set to the first value; if they do not have social relationships, the intimacy between them can be set to the second value. Method 2: Determine the intimacy between publishing objects based on the amount of interaction between them (such as mutual comments, likes, etc.). For example, the more interaction between publishing objects, the higher the intimacy between them.

[0152] In summary, for publishers, viewing media resources allows for better creation of content, improving the quality of their work and increasing user satisfaction. This enhances their personalized experience and overall engagement. Furthermore, providing intelligent support and optimization tools attracts more publishers to create content, enriching the content ecosystem and improving the overall quality of media resources. Recommending media resources with similar themes to the target content promotes interaction among publishers, encouraging learning and exchange, fostering a positive creative community. Data-driven optimization suggestions further refine media content, enhancing interactivity and improving the viewing experience.

[0153] In this embodiment, in response to the data analysis operation on the first media resource, a data analysis interface is displayed, and a data analysis report of the first media resource is output in the data analysis interface. Therefore, this application can perform data analysis on media resources and present the data analysis results visually and intuitively through the data analysis interface. This allows the recipients of the media resources to clearly understand the specific data of the media resources. Furthermore, the data analysis report also includes recommendation prompts, which suggest a set of media resources that match the content theme of the first media resource. In response to the triggering operation of the recommendation prompts, the media resources in the media resource set are displayed. In other words, during the data analysis of the media resource, this application can also recommend media resources that match the content theme of the media resource. This not only realizes a media resource recommendation service at the content theme level, improving the accuracy of media resource recommendations, but also effectively guides the recipients of the media resources to compare and optimize media resources in a timely manner, thereby improving the quality of the media resources.

[0154] The following describes the generation of media resource sets and recommendation prompts involved in the embodiments of this application. In one embodiment, the generation method of media resource sets and recommendation prompts includes:

[0155] S91. Identify and process the content theme of the first media resource to obtain the multimodal features of the first media resource.

[0156] Specifically, a first media resource can be acquired, and its content theme can be identified and processed to obtain its multimodal features. How to obtain these multimodal features will be explained in subsequent embodiments and will not be repeated here.

[0157] S92. Call the trained vector representation model to perform vector representation processing on the multimodal features of the first media resource to obtain the representation vector of the first media resource. The representation vector of the first media resource is used to characterize the content theme of the first media resource.

[0158] S93. Based on the representation vector of the first media resource, search the resource library for media resources that match the content theme of the first media resource. The similarity between the representation vector of the matching media resource and the representation vector of the first media resource is greater than a preset similarity threshold.

[0159] Since the multimodal features of media resources are used to characterize the content theme of the corresponding media resources, finding recommended media resources that match the target media resource means finding media resources that are similar to or the same as the first media resource in terms of content theme.

[0160] In one implementation, S93 may include: determining the similarity between the representation vector of the first media resource and the representation vectors of each media resource in the resource library. The similarity between the representation vector of a media resource and the representation vector of the first media resource reflects the similarity between the media resources in the resource library and the first media resource in terms of content theme. The higher the similarity, the more similar the media resources in the resource library are to the first media resource in terms of content theme. Media resources whose representation vectors have a similarity greater than a preset similarity threshold to the representation vector of the first media resource can be selected from the resource library as media resources that match the content theme of the first media resource. The preset similarity threshold is set according to requirements.

[0161] In another implementation, the media resources in the resource library are clustered into N resource clusters according to their release time. Each resource cluster corresponds to a release time period, where N is a positive integer. Each resource cluster includes at least one media resource. For example, the media resources in the resource library are clustered into 3 (i.e., N=3) resource clusters according to their release time. These 3 resource clusters include: resource cluster 1 corresponding to the release time period "January 1st - January 31st, XX year", resource cluster 2 corresponding to the release time period "February 1st - February 31st, XX year", and resource cluster 3 corresponding to the release time period "March 1st - March 31st, XX year". In this implementation, S93 may specifically include: ① determining the first resource cluster with the latest release time period from the resource library in reverse chronological order. ② matching the representation vector of the first media resource with the representation vectors of each media resource in the first resource cluster to obtain the first search result. In one implementation, the similarity between the representation vector of the first media resource and the representation vectors of each media resource in the first resource cluster can be calculated, and it can be determined whether the similarity between the representation vector of the first media resource and the representation vectors of each media resource in the first resource cluster is greater than a preset similarity threshold. If a media resource whose representation vector is matched from the first resource cluster has a similarity greater than the preset similarity threshold, then the first search result includes: media resources whose representation vector is matched from the first resource cluster and whose representation vector is matched from the first media resource has a similarity greater than the preset similarity threshold. ③ If no media resource whose representation vector is matched from the first resource cluster has a similarity greater than the preset similarity threshold, or if the number of media resources whose representation vector is matched from the first resource cluster has a similarity greater than the preset similarity threshold does not reach the recommended quantity threshold Q, then the second resource cluster with the latest release time, excluding the first resource cluster, is determined from the resource library in reverse chronological order. The recommended quantity threshold Q can be set according to requirements, such as a recommended quantity threshold of 3, 5, etc., and this application embodiment does not impose any limitation on this. ④ Match the representation vector of the first media resource with the representation vectors of each media resource in the second resource cluster to determine the media resource that matches the content theme of the first media resource.

[0162] For example, among the three resource clusters mentioned above, resource cluster 3, which has the latest release time, can be determined from the resource library in reverse chronological order. Then, the representation vector of the first media resource is matched with the representation vectors of each media resource in resource cluster 3. If no media resource with a similarity greater than a preset similarity threshold is found in the first resource cluster, the resource cluster 2, which has the latest release time (excluding resource cluster 3), is determined from the resource library in reverse chronological order. The representation vector of the first media resource is then matched with the representation vectors of each media resource in resource cluster 2. At this point, media resources with a similarity greater than a preset similarity threshold are found in resource cluster 2. These media resources are then identified as media resources that match the content theme of the first media resource. At this point, the number of media resources with a similarity greater than a preset similarity threshold reaches the recommended number threshold. This application embodiment clusters media resources in the resource library according to their release time periods, which allows for searching in reverse chronological order, reducing the complexity of media resource matching and improving the efficiency of media resource recommendation.

[0163] Optionally, the first media resource and its representation vector can be clustered into corresponding resource clusters according to their publication time.

[0164] In this embodiment, a resource library can be pre-constructed before matching. This resource library includes multiple media resources and their representation vectors. The specific implementation of constructing the resource library is as follows: Media resources published in the business recommendation scenario are acquired, and the content themes of the media resources are identified to obtain multimodal features. Then, a trained vector representation model is called to perform vector representation processing on the multimodal features of the media resources to obtain the target representation vector of the media resources. The media resources and their corresponding representation vectors are then stored in the resource library, or the media resources and their corresponding representation vectors are clustered into corresponding resource clusters according to their publication time. It should be understood that when newly published media resources exist in the business recommendation scenario, the representation vector of the newly published media resources can be directly obtained and stored in the resource library. This can, to a certain extent, ensure that the resource library contains relatively new media resources so that newer media resources can be recommended subsequently. Optionally, media resources in the resource library whose publication time is earlier than a preset publication time can be deleted. This can free up resource storage space while ensuring that the media resources in the resource library are relatively new.

[0165] S94. Add the matching media resources to the media resource collection and generate recommendation prompts.

[0166] In this embodiment, the content theme of the first media resource is identified to obtain its multimodal features. A trained vector representation model is then used to represent these multimodal features, resulting in a representation vector for the first media resource. Based on this representation vector, media resources matching the content theme of the first media resource are searched in the resource library. If the similarity between the representation vector of the matching media resource and the representation vector of the first media resource is greater than a preset similarity threshold, the matching media resource is added to the media resource set, and recommendation suggestions are generated. Therefore, this embodiment demonstrates that the trained vector representation model can generate representation vectors expressing the content theme of media resources. Furthermore, the similarity between the representation vector of the media resource and the representation vector of the first media resource reflects their similarity in content theme, enabling the recommendation of media resources similar to the first media resource in content theme. This achieves media resource recommendation at the content theme level, effectively improving the accuracy of media resource recommendations.

[0167] The training process for the vector representation model will be described below using embodiments of this application. Please refer to [link / reference]. Figure 9 This is a flowchart illustrating a media resource processing method according to an exemplary embodiment of this application. The media resource processing method can be executed by a computer device, which can be any of the aforementioned terminals or servers; the media resource processing method may include the following steps S901-S905:

[0168] S901. Obtain the first feature of the first sample media resource. The first feature is a multimodal feature obtained by identifying and processing the content theme of the first sample media resource.

[0169] The first sample media resource can be any media resource, including text, audio, images, etc.; for example, the first sample media resource is a video; a video describes the process of playing a fun game, and the content theme of the video can be playing "fun games" indoors. Multimodal features include modal features under different modalities. In one implementation, step S901 may include: performing multimodal recognition processing on the content theme of the first sample media resource to obtain modal features under each modality. The modal features under each modality are used to represent the content theme of the first sample media resource from the dimension of the corresponding modality, and the modal features under each modality are concatenated to form the first feature of the first sample media resource. As one implementation, multimodality includes at least a first modality and a second modality, and the multimodal features may include first modal features under the dimension of the first modality and second modal features under the dimension of the second modality. At this point, the content theme of the first sample media resource is subjected to first modality recognition processing to obtain first modality features, which are used to characterize the content theme of the first sample media resource from the first modality dimension; the content theme of the first sample media resource is subjected to second modality recognition processing to obtain second modality features, which are used to characterize the content theme of the first sample media resource from the second modality dimension; the first modality features and the second modality features are concatenated to form the first feature of the first sample media resource.

[0170] The first modality can be a text modality, and the second modality can be an image modality. The features of the first modality can be text features, and the features of the second modality can be image features. The content theme of the first sample media resource is processed to identify the first modality, resulting in the first modality features. This can be achieved through the following methods:

[0171] Method 1: The first media resource includes text, which can be used to express the content theme of the first sample media resource. Text recognition processing can be performed on the text in the first sample media resource to obtain the first text information, and feature extraction can be performed on the first text information to obtain the first modal features. The text included in the first media resource may include, but is not limited to: title, attribute tags (e.g., if the first media resource is a video, then the attribute tags could be video type (e.g., food, fitness), shooting time, etc.). If the text includes a title, text recognition processing can be performed on the title in the first sample media resource to obtain the first text information.

[0172] Method 2: The first media resource includes audio, which can be used to express the content theme of the first sample media resource. Therefore, audio recognition processing can be performed on the audio in the first sample media resource to obtain the second text information. As one implementation method, ASR (Automatic Speech Recognition) technology can be used to perform audio recognition processing on the audio in the first sample media resource to obtain the second text information, and feature extraction can be performed on the second text information to obtain the first modality features.

[0173] Method 3: The first media resource includes an image, which can be used to express the content theme of the first sample media resource. Then, the image in the first media resource can be processed to convert it into text, obtaining third text information. For example, OCR (Optical Character Recognition) technology can be used to convert the image in the first media resource into text, obtaining third text information, and then feature extraction can be performed on the third text information to obtain first modality features. In one implementation, the first media resource is a video; frame extraction can be performed on the video to obtain images from the video.

[0174] Method 4: Perform entity recognition on the primary media resource to obtain entity information. Entity information here can include: names of people, place names, proper nouns, etc. As one method, an entity recognition model can be used to perform entity recognition on the primary media resource to obtain entity information.

[0175] Method 5: If the first media resource includes at least two of the following: text, audio, and images, then at least one type of information can be extracted from the first text information, the second text information, the third text information, and the entity information to perform feature extraction processing and obtain the first modality feature.

[0176] Specifically, the content theme of the first sample media resources is processed for second-modal identification to obtain second-modal features, including the following implementation methods:

[0177] Method 1: The first sample media resource includes images, which can be static or dynamic. When the first sample media resource is a video, the image can be any frame from the first sample media resource, or a key image from the first sample media resource. A key image can refer to an image that embodies the content theme of the first sample media resource. For example, if the first sample media resource is a "tea ceremony video," then the key image could be a frame containing teaware, a frame containing an object brewing tea, etc.; similarly, if the first sample media resource is a "food video," then the key image could be a frame containing food. In this implementation, performing second-modal recognition processing on the content theme of the first sample media resource to obtain second-modal features can include: performing image recognition processing (such as feature extraction, scene understanding, encoding, etc.) on the images in the first sample media resource to obtain second-modal features.

[0178] Method 2: If the first sample media resource does not include images, a text-to-image format can also be used. Based on the text information obtained from processing the first sample media resource (such as the first text information, the second text information, and the third text information mentioned above), an image expressing the content theme of the first sample media resource is generated, and image recognition processing is performed on the generated image to obtain the second modality features.

[0179] S902, Construct positive and negative samples for the first feature.

[0180] In this context, a positive sample refers to a sample that is similar to the first feature (with a similarity greater than a preset value). The positive sample includes a second feature, which is obtained by data augmentation of the first feature. The first feature includes image features and text features. Data augmentation can include at least one of the following: text augmentation (i.e., performing EDA (EasyData Augmentation, a simple but very effective text data augmentation method) on text features) and image augmentation. Text augmentation includes at least one of the following: synonym replacement of text features, random repetition of text features, random cropping of text features (e.g., randomly removing character features from text features with probability P), and position swapping of character features included in text features (e.g., if text features include a first character feature and a second character feature, their positions can be swapped within the text features). Image augmentation includes at least one of the following: image feature flipping, image feature translation, and image feature noise reduction (e.g., adding noise to image features). Data augmentation can make the constructed positive samples have a certain similarity to the first feature. The methods for constructing positive samples for the first feature can include, but are not limited to, the following:

[0181] Method (1): Image enhancement processing is performed on the image features to obtain the second feature, that is, the second feature only includes the image features after image enhancement.

[0182] Method (2): Perform text enhancement processing on the text features to obtain the second feature, that is, the second feature only includes the text features after text enhancement.

[0183] Method (3): Perform image enhancement processing on the image features, and replace the image features in the first feature with the enhanced image features to obtain the second feature. That is, the second feature includes the enhanced image features and the text features. For example, perform a flipping process on the image features to obtain the flipped image features, and replace the image features in the first feature with the flipped image features to obtain the second feature, which includes the flipped image features and the text features.

[0184] Method (4): Perform text enhancement processing on the text features, and replace the text features in the first feature with the text enhancement text features to obtain the second feature. That is, the second feature includes the image features and the text enhancement text features. For example, randomly crop the text features to obtain the randomly cropped text features. The randomly cropped text features can replace the text features in the first feature to obtain the second feature. The second feature includes the image features and the randomly cropped text features.

[0185] Method (5): Perform image enhancement processing on image features and text enhancement processing on text features, and replace the image features in the first feature with the image features after image enhancement, and replace the text features in the first feature with the text features after text enhancement to obtain the second feature, which includes the image features after image enhancement and the text features after text enhancement.

[0186] It should be understood that the embodiments of this application can construct a second feature that is similar to the first feature in any of the above-described ways, and the embodiments of this application do not limit this in any way.

[0187] Here, a negative sample refers to a sample that is not similar to the first feature (the similarity is less than a preset value); the negative sample includes a third feature, which is a multimodal feature obtained by identifying the content theme of the second sample media resource. The second sample media resource is any media resource that is different from the first sample media resource. In this implementation, constructing a negative sample for the first feature may include: randomly obtaining the second sample media resource from the business recommendation scenario, and identifying the content theme of the second sample media resource to obtain the second feature.

[0188] It should be understood that by constructing positive and negative samples of the first feature, the first feature and the positive sample form a positive sample pair, and the first feature and the negative sample form a negative sample pair. This allows the vector representation model to learn the similarity between the first feature and the positive sample in the positive sample pair, and the difference between the first feature and the negative sample in the negative sample pair. This enables the vector representation model to map multimodal features to the same vector space, so that similar multimodal features are closer together, while dissimilar multimodal features are farther apart.

[0189] S903. Call the vector representation model to obtain the first representation vector corresponding to the first feature, the second representation vector corresponding to the positive sample, and the third representation vector corresponding to the negative sample.

[0190] In one implementation, the first feature, positive samples, and negative samples can be input into a vector representation model for vector representation processing to obtain a first representation vector corresponding to the first feature, a second representation vector corresponding to the positive samples, and a second representation vector corresponding to the negative samples. In this embodiment, vector representation refers to cross-modal alignment of features from different modalities, that is, mapping features from different modalities to the same vector space. In this case, the first feature can be input into the vector representation model for vector space mapping to obtain the first representation vector corresponding to the first feature; the second feature included in the positive samples can be input into the vector representation space for vector space mapping to obtain the second representation vector corresponding to the positive samples; and the third feature included in the negative samples can be input into the vector representation space for vector space mapping to obtain the second representation vector corresponding to the negative samples.

[0191] The vector representation model can include a projection layer and an output layer. Taking the first representation vector corresponding to the first feature as an example, the first feature is cross-modal aligned through the projection layer (that is, the features of different modalities are mapped to the same vector space) to obtain the first representation vector corresponding to the first feature, and the first representation vector corresponding to the first feature is output through the output layer.

[0192] S904. Perform multimodal contrastive learning on the first representation vector and the second representation vector to obtain the first similarity between the first representation vector and the second representation vector; and perform multimodal contrastive learning on the first representation vector and the third representation vector to obtain the second similarity between the first representation vector and the second representation vector.

[0193] In one implementation, multimodal contrastive learning can be performed by a multimodal model, which includes a feature fusion module (such as a Transformer) and a multimodal task processing module. Step S304 specifically includes: calling the feature fusion module to perform feature fusion on the representation vectors of each modality included in the first representation vector, obtaining the fused features of the first representation vector; similarly, the fused features of the second and third representation vectors can be obtained. Feature fusion can achieve information complementarity between modal features of different modalities, which facilitates more accurate determination of the similarity between representation vectors. The multimodal task processing module is then called to predict the similarity between the fused features of the first and second representation vectors, obtaining a first similarity between the first and second representation vectors; and similarity prediction is performed between the fused features of the first and third representation vectors, obtaining a second similarity between the first and third representation vectors.

[0194] S905. Train the vector representation model according to the rule of increasing the first similarity and decreasing the second similarity.

[0195] In the scenario of data analysis of the first media resource, the trained vector representation model is used to generate a representation vector representing the content theme of the first media resource, so as to recommend a set of media resources similar to the content theme of the first media resource based on the representation vector of the first media resource. In the embodiments of this application, the model parameters of the vector representation model are continuously adjusted according to the rule of increasing the first similarity and decreasing the second similarity until the model converges, and the trained vector representation model is obtained. For example, the indicators for judging whether the model has converged may include, but are not limited to, any of the following: (1) Construct a loss function based on the first similarity and the second similarity. When the loss value of the loss function is less than the preset loss value, the model has converged; (2) Construct a loss function based on the first similarity and the second similarity. When the loss value of the loss function tends to be stable (i.e., it remains within a certain loss range without changing), the model has converged; (3) When the gradient of the model parameters is close to zero, it means that the adjustment required for updating the model parameters is very small, and the model has converged; (4) When the update magnitude of the model parameters gradually decreases, the model has converged. The trained vector representation model can be used to generate representation vectors that express the content themes of any media resource. For two similar multimodal features, after being mapped to the same vector space by the trained vector representation model, the two similar multimodal features will be closer in the same vector space. For two dissimilar multimodal features, after being mapped to the same vector space by the trained vector representation model, the two dissimilar multimodal features will be farther apart in the same vector space.

[0196] In summary, this application utilizes multimodal features as the data foundation for model training. Since multimodal features cover various data types (such as text, images, and audio), this provides more comprehensive and accurate information for the model training process. Furthermore, multimodal features can reflect the content themes of media resources, enabling the trained model to provide media resource recommendation services at the content theme granularity. In addition, this application adopts a self-supervised training mode, constructing positive and negative samples based on the first features of the first sample media resource during training. This reduces the cost of acquiring sample data and improves data utilization efficiency. Moreover, this application can train a vector representation model capable of expressing the content theme of any media resource. Applying the trained vector representation model to business recommendation scenarios can generate representation vectors expressing the content theme of media resources. Media resource recommendations are then performed based on these representation vectors, achieving media resource recommendation at the content theme granularity, effectively improving recommendation accuracy in business recommendation scenarios (especially those with high content theme matching requirements).

[0197] The media resource processing apparatus provided in the embodiments of this application will now be described.

[0198] Please see Figure 10 , Figure 10 This is a schematic diagram of the structure of a media resource processing apparatus according to an exemplary embodiment of this application. The media resource processing apparatus can be a computer program (including program code) in a computer device; for example, the media resource processing apparatus can be application software in a computer device. The media resource processing apparatus can be used to execute... Figure 3 Some or all of the steps in the method embodiments shown. Please refer to [link / reference]. Figure 10 The media resource processing device includes the following units:

[0199] Display unit 1001 is used to display a data analysis interface in response to a data analysis operation on the first media resource;

[0200] The processing unit 1002 is used to output a data analysis report of the first media resource in the data analysis interface. The data analysis report includes recommendation prompts, which are used to suggest a set of media resources that match the content theme of the first media resource.

[0201] The display unit 1001 is also used to display media resources in the media resource set in response to the triggering operation of the recommendation prompt information.

[0202] In one embodiment, the data analysis report includes at least one data diagnostic indicator; the processing unit 1002 may be specifically used for:

[0203] The data analysis interface displays at least one data diagnostic indicator in separate areas; and,

[0204] Display recommended tips in the display area of ​​any data diagnostic indicator.

[0205] In one embodiment, the data analysis report includes at least one data optimization suggestion; the processing unit 1002 may be specifically used for:

[0206] Display at least one data optimization suggestion in a separate area within the data analysis interface; and,

[0207] Display recommended tips in the display area of ​​any data optimization suggestion.

[0208] In one embodiment, the processing unit 1002, which outputs a data analysis report of the first media resource in the data analysis interface, is further configured to:

[0209] Display the differences in indicators between the primary media resource and the media resources within the media resource set in the display area for one or more data diagnostic indicators; or,

[0210] Display the differences in metrics between the first media resource and the media resources in the media resource set in the display area of ​​one or more data optimization suggestions.

[0211] In one embodiment, the media resource set includes M media resources that match the content theme of the first media resource, where M is a positive integer; the display unit 1001 can be specifically used for:

[0212] In response to a trigger action on the recommended suggestion information, display M media resources from the media resource set; or,

[0213] In response to a trigger action on the recommended information, the second media resource in the media resource set is displayed. The second media resource refers to a media resource in the media resource set whose playback popularity exceeds a preset popularity threshold. Playback popularity is determined based on at least one of the following: number of views, number of followers, number of comments, and number of shares; or...

[0214] In response to a trigger action on a recommendation message, display a third-party media resource within the media resource set, where the value of that third-party media resource on a specified data diagnostic indicator exceeds a preset threshold; or...

[0215] In response to the triggering of the recommendation prompt, the fourth media resource in the media resource set is displayed. The proximity between the publishing object of the fourth media resource and the publishing object of the first media resource is greater than a preset threshold.

[0216] In one embodiment, the processing unit 1002 is further configured to:

[0217] The content themes of the primary media resources are identified and processed to obtain the multimodal features of the primary media resources;

[0218] The trained vector representation model is invoked to perform vector representation processing on the multimodal features of the first media resource, resulting in a representation vector of the first media resource. The representation vector of the first media resource is used to characterize the content theme of the first media resource.

[0219] Based on the representation vector of the first media resource, media resources that match the content theme of the first media resource are searched from the resource library. The similarity between the representation vector of the matching media resource and the representation vector of the first media resource is greater than a preset similarity threshold.

[0220] Add matching media resources to the media resource collection and generate recommendation prompts.

[0221] In one embodiment, the media resources in the resource library are clustered into N resource clusters according to their release time, with each resource cluster corresponding to a release time period; N is a positive integer; the processing unit 1002 can be specifically used for:

[0222] Determine the first resource cluster with the latest release time from the resource library in reverse chronological order;

[0223] The representation vector of the first media resource is matched with the representation vectors of each media resource in the first resource cluster to obtain the first search result. The first search result includes media resources whose similarity between the representation vector matched from the first resource cluster and the representation vector of the first media resource is greater than a preset similarity threshold.

[0224] In one embodiment, the processing unit 1002 is further configured to:

[0225] If no media resource with a similarity greater than a preset similarity threshold is found between the representation vector of the first media resource and the representation vector of the first media resource from the first resource cluster, or if the number of media resources with a similarity greater than a preset similarity threshold between the representation vector of the first media resource and the representation vector of the first media resource from the first resource cluster does not reach the recommended number threshold M, then the second resource cluster with the latest release time, excluding the first resource cluster, is determined from the resource library in reverse chronological order.

[0226] The representation vector of the first media resource is matched with the representation vectors of each media resource in the second resource cluster.

[0227] In one embodiment, the processing unit 1002 is further configured to:

[0228] The representation vector of the first media resource and the first media resource are clustered into the corresponding resource cluster in the resource library according to the release time.

[0229] In response to a data analysis operation on a first media resource, this application displays a data analysis interface and outputs a data analysis report for the first media resource within the interface. Therefore, this application can perform data analysis on media resources and present the analysis results visually and intuitively through the data analysis interface. This allows the recipients of the media resources to clearly understand the specific data details of the media resources. Furthermore, the data analysis report includes recommendation prompts to suggest a set of media resources that match the content theme of the first media resource. In response to triggering the recommendation prompts, the media resources in the set are displayed. In other words, during the data analysis of a media resource, this application can also recommend media resources that match its content theme. This achieves a media resource recommendation service at the content theme level, improving the accuracy of media resource recommendations. It also effectively guides recipients of the media resources to compare and optimize them in a timely manner, thereby improving the quality of the media resources.

[0230] Please see Figure 11 , Figure 11 This is a schematic diagram of the structure of a media resource processing apparatus according to an exemplary embodiment of this application. The media resource processing apparatus can be a computer program (including program code) in a computer device; for example, the media resource processing apparatus can be application software in a computer device. The media resource processing apparatus can be used to execute... Figure 9 Some or all of the steps in the method embodiments shown. Please refer to [link / reference]. Figure 11 The media resource processing device includes the following units:

[0231] The acquisition unit 1101 is used to acquire a first feature of the first sample media resource, wherein the first feature is a multimodal feature obtained by identifying and processing the content theme of the first sample media resource.

[0232] The processing unit 1102 is used to construct positive samples and negative samples of the first feature. The positive sample contains the second feature, which is obtained by performing data augmentation processing on the first feature. The negative sample contains the third feature, which is a multimodal feature obtained by performing content theme recognition processing on the media resource of the second sample. The media resource of the second sample is any media resource that is different from the media resource of the first sample.

[0233] The processing unit 1102 is also used to call the vector representation model to obtain the first representation vector corresponding to the first feature, the second representation vector corresponding to the positive sample, and the third representation vector corresponding to the negative sample, respectively.

[0234] The processing unit 1102 is further configured to perform multimodal comparison learning on the first representation vector and the second representation vector to obtain a first similarity between the first representation vector and the second representation vector; and to perform multimodal comparison learning on the first representation vector and the third representation vector to obtain a second similarity between the first representation vector and the second representation vector.

[0235] The processing unit 1102 is also used to train the vector representation model according to the rule of increasing the first similarity and decreasing the second similarity;

[0236] In the scenario of data analysis of the first media resource, the trained vector representation model is used to generate a representation vector representing the content theme of the first media resource, so as to recommend a set of media resources similar to the content theme of the first media resource based on the representation vector of the first media resource.

[0237] In one embodiment, the multimodal mode includes a first mode and a second mode; the processing unit 1102 is specifically used for:

[0238] The content theme of the first sample media resource is processed by the identification of the first modality to obtain the first modality feature. The first modality feature is used to characterize the content theme of the first sample media resource from the dimension of the first modality.

[0239] The content theme of the first sample media resource is processed by the second modality recognition to obtain the second modality feature. The second modality feature is used to characterize the content theme of the first sample media resource from the dimension of the second modality.

[0240] The first modal feature and the second modal feature are spliced ​​together to form the first feature of the first sample media resource.

[0241] In one embodiment, the first sample media resource includes text, audio, and images; the first modality is a text modality, and the second modality is an image modality; the processing unit 1102 is specifically used for:

[0242] The text in the first sample media resource is processed by text recognition to obtain the first text information;

[0243] The audio in the first sample media resource is processed by audio recognition to obtain the second text information;

[0244] The images in the first sample media resources are processed to convert them into text, resulting in third text information.

[0245] Entity identification is performed on the first sample media resource to obtain entity information;

[0246] At least one type of information is extracted from the first text information, the second text information, the third text information, and the entity information, and feature extraction processing is performed to obtain the first modality feature.

[0247] In one embodiment, the first sample media resource includes an image; the first modality is a text modality, and the second modality is an image modality; the processing unit 1102 is specifically used for:

[0248] Image recognition processing is performed on the images in the first sample media resources to obtain the second modality features.

[0249] In one embodiment, multimodal contrastive learning is performed by invoking a multimodal model; the first features include image features and text features;

[0250] Data augmentation processing includes at least one of the following: synonym replacement of text features, random repetition of text features, random cropping of text features, position swapping of character features included in text features, flipping of image features, translation of image features, and noise reduction of image features.

[0251] In this embodiment, a first feature of the first sample media resource can be obtained. This first feature is a multimodal feature obtained by identifying the content theme of the first sample media resource. Therefore, this embodiment utilizes multimodal features as the data foundation for model training. Since multimodal features cover multiple modal types of data (such as text, images, and audio), this provides more comprehensive and accurate information for the model training process. Furthermore, multimodal features can reflect the content theme of the media resource, enabling the trained model to provide recommendation services for media resources at the content theme granularity. Positive and negative samples of the first feature are constructed. Positive samples include a second feature, which is obtained by data augmentation processing of the first feature. Negative samples include a third feature, which is a multimodal feature obtained by identifying the content theme of the second sample media resource. The second sample media resource is any media resource different from the first sample media resource. Therefore, this embodiment adopts a self-supervised training mode, constructing positive and negative samples based on the first feature of the first sample media resource during training. This reduces the cost of acquiring sample data and increases data utilization efficiency. The vector representation model is invoked to obtain a first representation vector corresponding to a first feature, a second representation vector corresponding to a positive sample, and a third representation vector corresponding to a negative sample. Multimodal comparison learning is performed on the first and second representation vectors to obtain a first similarity between them. Furthermore, multimodal comparison learning is performed on the first and third representation vectors to obtain a second similarity between them. The vector representation model is trained according to the rule of increasing the first similarity and decreasing the second similarity. Therefore, this embodiment can train a vector representation model capable of expressing the content theme of any media resource. Applying the trained vector representation model to a business recommendation scenario can generate representation vectors expressing the content theme of media resources. Media resource recommendations are then performed based on these representation vectors, achieving media resource recommendation at the content theme granularity, effectively improving recommendation accuracy in business recommendation scenarios (especially those with high requirements for content theme matching).

[0252] The computer device provided in the embodiments of this application will now be described.

[0253] Furthermore, this application also provides a schematic diagram of the structure of a computer device, which can be found in [reference needed]. Figure 12The computer device may include a processor 1201, an input device 1202, an output device 1203, and a memory 1204. The processor 1201, input device 1202, output device 1203, and memory 1204 are connected via a bus. The memory 1204 stores computer programs, including program instructions, and the processor 1201 executes the program instructions stored in the memory 1204.

[0254] In one embodiment, the processor 1201 performs the following operations by running program instructions stored in memory 1204:

[0255] In response to data analysis operations on the primary media resource, the data analysis interface is displayed;

[0256] The data analysis interface outputs a data analysis report for the first media resource. The data analysis report includes recommendation prompts, which are used to suggest a set of media resources that match the content theme of the first media resource.

[0257] In response to a trigger action on a recommendation message, display media resources from the media resource collection.

[0258] In one embodiment, the data analysis report includes at least one data diagnostic indicator; when the processor 1201 outputs the data analysis report of the first media resource in the data analysis interface, it may specifically perform the following operations:

[0259] The data analysis interface displays at least one data diagnostic indicator in separate areas; and,

[0260] Display recommended tips in the display area of ​​any data diagnostic indicator.

[0261] In one embodiment, the data analysis report includes at least one data optimization suggestion; when the processor 1201 outputs the data analysis report of the first media resource in the data analysis interface, it may specifically perform the following operations:

[0262] Display at least one data optimization suggestion in a separate area within the data analysis interface; and,

[0263] Display recommended tips in the display area of ​​any data optimization suggestion.

[0264] In one embodiment, when the processor 1201 outputs a data analysis report of the first media resource in the data analysis interface, it may also perform the following operations:

[0265] Display the differences in indicators between the primary media resource and the media resources within the media resource set in the display area for one or more data diagnostic indicators; or,

[0266] Display the differences in metrics between the first media resource and the media resources in the media resource set in the display area of ​​one or more data optimization suggestions.

[0267] In one embodiment, the media resource set includes P media resources that match the content theme of the first media resource, where P is a positive integer;

[0268] When processor 1201 displays media resources from the media resource set in response to a trigger operation on the recommendation prompt information, it may specifically perform the following operations:

[0269] In response to a trigger action on the recommended information, display P media resources from the media resource set; or,

[0270] In response to a trigger action on the recommended information, the second media resource in the media resource set is displayed. The second media resource refers to a media resource in the media resource set whose playback popularity exceeds a preset popularity threshold. Playback popularity is determined based on at least one of the following: number of views, number of followers, number of comments, and number of shares; or...

[0271] In response to a trigger action on a recommendation message, display a third-party media resource within the media resource set, where the value of that third-party media resource on a specified data diagnostic indicator exceeds a preset threshold; or...

[0272] In response to the triggering of the recommendation prompt, the fourth media resource in the media resource set is displayed. The proximity between the publishing object of the fourth media resource and the publishing object of the first media resource is greater than a preset threshold.

[0273] In one embodiment, the processor 1201 may also perform the following operations:

[0274] The content themes of the primary media resources are identified and processed to obtain the multimodal features of the primary media resources;

[0275] The trained vector representation model is invoked to perform vector representation processing on the multimodal features of the first media resource, resulting in a representation vector of the first media resource. The representation vector of the first media resource is used to characterize the content theme of the first media resource.

[0276] Based on the representation vector of the first media resource, media resources that match the content theme of the first media resource are searched from the resource library. The similarity between the representation vector of the matching media resource and the representation vector of the first media resource is greater than a preset similarity threshold.

[0277] Add matching media resources to the media resource collection and generate recommendation prompts.

[0278] In one embodiment, the media resources in the resource library are clustered into N resource clusters according to their release time, and each resource cluster corresponds to a release time period; N is a positive integer.

[0279] When processor 1201 searches for media resources in the resource library that match the content theme of the first media resource based on the representation vector of the first media resource, it may specifically perform the following operations:

[0280] Determine the first resource cluster with the latest release time from the resource library in reverse chronological order;

[0281] The representation vector of the first media resource is matched with the representation vectors of each media resource in the first resource cluster to obtain the first search result. The first search result includes media resources whose similarity between the representation vector matched from the first resource cluster and the representation vector of the first media resource is greater than a preset similarity threshold.

[0282] In one embodiment, when the processor 1201 searches the resource library for a media resource that matches the content theme of the first media resource based on the representation vector of the first media resource, it may also perform the following operations:

[0283] If no media resource with a similarity greater than a preset similarity threshold is found between the representation vector of the first media resource and the representation vector of the first media resource from the first resource cluster, or if the number of media resources with a similarity greater than a preset similarity threshold between the representation vector of the first media resource and the representation vector of the first media resource from the first resource cluster does not reach the recommended number threshold Q, then the second resource cluster with the latest release time, excluding the first resource cluster, is determined from the resource library in reverse chronological order.

[0284] The representation vector of the first media resource is matched with the representation vectors of each media resource in the second resource cluster to determine the media resource that matches the content theme of the first media resource.

[0285] In one embodiment, the processor 1201 may also perform the following operations:

[0286] The representation vector of the first media resource and the first media resource are clustered into the corresponding resource cluster in the resource library according to the release time.

[0287] In response to a data analysis operation on a first media resource, this application displays a data analysis interface and outputs a data analysis report for the first media resource within the interface. Therefore, this application can perform data analysis on media resources and present the analysis results visually and intuitively through the data analysis interface. This allows the recipients of the media resources to clearly understand the specific data details of the media resources. Furthermore, the data analysis report includes recommendation prompts to suggest a set of media resources that match the content theme of the first media resource. In response to triggering the recommendation prompts, the media resources in the set are displayed. In other words, during the data analysis of a media resource, this application can also recommend media resources that match its content theme. This achieves a media resource recommendation service at the content theme level, improving the accuracy of media resource recommendations. It also effectively guides recipients of the media resources to compare and optimize them in a timely manner, thereby improving the quality of the media resources.

[0288] In another implementation, processor 1201 performs the following operations by running program instructions stored in memory 1204:

[0289] The first feature of the first sample media resource is obtained by identifying and processing the content theme of the first sample media resource as a multimodal feature.

[0290] Construct positive and negative samples for the first feature. Positive samples contain the second feature, which is obtained by data augmentation of the first feature. Negative samples contain the third feature, which is a multimodal feature obtained by identifying the content theme of the media resource of the second sample. The media resource of the second sample is any media resource that is different from the media resource of the first sample.

[0291] The vector representation model is called to obtain the first representation vector corresponding to the first feature, the second representation vector corresponding to the positive sample, and the third representation vector corresponding to the negative sample.

[0292] Multimodal contrastive learning is performed on the first representation vector and the second representation vector to obtain a first similarity between the first representation vector and the second representation vector; and multimodal contrastive learning is performed on the first representation vector and the third representation vector to obtain a second similarity between the first representation vector and the second representation vector.

[0293] The vector representation model is trained according to the rule of increasing the first similarity and decreasing the second similarity;

[0294] In the scenario of data analysis of the first media resource, the trained vector representation model is used to generate representation vectors that characterize the content theme of the first media resource, so as to recommend a set of media resources similar to the content theme of the first media resource based on the representation vector of the first media resource.

[0295] In one embodiment, the multimodal includes a first modality and a second modality; when the processor 1201 acquires the first feature of the first sample media resource, it may specifically perform the following operations:

[0296] The content theme of the first sample media resource is processed by the identification of the first modality to obtain the first modality feature. The first modality feature is used to characterize the content theme of the first sample media resource from the dimension of the first modality.

[0297] The content theme of the first sample media resource is processed by the second modality recognition to obtain the second modality feature. The second modality feature is used to characterize the content theme of the first sample media resource from the dimension of the second modality.

[0298] The first modal feature and the second modal feature are spliced ​​together to form the first feature of the first sample media resource.

[0299] In one embodiment, the first sample media resource includes text, audio, and images; the first modality is a text modality, and the second modality is an image modality; when the processor 1201 performs first modality recognition processing on the content theme of the first sample media resource to obtain first modality features, it may specifically perform the following operations:

[0300] The text in the first sample media resource is processed by text recognition to obtain the first text information;

[0301] The audio in the first sample media resource is processed by audio recognition to obtain the second text information;

[0302] The images in the first sample media resources are processed to convert them into text, resulting in third text information.

[0303] Entity identification is performed on the first sample media resource to obtain entity information;

[0304] At least one type of information is extracted from the first text information, the second text information, the third text information, and the entity information, and feature extraction processing is performed to obtain the first modality feature.

[0305] The first sample media resource includes images; the first modality is text modality, and the second modality is image modality; when the processor 1201 performs second modality recognition processing on the content theme of the first sample media resource to obtain second modality features, it can specifically perform the following operations:

[0306] Image recognition processing is performed on the images in the first sample media resources to obtain the second modality features.

[0307] In one embodiment, multimodal contrastive learning is performed by invoking a multimodal model; the first features include image features and text features;

[0308] Data augmentation processing includes at least one of the following: synonym replacement of text features, random repetition of text features, random cropping of text features, position swapping of character features included in text features, flipping of image features, translation of image features, and noise reduction of image features.

[0309] In this embodiment, a first feature of the first sample media resource can be obtained. This first feature is a multimodal feature obtained by identifying the content theme of the first sample media resource. Therefore, this embodiment utilizes multimodal features as the data foundation for model training. Since multimodal features cover multiple modal types of data (such as text, images, and audio), this provides more comprehensive and accurate information for the model training process. Furthermore, multimodal features can reflect the content theme of the media resource, enabling the trained model to provide recommendation services for media resources at the content theme granularity. Positive and negative samples of the first feature are constructed. Positive samples include a second feature, which is obtained by data augmentation processing of the first feature. Negative samples include a third feature, which is a multimodal feature obtained by identifying the content theme of the second sample media resource. The second sample media resource is any media resource different from the first sample media resource. Therefore, this embodiment adopts a self-supervised training mode, constructing positive and negative samples based on the first feature of the first sample media resource during training. This reduces the cost of acquiring sample data and increases data utilization efficiency. The vector representation model is invoked to obtain a first representation vector corresponding to a first feature, a second representation vector corresponding to a positive sample, and a third representation vector corresponding to a negative sample. Multimodal comparison learning is performed on the first and second representation vectors to obtain a first similarity between them. Furthermore, multimodal comparison learning is performed on the first and third representation vectors to obtain a second similarity between them. The vector representation model is trained according to the rule of increasing the first similarity and decreasing the second similarity. Therefore, this embodiment can train a vector representation model capable of expressing the content theme of any media resource. Applying the trained vector representation model to a business recommendation scenario can generate representation vectors expressing the content theme of media resources. Media resource recommendations are then performed based on these representation vectors, achieving media resource recommendation at the content theme granularity, effectively improving recommendation accuracy in business recommendation scenarios (especially those with high requirements for content theme matching).

[0310] In this application, the term "unit" refers to a computer program or part of a computer program that has a predetermined function and works with other related parts to achieve a predetermined goal, and can be implemented wholly or partially using software, hardware (such as processing circuitry or memory), or a combination thereof. Similarly, a processor (or multiple processors or memory) can be used to implement one or more units. Furthermore, each unit can be part of an overall unit that includes the functionality of that unit.

[0311] Furthermore, embodiments of this application also provide a computer-readable storage medium storing a computer program, which includes program instructions. When a processor executes these program instructions, it can perform the aforementioned... Figure 3 and Figure 9 The methods described in the corresponding embodiments are therefore not repeated here. For technical details not disclosed in the computer-readable storage medium embodiments related to this application, please refer to the description of the method embodiments of this application. As an example, program instructions may be deployed on a computer device, executed on multiple computer devices located in one location, or executed on multiple computer devices distributed in multiple locations and interconnected through a communication network.

[0312] According to one aspect of this application, a computer program product is provided, comprising a computer program stored in a computer-readable storage medium. A processor of a computer device reads the computer program from the computer-readable storage medium, and the processor executes the computer program, enabling the computer device to perform the aforementioned... Figure 3 and Figure 9 The methods described in the corresponding embodiments are therefore not repeated here.

[0313] Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The program can be stored in a computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. The storage medium can be a magnetic disk, optical disk, read-only memory (ROM), or random access memory (RAM), etc.

[0314] The above-disclosed embodiments are merely preferred embodiments of this application and should not be construed as limiting the scope of this application. Therefore, any equivalent variations made in accordance with the claims of this application shall still fall within the scope of this application.

Claims

1. A method for processing media resources, characterized in that, include: In response to data analysis operations on the primary media resource, the data analysis interface is displayed; The data analysis interface outputs a data analysis report of the first media resource. The data analysis report includes recommendation prompts, which are used to suggest a set of media resources that match the content theme of the first media resource. In response to the triggering operation of the recommended prompt information, the media resources in the media resource set are displayed.

2. The method as described in claim 1, characterized in that, The data analysis report includes at least one data diagnostic indicator; the step of outputting the data analysis report of the first media resource in the data analysis interface includes: The data analysis interface displays the at least one data diagnostic indicator in separate areas; and, The recommended prompt information is displayed in the display area of ​​any of the aforementioned data diagnostic indicators.

3. The method as described in claim 1, characterized in that, The data analysis report includes at least one data optimization suggestion; The step of outputting a data analysis report of the first media resource in the data analysis interface includes: The data analysis interface displays at least one data optimization suggestion in separate areas. as well as, The recommended prompt information is displayed in the display area of ​​any of the data optimization suggestions.

4. The method as described in claim 2 or 3, characterized in that, The step of outputting a data analysis report of the first media resource in the data analysis interface further includes: Display the differences in indicators between the first media resource and the media resources in the media resource set in the display area of ​​one or more data diagnostic indicators; or, The display area for one or more data optimization suggestions shows the differences in metrics between the first media resource and the media resources in the media resource set.

5. The method as described in claim 1, characterized in that, The media resource set contains P media resources that match the content theme of the first media resource, where P is a positive integer; The operation of displaying media resources in the media resource set in response to the triggering of the recommendation prompt information includes any of the following: In response to the triggering operation of the recommended prompt information, display P media resources in the media resource set; or, In response to the triggering operation of the recommended prompt information, a second media resource in the media resource set is displayed. The second media resource refers to a media resource in the media resource set whose playback popularity is higher than a preset popularity threshold. The playback popularity is determined based on at least one of the following: number of views, number of followers, number of comments, and number of shares. or, In response to the triggered operation of the recommended prompt information, a third media resource in the media resource set is displayed, wherein the indicator value of the third media resource on the specified data diagnostic indicator is higher than a preset indicator threshold; or... In response to the triggering operation of the recommended prompt information, a fourth media resource in the media resource set is displayed, wherein the proximity between the publishing object of the fourth media resource and the publishing object of the first media resource is greater than a preset threshold.

6. The method as described in claim 1, characterized in that, The method further includes: The content theme of the first media resource is identified and processed to obtain the multimodal features of the first media resource; The trained vector representation model is invoked to perform vector representation processing on the multimodal features of the first media resource to obtain the representation vector of the first media resource. The representation vector of the first media resource is used to characterize the content theme of the first media resource. Based on the representation vector of the first media resource, media resources that match the content theme of the first media resource are searched from the resource library. The similarity between the representation vector of the matching media resource and the representation vector of the first media resource is greater than a preset similarity threshold. The matching media resources are added to the media resource set, and the recommendation prompt information is generated.

7. The method as described in claim 6, characterized in that, The media resources in the resource library are clustered into N resource clusters according to their release time, and each resource cluster corresponds to a release time period; N is a positive integer. The step of searching the resource library for media resources that match the content theme of the first media resource based on the representation vector of the first media resource includes: Determine the first resource cluster with the latest release time from the resource library in reverse chronological order; The representation vector of the first media resource is matched with the representation vectors of each media resource in the first resource cluster to obtain a first search result. The first search result includes media resources whose similarity between the representation vector matched from the first resource cluster and the representation vector of the first media resource is greater than a preset similarity threshold.

8. The method as described in claim 7, characterized in that, The step of searching the resource library for media resources that match the content theme of the first media resource based on the representation vector of the first media resource further includes: If no media resource with a similarity greater than a preset similarity threshold is found between the representation vector of the first media resource and the representation vector of the first media resource from the first resource cluster, or if the number of media resources with a similarity greater than a preset similarity threshold between the representation vector of the first media resource and the representation vector of the first media resource from the first resource cluster does not reach the recommended number threshold Q, then the second resource cluster with the latest release time, excluding the first resource cluster, is determined from the resource library in reverse chronological order. The representation vector of the first media resource is matched with the representation vectors of each media resource in the second resource cluster to determine the media resource that matches the content theme of the first media resource.

9. The method according to any one of claims 6-8, characterized in that, The method further includes: The representation vector of the first media resource and the first media resource are clustered into the resource clusters corresponding to the resource library according to the release time.

10. A method for processing media resources, characterized in that, The method includes: Obtain a first feature of the first sample media resource, wherein the first feature is a multimodal feature obtained by identifying and processing the content theme of the first sample media resource; Construct positive and negative samples of the first feature. The positive sample includes a second feature, which is obtained by data augmentation processing of the first feature. The negative sample includes a third feature, which is a multimodal feature obtained by identifying the content theme of a second sample media resource. The second sample media resource is any media resource different from the first sample media resource. The vector representation model is invoked to obtain the first representation vector corresponding to the first feature, the second representation vector corresponding to the positive sample, and the third representation vector corresponding to the negative sample; Multimodal comparison learning is performed on the first representation vector and the second representation vector to obtain a first similarity between the first representation vector and the second representation vector; and multimodal comparison learning is performed on the first representation vector and the third representation vector to obtain a second similarity between the first representation vector and the second representation vector. The vector representation model is trained according to the rule of increasing the first similarity and decreasing the second similarity; In the scenario of data analysis of the first media resource, the trained vector representation model is used to generate a representation vector representing the content theme of the first media resource, so as to recommend a set of media resources similar to the content theme of the first media resource based on the representation vector of the first media resource.

11. The method as described in claim 10, characterized in that, The multimodal approach includes a first modality and a second modality; the acquisition of the first feature of the first sample media resource includes: The content theme of the first sample media resource is subjected to first modality recognition processing to obtain first modality features. The first modality features are used to characterize the content theme of the first sample media resource from the dimension of the first modality. The content theme of the first sample media resource is subjected to second modality recognition processing to obtain second modality features. The second modality features are used to characterize the content theme of the first sample media resource from the dimension of the second modality. The first modal feature and the second modal feature are spliced ​​together to form the first feature of the first sample media resource.

12. The method as described in claim 11, characterized in that, The first sample media resource includes text, audio, and images; the first modality is a text modality, and the second modality is an image modality; The first modality recognition process, which involves identifying the content theme of the first sample media resource to obtain first modality features, includes: The text in the first sample media resource is subjected to text recognition processing to obtain the first text information; The audio in the first sample media resource is subjected to audio recognition processing to obtain the second text information; The images in the first sample media resource are subjected to image-to-text conversion processing to obtain third text information; Entity recognition is performed on the first sample media resource to obtain entity information; The first modality feature is obtained by extracting at least one piece of information from the first text information, the second text information, the third text information, and the entity information.

13. The method as described in claim 11, characterized in that, The first sample media resource includes images; the first modality is a text modality, and the second modality is an image modality; The second modality recognition process for the content theme of the first sample media resource, to obtain second modality features, includes: The images in the first sample media resource are subjected to image recognition processing to obtain the second modal features.

14. The method as described in claim 1, characterized in that, The multimodal contrastive learning is performed by calling a multimodal model; the first feature includes image features and text features; The data augmentation process includes at least one of the following: performing synonym replacement on the text features, randomly repeating the text features, randomly cropping the text features, swapping the positions of the character features included in the text features, flipping the image features, translating the image features, and removing noise from the image features.

15. A media resource processing device, characterized in that, include: The display unit is used to display the data analysis interface in response to data analysis operations on the first media resource; The processing unit is configured to output a data analysis report of the first media resource in the data analysis interface. The data analysis report includes recommendation prompts, which are used to suggest a set of media resources that match the content theme of the first media resource. The display unit is also configured to display media resources in the media resource set in response to a trigger operation on the recommended prompt information.

16. A media resource processing device, characterized in that, include: The acquisition unit is used to acquire a first feature of the first sample media resource, wherein the first feature is a multimodal feature obtained by identifying and processing the content theme of the first sample media resource. The processing unit is configured to construct positive and negative samples of the first feature, wherein the positive sample includes a second feature, which is obtained by performing data augmentation processing on the first feature; and the negative sample includes a third feature, which is a multimodal feature obtained by performing content theme recognition processing on the second sample media resource, wherein the second sample media resource is any media resource different from the first sample media resource. The processing unit is further configured to call a vector representation model to obtain the first representation vector corresponding to the first feature, the second representation vector corresponding to the positive sample, and the third representation vector corresponding to the negative sample, respectively; The processing unit is further configured to perform multimodal comparison learning on the first representation vector and the second representation vector to obtain a first similarity between the first representation vector and the second representation vector; and to perform multimodal comparison learning on the first representation vector and the third representation vector to obtain a second similarity between the first representation vector and the second representation vector. The processing unit is further configured to train the vector representation model according to the rule of increasing the first similarity and decreasing the second similarity; In the scenario of data analysis of the first media resource, the trained vector representation model is used to generate a representation vector representing the content theme of the first media resource, so as to recommend a set of media resources similar to the content theme of the first media resource based on the representation vector of the first media resource.

17. A computer device, characterized in that, include: A processor is used to execute computer programs; A computer-readable storage medium storing a computer program, which, when executed by the processor, performs the method for processing media resources as described in any one of claims 1-14.

18. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program, which, when executed by a processor, performs the method for processing media resources as described in any one of claims 1-14.

19. A computer program product, characterized in that, The computer program product includes a computer program that, when executed by a processor, implements the method for processing media resources as described in any one of claims 1-14.

Images