Material processing method and device, computer program product and electronic equipment

Abstract

The invention relates to the technical field of computers, in particular to a material processing method and device, a computer program product and electronic equipment. The material processing method comprises the steps of obtaining task description information, wherein the task description information comprises first description information corresponding to an intention attribute and second description information corresponding to a content attribute; acquiring published materials through a real-time task flow, and performing content understanding on the published materials by using a multi-modal large model based on task description information to obtain an intention recognition result and a content evaluation result; wherein the published material comprises material contents of at least two modes; the multi-modal large model is obtained based on multi-modal sample data training; and carrying out associated storage on the intention recognition result and the content evaluation result and the published material. According to the invention, the processing accuracy of the multi-modal material can be improved, and the real-time performance of material processing is realized.

Description

Technical Field

[0001] The present disclosure relates to the field of computer technology, and more specifically, to a material processing method, a material processing device, a computer program product, and an electronic device. Background Art

[0002] With the development of computer and Internet technology, delivering materials to various platforms based on the Internet is characterized by immediacy and wide coverage. For example, videos, audios, texts, pictures and combinations of various types of materials can be delivered to web pages and applications. How to understand and manage the materials plays a significant role in recommending materials of interest to users and improving the effectiveness of material delivery.

[0003] However, current methods for understanding and processing materials focus on single-modal data analysis and cannot effectively integrate multi-modal data analysis, which affects the accuracy of material understanding and the quality of material processing. In addition, reliance on offline processing results in a lack of real-time material processing and is unable to meet real-time processing requirements.

[0004] It should be noted that the information disclosed in the above background technology section is only used to enhance the understanding of the background of the present disclosure, and therefore may include information that does not constitute the prior art known to ordinary technicians in the field. Summary of the invention

[0005] The purpose of the present disclosure is to provide a material processing method and device, a computer program product and an electronic device, thereby improving the processing accuracy of multimodal materials at least to a certain extent and achieving real-time material processing.

[0006] Other features and advantages of the present disclosure will become apparent from the following detailed description, or may be learned in part by the practice of the present disclosure.

[0007] According to one aspect of the present disclosure, a material processing method is provided, including: obtaining task description information, the task description information including first description information corresponding to intent attributes and second description information corresponding to content attributes; obtaining published materials through real-time task flows, and based on the task description information, using a multimodal large model to understand the content of the published materials, to obtain intent recognition results and content evaluation results; wherein the published materials include material content in at least two modalities; the multimodal large model is trained based on multimodal sample data; and the intent recognition results and content evaluation results are associated with the published materials and stored.

[0008] In an exemplary embodiment of the present disclosure, obtaining task description information includes: obtaining original input information of a multimodal large model; encapsulating first description information and second description information into the original input information to construct the task description information.

[0009] In an exemplary embodiment of the present disclosure, published materials are obtained through a real-time task flow, including: parsing the published materials to obtain material content of different modalities; if the material content includes material in video modality, splitting the material in video modality into multiple video frames, and extracting video frame features; based on the video frame features, calculating the inter-frame differences between adjacent video frames, and obtaining candidate key frames from the multiple video frames according to the inter-frame differences; clustering the candidate key frames to obtain multiple clustering clusters, and obtaining the central frame of each clustering cluster as a key frame to serve as the input of the multimodal large model.

[0010] In an exemplary embodiment of the present disclosure, the published material is obtained through the real-time task flow, including: responding to the material publishing operation of the subscription object, obtaining the published material through the real-time task flow, and calling the multimodal big model through the model interface, and sending the task description information to the multimodal big model.

[0011] In an exemplary embodiment of the present disclosure, there are multiple multimodal big models, and different multimodal big models have their own corresponding multimodal training tasks; based on the task description information, the multimodal big models are used to understand the content of the published materials to obtain intent recognition results and content evaluation results, including: inputting the published materials into each multimodal big model respectively to obtain the content understanding results corresponding to each multimodal big model, each content understanding result includes a reference intent result and a reference content result; fusing and calculating the reference intent results corresponding to each multimodal big model to obtain an intent recognition result; fusing and calculating the reference content results corresponding to each multimodal big model to obtain a content evaluation result.

[0012] In an exemplary embodiment of the present disclosure, the content evaluation results include content category results and content quality evaluation results; the intent recognition results and the content evaluation results are associated with the published material and stored, including: labeling the published material according to the intent recognition results and the content category results to obtain first label information corresponding to the intent attribute and second label information corresponding to the content attribute; the first label information, the second label information and the content quality evaluation results are associated with the published material and stored.

[0013] In an exemplary embodiment of the present disclosure, the method also includes: obtaining user interaction behavior data of published materials corresponding to different tag information; constructing a user interest model based on the different tag information and the corresponding user interaction behavior data, so as to perform material recommendation and / or material delivery based on the user interest model.

[0014] In an exemplary embodiment of the present disclosure, the method also includes: if there is a latest published material containing an image, determining content reward information of the latest published material according to a content quality assessment result corresponding to the latest published material; determining delivery reward information of the latest published material according to delivery indicator characteristics corresponding to the latest published material; and recommending material for the latest published material based on the content reward information and the delivery reward information.

[0015] In an exemplary embodiment of the present disclosure, the method also includes: based on the timing time of the scheduled task, obtaining the target published material that has not been associated and stored within the timing time; calling the offline interface of the multimodal large model, and performing content understanding processing on the target published material based on the task description information through the multimodal large model to obtain the target intent recognition result and the target content evaluation result; and associating and storing the target intent recognition result and the target content evaluation result with the target published material.

[0016] According to one aspect of the present disclosure, there is provided a material processing device, comprising: an information acquisition module, used to acquire task description information, the task description information comprising first description information corresponding to intent attributes and second description information corresponding to content attributes; a material processing module, used to acquire published materials through a real-time task flow, and based on the task description information, use a multimodal large model to understand the content of the published materials, to obtain intent recognition results and content evaluation results; wherein the published materials include material content in at least two modalities; the multimodal large model is obtained by training based on multimodal sample data; and an associated storage module, used to associate the intent recognition results and content evaluation results with the published materials for storage.

[0017] According to one aspect of the present disclosure, a computer program product is provided, including a computer program, and when the computer program is executed by a processor, any one of the above methods is implemented.

[0018] According to one aspect of the present disclosure, an electronic device is provided, comprising: a processor; and a memory for storing executable instructions of the processor; wherein the processor is configured to perform any one of the above methods by executing the executable instructions.

[0019] The material processing method in the exemplary embodiment of the present disclosure obtains task description information, which includes first description information corresponding to the intent attribute and second description information corresponding to the content attribute, obtains the published material through the real-time task flow, and based on the task description information, uses the multimodal large model to understand the content of the published material, and obtains the intent recognition result and the content evaluation result; wherein, the published material includes material content of at least two modes; the multimodal large model is obtained based on the training of multimodal sample data; finally, the intent recognition result and the content evaluation result are associated and stored with the published material. On the one hand, the published material can be obtained and the content can be understood through the real-time task flow, so as to realize the real-time processing of the published material and meet the real-time requirements for the material processing results. On the other hand, based on the task description information, the multimodal large model can be used to perform content understanding processing, thereby realizing the overall understanding and processing of the multimodal data, and the obtained content understanding result is more in line with the overall multimodal understanding result, thereby improving the accuracy of material processing. In addition, the obtained intent recognition results and content evaluation results are associated with the published materials and stored. By acquiring and processing the materials in real time, the real-time nature of the real-time associated materials and material processing results is improved, providing real-time and reliable material support for subsequent material recommendations or management, thereby improving its processing efficiency and accuracy.

[0020] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] The above and other objects, features and advantages of the exemplary embodiments of the present disclosure will become readily understood by reading the following detailed description with reference to the accompanying drawings, in which several embodiments of the present disclosure are shown in an exemplary and non-limiting manner.

[0022] Figure 1 An application environment according to an exemplary embodiment of the present disclosure is shown.

[0023] Figure 2 A flow chart of a material processing method according to an exemplary embodiment of the present disclosure is shown.

[0024] Figure 3 A flowchart of an implementation method for acquiring task description information according to an exemplary embodiment of the present disclosure is shown.

[0025] Figure 4 A flowchart of an implementation method for acquiring published materials according to an exemplary embodiment of the present disclosure is shown.

[0026] Figure 5 A flowchart is shown for understanding the content of published materials using a multimodal large model based on task description information according to an exemplary embodiment of the present disclosure.

[0027] Figure 6 A flow chart showing a method for implementing data association storage according to an exemplary embodiment of the present disclosure is shown.

[0028] Figure 7 A flowchart of a method for processing newly released materials according to an exemplary embodiment of the present disclosure is shown.

[0029] Figure 8 A flow chart showing an offline processing method of a material according to an exemplary embodiment of the present disclosure is shown.

[0030] Fig. 9 A flow chart showing a schematic diagram of a complete process of material processing according to an exemplary embodiment of the present disclosure.

[0031] Fig.10 A schematic diagram showing the composition of a material processing device according to an exemplary embodiment of the present disclosure is shown.

[0032] Fig.11 A block diagram of an electronic device according to an exemplary embodiment of the present disclosure is shown.

[0033] In the drawings, the same or corresponding reference numerals represent the same or corresponding parts. DETAILED DESCRIPTION

[0034] The exemplary embodiments will now be described more fully with reference to the accompanying drawings. However, the exemplary embodiments can be implemented in a variety of forms and should not be construed as limited to the examples set forth herein; rather, these embodiments are provided so that the present disclosure will be more comprehensive and complete and the concepts of the exemplary embodiments will be fully conveyed to those skilled in the art. The same reference numerals in the figures represent the same or similar structures, and thus their detailed description will be omitted.

[0035] In addition, the described features, structures or characteristics may be combined in one or more embodiments in any suitable manner. In the following description, many specific details are provided to provide a full understanding of the embodiments of the present disclosure. However, those skilled in the art will appreciate that the technical solutions of the present disclosure may be practiced without one or more of the specific details, or other methods, components, devices, steps, etc. may be adopted. In other cases, known structures, methods, devices, implementations or operations are not shown or described in detail to avoid blurring the various aspects of the present disclosure.

[0036] The block diagrams shown in the accompanying drawings are merely functional entities and do not necessarily correspond to physically independent entities. That is, these functional entities may be implemented in software form, or these functional entities or parts of functional entities may be implemented in one or more software hardened modules, or these functional entities may be implemented in different networks and / or processor devices and / or microcontroller devices.

[0037] At present, understanding, evaluating or classifying the released materials can provide accurate data support for subsequent material review, personalized recommendations, etc. For example, users can publish videos, texts, pictures and combinations of various types of materials on web pages and applications. These published materials usually contain multimodal information. However, only focusing on a single modal data, it is impossible to understand the multimodal information of the material as a whole, which affects the accuracy of material understanding to a certain extent. In addition, if the material is understood and processed by offline processing, it cannot meet the real-time requirements.

[0038] Based on this, an exemplary embodiment of the present disclosure provides a material processing method, which obtains published materials through real-time task flows, and based on the obtained task description information, uses a multimodal large model to understand the content of the published materials, thereby associating the obtained intent recognition results and content evaluation results with the published materials and storing them, thereby improving the real-time quality of delivery processing.

[0039] It should be noted that the material processing method of the exemplary embodiment of the present disclosure can be applied to various material processing scenarios, such as processing multimodal materials published on social platforms or content platforms, and of course can also be applied to processing multimodal commentary materials published on multimedia audio and video platforms. The exemplary embodiment of the present disclosure does not specifically limit the application field of the material processing method.

[0040] The material processing method provided by the exemplary embodiment of the present disclosure can be applied to Figure 1 In the application environment shown, the terminal 101 communicates with the server 102 via a network. The data storage system can store data that the server 102 needs to process. The data storage system can be integrated on the server 102, or placed on the cloud or other network servers.

[0041] In an exemplary embodiment, the material processing method provided by the exemplary embodiment of the present disclosure may be executed by the server 102, and the corresponding material processing device is disposed in the server 102. Correspondingly, in this mode of execution by the server 102, the server 102 may start to execute the steps in the technical solution of the exemplary embodiment of the present disclosure in response to a trigger command, wherein the trigger command may be sent by a terminal used by a user, or may be triggered locally by the server in response to some automation events.

[0042] The server 102 may be an independent physical server, a server cluster or a distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms. The server 102 may perform background tasks.

[0043] Furthermore, in another exemplary embodiment, the terminal 101 may also have similar functions to the server 102 , so as to execute the material processing method provided by the exemplary embodiment of the present disclosure.

[0044] The terminal 101 may be a smart phone, a tablet computer, a laptop computer, a desktop computer, a smart speaker, a smart watch, an IoT device, and a portable wearable device. The IoT device may be a smart speaker, a smart TV, a smart air conditioner, and a smart vehicle-mounted device, etc. The portable wearable device may be a smart watch, a smart bracelet, a head-mounted device, etc. The terminal 101 may also be referred to as a mobile terminal, a terminal device, a mobile device, etc. The exemplary embodiments of the present disclosure do not limit the type of the terminal 101.

[0045] In addition, the technical solution of the exemplary embodiment of the present disclosure can also be executed collaboratively by the terminal 101 and the server 102. In this manner of collaborative execution by the terminal 101 and the server 102, some steps in the technical solution provided by the exemplary embodiment of the present disclosure are executed by the terminal 101, while other steps are executed by the server 102. It should be noted that in this manner of collaborative execution by the terminal 101 and the server 102, the steps respectively executed by the terminal 101 and the server 102 can be dynamically adjusted according to actual conditions, and there is no special limitation on this.

[0046] The terminal 101 and the server 102 may be connected directly or indirectly via wireless communication, and the exemplary embodiments of the present disclosure do not impose any special limitation thereto.

[0047] It should be noted that the information involved in the embodiments of the present disclosure (including but not limited to: information input by the user, for example, information input by the user into the input box), data (including but not limited to data used for analysis, stored data, displayed data, for example, published materials, user interaction behavior data, first description information, second description information, etc.) are all authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data need to comply with relevant laws, regulations and standards.

[0048] In an exemplary embodiment of the present disclosure, a material processing method is provided, referring to Figure 2FIG. 1 is a flow chart of a material processing method according to an exemplary embodiment of the present disclosure. The material processing method includes steps S210 to S230, which are specifically as follows:

[0049] In step S210, task description information is obtained, where the task description information includes first description information corresponding to the intent attribute and second description information corresponding to the content attribute.

[0050] In an exemplary embodiment of the present disclosure, the task description information is an instruction used to guide the multimodal model to generate text or perform a specific task. The task description information can be any form of information, such as a paragraph of text, a question, or a set of instructions, depending on the task requirements. The task description information can clearly and specifically guide the multimodal model, helping the multimodal model to understand the task intent and generate corresponding outputs. In an exemplary embodiment of the present disclosure, the multimodal model is used to understand the published material and output the understood intent recognition results and content evaluation results.

[0051] Among them, the first description information corresponding to the intent attribute can be information that guides the multimodal model to identify the intent of the published material. For example, it can be a set of texts that evaluate the intent of publishing the material. For example, Table 1 schematically shows the first description information. As shown in Table 1, it can be understood that the first description information is used to guide the multimodal model to perform semantic understanding, thereby mining the intention of the user to publish the material.

[0052] Table 1

[0053]

[0054] Similarly, the second description information corresponding to the content attribute may be information that guides the multimodal model to identify and evaluate the content of the published material. For example, it may be a set of description information for evaluating the content category of the published material, and / or it may be a set of description information for evaluating the quality of the material. Exemplarily, a second description information corresponding to the content attribute may be "first-level classification: music; second-level classification: music scene; notes: pictures and videos are of music scenes, concerts, music festivals and other performances", and another second description information corresponding to the content attribute may be "first-level classification: people; second-level classification: showing yourself; notes: pictures highlighting the content of people appearing abroad", etc. Therefore, it can be understood that the second description information is used to guide the multimodal model to objectively understand the content of the material, thereby classifying the content of the material published by the user.

[0055] Furthermore, in addition to content classification, the understanding of material content can also be based on content quality assessment. Therefore, the second description information can also be used to guide the multimodal model to assess content quality, that is, the second description information can be description information for assessing material quality.

[0056] Exemplarily, the second description information may include at least three levels (good, medium, and poor), and has corresponding description information for each level. For example, for materials whose content is classified as the "music scene" category, the level to which the material belongs can also be evaluated based on the object's appearance, picture clarity, lens stability, and picture atmosphere. For example, the second description information corresponding to the "music scene" category can also include "Excellent: there are characters - the appearance is above average; there are no characters - the picture quality is clear, the shooting angle is moderate, the lens is stable, and the atmosphere is good); Poor: the distance is too far, the picture quality is poor, and the performance content cannot be seen clearly; Good: the picture spectrum is clear and the composition is ordinary." In other words, the content quality of the material can also be evaluated based on the second description information.

[0057] It should be noted that the first description information and the second description information in the above examples are only examples. The exemplary embodiments of the present disclosure can adjust the first description information and the second description information according to the actual application scenarios to make them richer or even contain more fine-grained descriptions, and there is no special limitation on this.

[0058] In step S220, the published material is obtained through the real-time task flow, and based on the task description information, the multimodal big model is used to understand the content of the published material to obtain the intent recognition result and the content evaluation result; wherein, the published material includes material content of at least two modalities; the multimodal big model is trained based on the multimodal sample data.

[0059] In an exemplary embodiment of the present disclosure, a real-time task flow is a task flow based on a real-time stream processing and batch processing framework, and is used to extract a user's published material in real time. For example, the real-time task flow is implemented based on a Flink processing framework, and there is no limitation to this. Among them, the published material can also be obtained through the real-time task flow based on information subscription.

[0060] Published materials refer to materials that have been published on the target platform. The target platform can be a platform that can deliver materials, such as a social media platform. Once the user publishes the material to the target platform, the published material can be obtained through the real-time task flow and processed online in real time. Among them, multimodality refers to the use of two or more different types of information sources, such as text, images, videos, tactile feedback, etc., in the process of information processing, communication or perception. Each information source is called a "modality". Content understanding refers to the process of in-depth interpretation of any form of content, such as text, pictures, audio, video, etc. This process can reveal the meaning, emotions, themes and other implicit information behind the content.

[0061] Among them, the published material includes material content of at least two modes, such as materials obtained by any combination of text, pictures or videos. Accordingly, in order to understand the published material from an overall multimodal perspective, the multimodal sample data can be used in advance to train the training model, which refers to a model that combines multimodal information such as text, images, and videos for training. The model parameters are adjusted by setting the loss function to obtain a multimodal large model. It can also be obtained based on the pre-trained model and fine-tuned using multimodal sample data. The exemplary embodiments of the present disclosure do not limit the training process of the multimodal large model.

[0062] The intent recognition result obtained by using a multimodal large model to understand the content of the published material refers to the real intention of the user to publish the published material based on the analysis and understanding of the material. For example, sharing the seedlings. The content evaluation result includes the objective understanding of the content of the published material (that is, the content category result, such as music scene) and the content quality evaluation (such as excellent).

[0063] In step S230, the intention recognition result and the content evaluation result are associated with the published material and stored.

[0064] In an exemplary embodiment of the present disclosure, after obtaining the intent recognition results and content evaluation results, the intent recognition results, content evaluation results and corresponding published materials can be directly associated and stored. For example, these data can be written to the storage online, so that they can be directly used as supporting data for other tasks.

[0065] The material processing method in the exemplary embodiment of the present disclosure can, on the one hand, acquire the published material and perform content understanding through the real-time task flow, realize real-time processing of the published material, and meet the real-time requirements for the material processing results. On the other hand, based on the task description information, a multimodal large model can be used to perform content understanding processing, thereby realizing the overall understanding and processing of multimodal data, and the obtained content understanding results are more in line with the overall multimodal understanding results, thereby improving the accuracy of material processing. In addition, the obtained intent recognition results and content evaluation results are associated with the published materials and stored. By acquiring and processing the materials in real time, the real-time nature of the real-time association of the materials and the material processing results is improved, providing real-time and reliable material support for subsequent material recommendation or management, and improving its processing efficiency and accuracy.

[0066] In an exemplary embodiment, a method for obtaining task description information is provided. Figure 3 As shown, obtaining task description information may include:

[0067] Step S310: Obtain original input information of the multimodal large model.

[0068] The original input information of the multimodal large model refers to the basic content such as data format information used to guide the multimodal large model to understand the content and input and output information. For example, the Prompt of the multimodal large model can include basic task objectives (for example, please output the intent recognition results and content evaluation results of the material), but the basic task objectives lack details on the material processing, such as how to divide intent, how to distinguish different content categories, etc. In addition, the original input information also includes the specific format of outputting intent recognition results and content evaluation results, such as "intent category is: Category A; content category is: Category a, content quality is good". Of course, the original input information is related to the task objectives actually processed, so they are not listed one by one.

[0069] Step S320: Encapsulate the first description information and the second description information into the original input information to construct task description information.

[0070] As described above, in order to enable the multimodal large model to output more detailed material processing results, the first description information and the second description information can be encapsulated into the original input information to construct the task description information. For example, the first description information and the second description information can be inserted into the key nodes of the original input information, such as inserting the first description information and the second description information into the "xxx information" position in "Please process the material resources according to xxx information", or a hook function can be set in the original input information, and the additional first / second description information can be called through the hook function to construct the task description information.

[0071] By encapsulating the first description information and the second description information into the original input information to construct the task description information, the required description can be added to the original input information of the multimodal large model according to the task scenario, adapting to different task scenarios and improving the efficiency of generating task description information and scenario adaptability.

[0072] In an exemplary embodiment, a method for obtaining published materials is provided. Figure 4 As shown, obtaining published materials through a real-time task flow may include:

[0073] Step S410: parse the published material to obtain material contents of different modalities.

[0074] The material contents of different modalities include videos, texts, pictures, etc., wherein the videos are composed of a series of continuous video frames (image frames), for example, usually 24 to 60 frames per second.

[0075] Step S420: If the material content includes material in video mode, the material in video mode is split into multiple video frames, and video frame features are extracted.

[0076] Currently, it is still impossible to directly understand the material in the video modality. The exemplary embodiment of the present disclosure processes the material in the video modality by extracting key frames, that is, converting the video modality data into image modality data for processing.

[0077] Among them, the most representative frames are selected from the video modality material, which can reflect the main content and changes of the video material. The basic information of the video modality material such as frame rate (Frame Per Second, FPS), resolution, total number of frames, etc. can be obtained, and then the video material is decomposed into separate video frames based on the basic information, and the specific decomposition method is not limited.

[0078] The video frame features may be extracted using a SIFT (Scale-invariant feature transform) feature extraction model or a Unet model, without limitation.

[0079] Step S430: Based on the video frame features, the inter-frame differences between adjacent video frames are calculated, and candidate key frames are obtained from the multiple video frames according to the inter-frame differences.

[0080] The inter-frame difference between adjacent video frames may be calculated using feature distance, such as Euclidean distance, cosine similarity, etc., without limitation.

[0081] Among them, the inter-frame difference threshold can be set in advance, so as to screen out frames with larger differences as candidate key frames according to the calculated inter-frame differences. The inter-frame difference threshold can be set according to the required video frame data. When a large number of video frames are required, the inter-frame difference threshold can be reduced. Correspondingly, when a large number of video frames are required, the inter-frame difference threshold can be increased, and there is no restriction on its specific data.

[0082] Step S440: clustering the candidate key frames to obtain a plurality of clusters, and obtaining the central frame of each cluster as a key frame to serve as the input of the multimodal large model.

[0083] A k-means clustering algorithm, mean shift clustering, etc. may be used to cluster the candidate key frames to obtain a plurality of clusters.

[0084] Among them, the central frame of each cluster can be selected as the key frame, and the central frame can reflect the important information of the candidate key frame in a cluster to a certain extent. Finally, the obtained key frame is used for subsequent multimodal large model processing.

[0085] By converting video material into image material by frame extraction, it helps to reduce the amount of data and improve processing efficiency while retaining important information of the video material.

[0086] In an exemplary embodiment, a method for obtaining published materials is provided, that is, obtaining published materials through a real-time task flow may include:

[0087] In response to the material publishing operation of the subscription object, the published material is obtained through the real-time task flow, and the multimodal large model is called through the model interface to send the task description information to the multimodal large model.

[0088] The subscription object refers to an application or platform that needs to collect published materials for processing. A multimodal large model that supports multimodal material processing can be subscribed to through a model interface. The NDC (Notification Delivery Channel) subscription method can be used to read the materials published by users in the subscription object in real time. Of course, other subscription methods are also possible. The exemplary embodiment of the present disclosure is described by taking the NDC subscription method as an example.

[0089] It should be understood that through the subscription model, if the subscribed object has material publishing behavior, the material will be obtained in real time and the multimodal big model will be called through the interface, and the task description information will be passed to the multimodal big model for processing. This can not only improve the real-time performance of material processing, but also call different multimodal big models for processing according to the task objectives, thereby improving the flexibility of material processing.

[0090] In an exemplary embodiment, there are multiple multimodal large models, and different multimodal large models have their own corresponding multimodal training tasks; for example, some multimodal large models are more inclined to process multimodal data containing text and images, while some multimodal large models are more inclined to process multimodal data containing text and audio, and so on. Alternatively, some multimodal large models are better at processing intent recognition, but their content evaluation capabilities are slightly weaker. On the contrary, some multimodal large models are better at content evaluation. Based on this, intent recognition results and content evaluation results can be obtained by utilizing multi-model fusion.

[0091] Specifically, Figure 5 As shown, based on the task description information, a multimodal large model is used to understand the content of the published material, and the intent recognition results and content evaluation results obtained may include:

[0092] First, the published materials are input into each multimodal large model respectively to obtain the content understanding results corresponding to each multimodal large model. Each content understanding result includes a reference intent result and a reference content result. Then, the reference intent results corresponding to each multimodal large model are fused and calculated to obtain the intent recognition result. Finally, the reference content results corresponding to each multimodal large model are fused and calculated to obtain the content evaluation result.

[0093] Among them, the interface of each multimodal large model can be called separately, and the task description information can be passed to each multimodal large model separately, so that each multimodal large model processes the published material based on the task description information, and each multimodal large model outputs the content understanding result of the published material.

[0094] In an optional embodiment, the method of performing fusion calculation on the content understanding results may be to determine the intent recognition results from the reference intent results through voting.

[0095] The voting method can vote on the prediction results (reference intent results or reference content results) of multiple multimodal models, that is, the minority obeys the majority. Specifically, the hard voting method can perform a simple majority vote on the prediction results of multiple multimodal models, that is, select the reference intent result (or reference content result) with the most occurrences as the final prediction result (reference intent result or reference content result). Soft voting is to perform weighted processing on the prediction results of each multimodal model and select the category with the largest sum of probabilities as the final prediction result.

[0096] By determining the intent recognition result from multiple reference intent results output by multiple multimodal models through a voting method, and determining the content evaluation result from multiple reference content results output by multiple multimodal models, the generalization ability, stability and accuracy of the multimodal model can be improved, thereby improving the accuracy of determining the intent recognition result and the content evaluation result.

[0097] In an exemplary embodiment, a method for implementing data association storage is provided. The content evaluation result includes a content category result and a content quality evaluation result, such as Figure 6 As shown, associating the intent recognition result and the content evaluation result with the published material and storing them may include:

[0098] Step S610: labeling the published material according to the intent recognition result and the content classification result to obtain first label information corresponding to the intent attribute and second label information corresponding to the content attribute.

[0099] Labeling refers to setting category labels for published materials based on intent recognition results and content category results, for example, labeling published materials with "Sharing Grass-Life" and "Life". "Sharing Grass-Life" is the first label information corresponding to the intent attribute, and "Life" is the second label information corresponding to the content attribute. In other words, different attributes are labeled for intent recognition and content categories of published materials.

[0100] Step S620: The first tag information, the second tag information and the content quality assessment result are associated with the published material and stored.

[0101] After the labeling of the published material is completed, the first label information, the second label information and the content quality assessment result may be associated with the published material and stored.

[0102] It should be understood that the published material can also be labeled based on the content quality assessment result to obtain third label information, such as "excellent", so that the first label information, the second label information and the third label information are associated with the published material and stored.

[0103] By associating and storing the labeling results with the published materials, it is possible to provide material support with clear label information for subsequent material recommendation or material review, making material processing convenient and efficient.

[0104] In some optional embodiments, user interaction behavior data of published materials corresponding to different tag information may be obtained, and a user interest model may be constructed based on the different tag information and the corresponding user interaction behavior data to perform material recommendation and / or material delivery based on the user interest model.

[0105] Among them, user interaction behavior data includes browsing, liking, commenting, sharing, etc. of materials. A user interest model is constructed based on user interaction behavior data and tag information. The user interest model is a virtual representation of real users and can be used to capture users' relatively stable material needs over a period of time.

[0106] Specifically, based on the user interaction behavior data, it is possible to analyze and infer which categories of published materials are of interest to the user, and then build a user interest model based on this information. Based on this, it is possible to recommend materials of interest to the user or deliver materials of interest to the user based on the user interest model. The exemplary embodiment of the present disclosure is not limited to the way in which the user interest model is constructed, for example, it can be constructed based on the way of constructing a user portrait.

[0107] In some optional embodiments, since each published material has been labeled, the visibility and user engagement of certain categories of published materials can be improved by increasing exposure, etc. according to the task objectives. Correspondingly, the exposure of certain materials that do not meet the task objectives (such as categories with negative impacts) can be controlled, or the recommendation weight of published materials in this category can be reduced to reduce the impact.

[0108] By analyzing and integrating the user interaction behavior data of the labeled published materials, we can build a user interest model that reflects user material needs, provide model support for material recommendation / delivery, and improve the efficiency and accuracy of material processing.

[0109] Based on the aforementioned exemplary embodiments, a method for processing newly released materials is also provided. Specifically, Figure 7 As shown, the processing of newly released materials may include:

[0110] Step S710: If there is a latest published material containing an image, content reward information of the latest published material is determined according to a content quality assessment result corresponding to the latest published material.

[0111] For the method of obtaining the content quality assessment results corresponding to the latest released material, please refer to the above-mentioned content of obtaining the content quality assessment results of the released material, which will not be repeated here.

[0112] Among them, content reward information refers to the reward for material content calculated based on the content quality assessment results, which can be used to predict the content performance effect of the latest released material. The larger the content reward information, the better the release effect of the latest released material. The correspondence between the content quality assessment results and the reward data can be set in advance, so as to obtain the content reward information of the latest released material according to the correspondence. For example, the correspondence between "excellent" and 90 points (or 90 to 100 points) can be set, and the content reward information can be obtained based on the correspondence, such as 90, or when the reward data is an interval, the endpoint data on one side of the interval or the middle data of the interval is taken as the content reward information, which can be flexibly set according to actual needs.

[0113] Step S720: Determine the delivery reward information of the latest released material according to the delivery indicator characteristics corresponding to the latest released material.

[0114] The delivery indicator characteristics refer to indicators that reflect the performance of the material, including exposure, playback volume, completion rate, bounce rate, interaction rate (such as the number of interactive behaviors such as the number of likes, comments, and forwarding rates), viewing time, number of member views, etc. Based on the delivery indicator characteristics, the user's emotional response and interactive behavior to the released material can be known. Among them, exposure refers to the number of times the released material is displayed, playback volume refers to the number of times the released material is clicked to play, completion rate refers to the proportion of users who watch the released material in its entirety, and bounce rate refers to the proportion of users who leave during the process of watching the material (such as watching a video). The exemplary embodiment of the present disclosure can collect the above-mentioned user feedback characteristics in real time by deeply integrating the data collection module with the interface of the target platform, and there is no limitation on this.

[0115] The delivery reward information refers to the delivery effect information of the latest released material. The delivery reward information can be calculated based on the delivery indicator characteristics and the preset weight information of each corresponding indicator, wherein the weight information is set according to the contribution degree of the delivery indicator characteristics, and the weight information of the delivery indicator characteristics with a high contribution degree is greater than the weight information of the delivery indicator characteristics with a small contribution degree. The range of the weight information is [0,1], and there is no restriction on the specific value of the weight information here.

[0116] Step S730: Based on the content reward information and the delivery reward information, the latest released material is recommended.

[0117] The content reward information and the delivery reward information may be fused and calculated, such as weighted summation or simple summation, so as to recommend the latest released materials according to the obtained fusion reward.

[0118] Based on this, when newly released materials are obtained, content reward information can be obtained according to the content quality evaluation results of the newly released materials. When exploring materials, due to factors such as the number of daily active users and community activity of the newly released materials, it is impossible to recommend materials based on the accumulated material interaction data (such as clicks, forwarding, etc.), resulting in a cold start, and the newly released materials cannot be effectively recommended. By combining content reward information and delivery reward information, the above-mentioned cold start problem can be avoided, and effective recommendations for the latest released materials can be achieved, and the processing efficiency of the latest released materials can be improved. In addition, the influence of the content evaluation results of the materials is introduced when recommending materials, and the judgment of the quality of the content does not rely entirely on user behavior data. Otherwise, the judgment of inferior content requires the accumulation of user exposure with a high interaction rate, which affects the accuracy of material recommendations and the user experience.

[0119] In an exemplary embodiment, a method for processing materials offline is provided. Specifically, Figure 8 As shown, the offline processing method may include:

[0120] Step S810: based on the timing time of the scheduled task, obtaining the target published material that has not been associated and stored within the timing time.

[0121] The timing time of the scheduled task can be half a day, one day or one week, etc. For example, at dawn every day, offline statistics can be collected on the target published materials that failed to pass real-time tagging (i.e., failed to achieve associated storage) the previous day.

[0122] Step S820: Call the offline interface of the multimodal large model, and use the multimodal large model to perform content understanding processing on the target published material based on the task description information to obtain the target intent recognition result and the target content evaluation result.

[0123] Step S830: The target intent recognition result and the target content evaluation result are associated with the target published material and stored.

[0124] It should be noted that the method of understanding the content of the target published material and storing it in association is the same as the above-mentioned method of processing the published material in real time, which will not be described in detail here.

[0125] Among them, the target intent recognition results and target content evaluation results can be associated with the target published materials and stored in an online storage system to improve the comprehensiveness and accuracy of the online material processing results.

[0126] In addition, the target published materials can also be packaged and processed using traditional machine learning models, which are trained based on sample data, so that the accuracy of the labeling results can be improved by combining the multimodal large model of the traditional machine learning model. Among them, when the current prediction accuracy of the multimodal large model is lower than the set threshold (such as 75%), offline labeling can be performed based on the traditional large machine learning model as a supplement to online labeling, which can improve the accuracy of the labeling results to a certain extent.

[0127] In addition, regular manual spot checks can be performed to manually mark the target published materials that have not achieved associated storage within the scheduled time to improve the comprehensiveness of the marking results.

[0128] Fig. 9 A complete flow diagram of material processing is shown, and a material processing method of an exemplary embodiment of the present disclosure is described by taking a real-time task flow based on a Flink framework as an example and an NDC subscription method as an example.

[0129] First, in response to a material publishing operation of a subscription object (a certain social platform), the published material is obtained based on the NDC subscription.

[0130] Secondly, the task description information is obtained, and the multimodal large model is called through the model interface to send the task description information to the multimodal large model.

[0131] Then, based on the task description information, a large multimodal model is used to understand the content of the published materials to obtain intent recognition results and content evaluation results.

[0132] Next, the published material is labeled according to the intent recognition result and the content classification result to obtain first label information corresponding to the intent attribute and second label information corresponding to the content attribute.

[0133] Finally, the first tag information, the second tag information and the content quality assessment result are associated with the published material and stored.

[0134] It should be noted that the detailed descriptions of each step have been recorded in the above exemplary embodiments and will not be repeated here.

[0135] The material processing method in the exemplary embodiment of the present disclosure can, on the one hand, acquire the published material and perform content understanding through the real-time task flow, realize real-time processing of the published material, and meet the real-time requirements for the material processing results. On the other hand, based on the task description information, a multimodal large model can be used to perform content understanding processing, thereby realizing the overall understanding and processing of multimodal data, and the obtained content understanding results are more in line with the overall multimodal understanding results, thereby improving the accuracy of material processing. In addition, the obtained intent recognition results and content evaluation results are associated with the published materials and stored. By acquiring and processing the materials in real time, the real-time nature of the real-time association of the materials and the material processing results is improved, providing real-time and reliable material support for subsequent material recommendation or management, and improving its processing efficiency and accuracy.

[0136] In an exemplary embodiment of the present disclosure, a material processing device is also provided. Fig.10 As shown, the material processing device 1000 may include an information acquisition module 1010, a material processing module 1020 and an associated storage module 1030. Specifically:

[0137] An information acquisition module 1010 is used to acquire task description information, wherein the task description information includes first description information corresponding to an intent attribute and second description information corresponding to a content attribute;

[0138] The material processing module 1020 is used to obtain the published material through the real-time task flow, and based on the task description information, use the multimodal large model to understand the content of the published material to obtain the intention recognition result and the content evaluation result; wherein the published material includes material content of at least two modes; the multimodal large model is obtained by training based on multimodal sample data;

[0139] The associated storage module 1030 is used to associate and store the intention recognition result and the content evaluation result with the published material.

[0140] In an exemplary embodiment of the present disclosure, the information acquisition module 1010 is configured to execute: obtaining original input information of the multimodal large model; encapsulating the first description information and the second description information into the original input information to construct task description information.

[0141] In an exemplary embodiment of the present disclosure, the material processing module 1020 is configured to perform: parsing the published material to obtain material content of different modalities; if the material content includes material in video modality, splitting the material in video modality into multiple video frames and extracting video frame features; based on the video frame features, calculating the inter-frame differences between adjacent video frames, and obtaining candidate key frames from multiple video frames based on the inter-frame differences; clustering the candidate key frames to obtain multiple clustering clusters, and obtaining the central frame of each clustering cluster as a key frame to serve as the input of the multimodal large model.

[0142] In an exemplary embodiment of the present disclosure, the material processing module 1020 is configured to perform: in response to a material publishing operation of a subscription object, obtain the published material through a real-time task flow, call a multimodal large model through a model interface, and send task description information to the multimodal large model.

[0143] In an exemplary embodiment of the present disclosure, there are multiple multimodal large models, and different multimodal large models have their own corresponding multimodal training tasks; the material processing module 1020 is configured to execute: inputting the published materials into each multimodal large model respectively, and obtaining the content understanding results corresponding to each multimodal large model, each content understanding result includes a reference intent result and a reference content result; fusing and calculating the reference intent results corresponding to each multimodal large model to obtain an intent recognition result; fusing and calculating the reference content results corresponding to each multimodal large model to obtain a content evaluation result.

[0144] In an exemplary embodiment of the present disclosure, the content evaluation results include content category results and content quality evaluation results; the associated storage module 1030 is configured to execute: labeling the published material according to the intent recognition results and the content category results to obtain first label information corresponding to the intent attribute and second label information corresponding to the content attribute; and associating the first label information, the second label information and the content quality evaluation results with the published material for storage.

[0145] In an exemplary embodiment of the present disclosure, the associated storage module 1030 is configured to execute: obtaining user interaction behavior data of published materials corresponding to different tag information; constructing a user interest model based on different tag information and corresponding user interaction behavior data, so as to perform material recommendation and / or material delivery based on the user interest model.

[0146] In an exemplary embodiment of the present disclosure, the associated storage module 1030 is configured to execute: if there is a latest published material containing an image, determine the content reward information of the latest published material according to the content quality evaluation result corresponding to the latest published material; determine the delivery reward information of the latest published material according to the delivery indicator characteristics corresponding to the latest published material; and recommend the latest published material based on the content reward information and the delivery reward information.

[0147] In an exemplary embodiment of the present disclosure, the material processing module 1020 is also configured to execute: based on the timing time of the scheduled task, obtain the target published material that has not been associated and stored within the timing time; call the offline interface of the multimodal large model, and perform content understanding processing on the target published material based on the task description information through the multimodal large model to obtain the target intent recognition result and the target content evaluation result; associate the target intent recognition result and the target content evaluation result with the target published material for storage.

[0148] Since the details of the various functional modules of the material processing device of the exemplary embodiment of the present disclosure have been described in the exemplary embodiment of the material processing method described above, they will not be repeated here.

[0149] It should be noted that although several modules or units of the material processing device are mentioned in the above detailed description, this division is not mandatory. In fact, according to the embodiments of the present disclosure, the features and functions of two or more modules or units described above can be embodied in one module or unit. Conversely, the features and functions of one module or unit described above can be further divided into multiple modules or units to be embodied.

[0150] The exemplary embodiments of the present disclosure further provide a computer program product, which includes a computer program, and when the computer program is executed by a processor, the material processing method described above is implemented.

[0151] In one embodiment, the computer program product may be a tangible product containing a computer program, such as a computer-readable storage medium storing a computer program. The readable storage medium may be a storage medium based on electrical, magnetic, optical, electromagnetic, infrared, or other signals, including but not limited to: random access memory (RAM), read-only memory (ROM), magnetic tape, floppy disk, flash memory (Flash), mechanical hard disk (HDD), solid-state drive (SSD), and the like. Exemplarily, the computer program product may be implemented as a non-volatile storage medium storing a computer program, such as a read-only memory, a NAND flash memory, and the like.

[0152] In one embodiment, the computer program product may be an intangible product including a computer program. Exemplarily, the computer program product may be implemented as a virtual digital product, such as a digital file storing an executable file, an installation package, etc. of the computer program.

[0153] The code of the computer program can be written in one or more programming languages. Programming languages ​​such as C language, Java, C++, etc. The program code can be executed entirely on the user computing device, or partially on the user computing device, or as a separate software package, or partially on the user computing device and partially on a remote computing device, or entirely on a remote computing device or server. In the case of a remote computing device, the remote computing device can be connected to the user computing device through any type of network, such as a local area network (LAN), a wide area network (WAN), etc., or can be connected to an external computing device (e.g., an Internet connection provided by an operator).

[0154] Computer programs can be carried or transmitted through electrical, magnetic, optical, electromagnetic, infrared and other signals. Electronic devices can convert signals carrying computer programs into digital signals, thereby running computer programs. When a computer program runs on an electronic device, its code is used to enable the electronic device to execute (more specifically, the processor of the electronic device can execute) the method steps of various exemplary embodiments of the present disclosure, such as the material processing method described above, which includes the following steps: obtaining task description information, the task description information includes first description information corresponding to the intent attribute and second description information corresponding to the content attribute; obtaining published materials through real-time task flow, and based on the task description information, using a multimodal large model to understand the content of the published materials, and obtaining intent recognition results and content evaluation results; wherein the published materials include material content of at least two modes; the multimodal large model is trained based on multimodal sample data; the intent recognition results and content evaluation results are associated with the published materials and stored.

[0155] By executing the above method steps through a computer program, on the one hand, the published material can be obtained and the content can be understood through the real-time task flow, so as to realize the real-time processing of the published material and meet the real-time requirements for the material processing results. On the other hand, based on the task description information, a multimodal large model can be used to perform content understanding processing, thereby realizing the overall understanding and processing of multimodal data, and the obtained content understanding results are more in line with the overall multimodal understanding results, thereby improving the accuracy of material processing. In addition, the obtained intent recognition results and content evaluation results are associated with the published material and stored. By acquiring and processing the material in real time, the real-time nature of the real-time association of the material and the material content processing results is improved, providing real-time and reliable material support for subsequent material recommendation or management, and improving its processing efficiency and accuracy.

[0156] In an exemplary embodiment of the present disclosure, obtaining task description information includes: obtaining original input information of a multimodal large model; encapsulating first description information and second description information into the original input information to construct the task description information.

[0157] In an exemplary embodiment of the present disclosure, published materials are obtained through a real-time task flow, including: parsing the published materials to obtain material content of different modalities; if the material content includes material in video modality, splitting the material in video modality into multiple video frames, and extracting video frame features; based on the video frame features, calculating the inter-frame differences between adjacent video frames, and obtaining candidate key frames from the multiple video frames according to the inter-frame differences; clustering the candidate key frames to obtain multiple clustering clusters, and obtaining the central frame of each clustering cluster as a key frame to serve as the input of the multimodal large model.

[0158] In an exemplary embodiment of the present disclosure, the published material is obtained through the real-time task flow, including: responding to the material publishing operation of the subscription object, obtaining the published material through the real-time task flow, and calling the multimodal big model through the model interface, and sending the task description information to the multimodal big model.

[0159] In an exemplary embodiment of the present disclosure, there are multiple multimodal big models, and different multimodal big models have their own corresponding multimodal training tasks; based on the task description information, the multimodal big models are used to understand the content of the published materials to obtain intent recognition results and content evaluation results, including: inputting the published materials into each multimodal big model respectively to obtain the content understanding results corresponding to each multimodal big model, each content understanding result includes a reference intent result and a reference content result; fusing and calculating the reference intent results corresponding to each multimodal big model to obtain an intent recognition result; fusing and calculating the reference content results corresponding to each multimodal big model to obtain a content evaluation result.

[0160] In an exemplary embodiment of the present disclosure, the content evaluation results include content category results and content quality evaluation results; the intent recognition results and the content evaluation results are associated with the published material and stored, including: labeling the published material according to the intent recognition results and the content category results to obtain first label information corresponding to the intent attribute and second label information corresponding to the content attribute; the first label information, the second label information and the content quality evaluation results are associated with the published material and stored.

[0161] In an exemplary embodiment of the present disclosure, the method also includes: obtaining user interaction behavior data of published materials corresponding to different tag information; constructing a user interest model based on the different tag information and the corresponding user interaction behavior data, so as to perform material recommendation and / or material delivery based on the user interest model.

[0162] In an exemplary embodiment of the present disclosure, the method also includes: if there is a latest published material containing an image, determining content reward information of the latest published material according to a content quality assessment result corresponding to the latest published material; determining delivery reward information of the latest published material according to delivery indicator characteristics corresponding to the latest published material; and recommending material for the latest published material based on the content reward information and the delivery reward information.

[0163] In an exemplary embodiment of the present disclosure, the method also includes: based on the timing time of the scheduled task, obtaining the target published material that has not been associated and stored within the timing time; calling the offline interface of the multimodal large model, and performing content understanding processing on the target published material based on the task description information through the multimodal large model to obtain the target intent recognition result and the target content evaluation result; and associating and storing the target intent recognition result and the target content evaluation result with the target published material.

[0164] In addition, in an exemplary embodiment of the present disclosure, an electronic device capable of implementing the above method is also provided. It will be appreciated by those skilled in the art that various aspects of the present disclosure may be implemented as a system, method, or program product. Therefore, various aspects of the present disclosure may be specifically implemented in the following forms, namely: a complete hardware embodiment, a complete software embodiment (including firmware, microcode, etc.), or an embodiment combining hardware and software aspects, which may be collectively referred to herein as a "circuit", "module", or "system".

[0165] Refer to the following Fig.11 1100 according to this embodiment of the present disclosure is described. Fig.11 The electronic device 1100 shown is merely an example and should not bring any limitation to the functions and scope of use of the embodiments of the present disclosure.

[0166] like Fig.11 As shown, the electronic device 1100 is in the form of a general computing device. The components of the electronic device 1100 may include, but are not limited to: the at least one processing unit 1110, the at least one storage unit 1120, a bus 1130 connecting different system components (including the storage unit 1120 and the processing unit 1110), and a display unit 1140.

[0167] The storage unit stores a program code, and the program code can be executed by the processing unit 1110, so that the processing unit 1110 executes the steps described in the above "Exemplary Method" section of this specification according to various exemplary embodiments of the present disclosure. For example, the processing unit 1110 can execute the following steps: obtaining task description information, the task description information includes first description information corresponding to the intent attribute and second description information corresponding to the content attribute; obtaining published materials through real-time task flow, and based on the task description information, using a multimodal large model to understand the content of the published materials, and obtaining intent recognition results and content evaluation results; wherein the published materials include material content of at least two modes; the multimodal large model is trained based on multimodal sample data; the intent recognition results and content evaluation results are associated and stored with the published materials.

[0168] In an exemplary embodiment of the present disclosure, obtaining task description information includes: obtaining original input information of a multimodal large model; encapsulating first description information and second description information into the original input information to construct the task description information.

[0169] In an exemplary embodiment of the present disclosure, published materials are obtained through a real-time task flow, including: parsing the published materials to obtain material content of different modalities; if the material content includes material in video modality, splitting the material in video modality into multiple video frames, and extracting video frame features; based on the video frame features, calculating the inter-frame differences between adjacent video frames, and obtaining candidate key frames from the multiple video frames according to the inter-frame differences; clustering the candidate key frames to obtain multiple clustering clusters, and obtaining the central frame of each clustering cluster as a key frame to serve as the input of the multimodal large model.

[0170] In an exemplary embodiment of the present disclosure, the published material is obtained through the real-time task flow, including: responding to the material publishing operation of the subscription object, obtaining the published material through the real-time task flow, and calling the multimodal big model through the model interface, and sending the task description information to the multimodal big model.

[0171] In an exemplary embodiment of the present disclosure, there are multiple multimodal big models, and different multimodal big models have their own corresponding multimodal training tasks; based on the task description information, the multimodal big models are used to understand the content of the published materials to obtain intent recognition results and content evaluation results, including: inputting the published materials into each multimodal big model respectively to obtain the content understanding results corresponding to each multimodal big model, each content understanding result includes a reference intent result and a reference content result; fusing and calculating the reference intent results corresponding to each multimodal big model to obtain an intent recognition result; fusing and calculating the reference content results corresponding to each multimodal big model to obtain a content evaluation result.

[0172] In an exemplary embodiment of the present disclosure, the content evaluation results include content category results and content quality evaluation results; the intent recognition results and the content evaluation results are associated with the published material and stored, including: labeling the published material according to the intent recognition results and the content category results to obtain first label information corresponding to the intent attribute and second label information corresponding to the content attribute; the first label information, the second label information and the content quality evaluation results are associated with the published material and stored.

[0173] In an exemplary embodiment of the present disclosure, the method also includes: obtaining user interaction behavior data of published materials corresponding to different tag information; constructing a user interest model based on the different tag information and the corresponding user interaction behavior data, so as to perform material recommendation and / or material delivery based on the user interest model.

[0174] In an exemplary embodiment of the present disclosure, the method also includes: if there is a latest published material containing an image, determining content reward information of the latest published material according to a content quality assessment result corresponding to the latest published material; determining delivery reward information of the latest published material according to delivery indicator characteristics corresponding to the latest published material; and recommending material for the latest published material based on the content reward information and the delivery reward information.

[0175] In an exemplary embodiment of the present disclosure, the method also includes: based on the timing time of the scheduled task, obtaining the target published material that has not been associated and stored within the timing time; calling the offline interface of the multimodal large model, and performing content understanding processing on the target published material based on the task description information through the multimodal large model to obtain the target intent recognition result and the target content evaluation result; and associating and storing the target intent recognition result and the target content evaluation result with the target published material.

[0176] The storage unit 1120 may include a readable medium in the form of a volatile storage unit, such as a random access memory unit (RAM) 1121 and / or a cache memory unit 1122 , and may further include a read-only memory unit (ROM) 1123 .

[0177] The storage unit 1120 may also include a program / utility 1124 having a set (at least one) of program modules 1125, such program modules 1125 including but not limited to: an operating system, one or more application programs, other program modules, and program data, each of which or some combination may include an implementation of a network environment.

[0178] Bus 1130 may represent one or more of several types of bus structures, including a memory unit bus or memory unit controller, a peripheral bus, an accelerated graphics port, a processing unit, or a local bus using any of a variety of bus architectures.

[0179] The electronic device 1100 may also communicate with one or more external devices 1200 (e.g., keyboards, pointing devices, Bluetooth devices, etc.), may also communicate with one or more devices that enable a user to interact with the electronic device 1100, and / or communicate with any device that enables the electronic device 1100 to communicate with one or more other computing devices (e.g., routers, modems, etc.). Such communication may be performed via an input / output (I / O) interface 1150. Furthermore, the electronic device 1100 may also communicate with one or more networks (e.g., local area networks (LANs), wide area networks (WANs), and / or public networks, such as the Internet) via a network adapter 1160. As shown, the network adapter 1160 communicates with other modules of the electronic device 1100 via a bus 1130. It should be understood that, although not shown in the figure, other hardware and / or software modules may be used in conjunction with the electronic device 1100, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0180] Through the description of the above embodiments, it is easy for those skilled in the art to understand that the example embodiments described here can be implemented by software, or by software combined with necessary hardware. Therefore, the technical solution according to the embodiment of the present disclosure can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a USB flash drive, a mobile hard disk, etc.) or on a network, and includes several instructions to enable a computing device (which can be a personal computer, a server, a terminal device, or a network device, etc.) to execute the method according to the embodiment of the present disclosure.

[0181] In addition, the above-mentioned figures are only schematic illustrations of the processes included in the method according to the exemplary embodiments of the present disclosure, and are not intended to be limiting. It is easy to understand that the processes shown in the above-mentioned figures do not indicate or limit the time sequence of these processes. In addition, it is also easy to understand that these processes can be performed synchronously or asynchronously, for example, in multiple modules.

[0182] Those skilled in the art will readily appreciate other embodiments of the present disclosure after considering the specification and practicing the invention disclosed herein. The present disclosure is intended to cover any variations, uses, or adaptations of the present disclosure that follow the general principles of the present disclosure and include common knowledge or customary technical means in the art that are not disclosed in the present disclosure. The description and examples are to be considered exemplary only, and the true scope and spirit of the present disclosure are indicated by the claims.

Claims

1. A material processing method, characterized in that: include: Acquire task description information, wherein the task description information includes first description information corresponding to the intent attribute and second description information corresponding to the content attribute; Acquire published materials through real-time task flow, and use a multimodal large model to understand the content of the published materials based on the task description information to obtain intent recognition results and content evaluation results; wherein the published materials include material content in at least two modalities; and the multimodal large model is trained based on multimodal sample data; The intention recognition result and the content evaluation result are associated with the published material and stored.

2. The method according to claim 1, characterized in that The obtaining of task description information includes: Obtaining original input information of the multimodal large model; The first description information and the second description information are encapsulated into the original input information to construct the task description information.

3. The method according to claim 1 or 2, characterized in that: The step of obtaining the published material through the real-time task flow includes: Parsing the published material to obtain material contents of different modes; If the material content includes material in a video mode, splitting the material in the video mode into multiple video frames, and extracting video frame features; Based on the video frame features, calculating the inter-frame differences between adjacent video frames, and acquiring candidate key frames from the multiple video frames according to the inter-frame differences; The candidate key frames are clustered to obtain a plurality of clusters, and the central frame of each cluster is obtained as a key frame to serve as an input of the multimodal large model.

4. The method according to claim 1, characterized in that The step of obtaining the published material through the real-time task flow includes: In response to the material publishing operation of the subscription object, the published material is obtained through the real-time task flow, and the multimodal large model is called through the model interface to send the task description information to the multimodal large model.

5. The method according to claim 4, characterized in that There are multiple multimodal large models, and different multimodal large models have their own corresponding multimodal training tasks; based on the task description information, the multimodal large model is used to understand the content of the published material to obtain the intent recognition result and the content evaluation result, including: Inputting the published materials into each of the multimodal large models respectively, and obtaining content understanding results corresponding to each of the multimodal large models, each of the content understanding results including a reference intention result and a reference content result; The reference intention results corresponding to each of the multimodal large models are fused and calculated to obtain the intention recognition result; The reference content results corresponding to each of the multimodal large models are fused and calculated to obtain the content evaluation result.

6. The method according to claim 1, characterized in that The content evaluation result includes a content category result and a content quality evaluation result; and the intention recognition result and the content evaluation result are stored in association with the published material, including: Tagging the published material according to the intent recognition result and the content category result to obtain first tag information corresponding to the intent attribute and second tag information corresponding to the content attribute; The first tag information, the second tag information and the content quality assessment result are associated with the published material and stored.

7. The method according to claim 6, characterized in that The method further comprises: Obtain user interaction behavior data of published materials corresponding to different tag information; According to different tag information and corresponding user interaction behavior data, a user interest model is constructed to perform material recommendation and / or material delivery based on the user interest model.

8. The method according to claim 6, characterized in that The method further comprises: If there is a latest published material containing an image, determining content reward information for the latest published material according to a content quality assessment result corresponding to the latest published material; Determining the delivery reward information of the latest released material according to the delivery indicator characteristics corresponding to the latest released material; Based on the content reward information and the delivery reward information, the latest released material is recommended.

9. The method according to claim 1, characterized in that: The method further comprises: Based on the timing time of the scheduled task, obtaining target published materials that have not been associated and stored within the timing time; Calling the offline interface of the multimodal large model, and performing content understanding processing on the target published material based on the task description information through the multimodal large model to obtain a target intent recognition result and a target content evaluation result; The target intent recognition result and the target content evaluation result are associated with the target published material and stored.

10. A material processing device, characterized in that: include: An information acquisition module, used to acquire task description information, wherein the task description information includes first description information corresponding to an intent attribute and second description information corresponding to a content attribute; A material processing module is used to obtain published materials through a real-time task flow, and based on the task description information, use a multimodal large model to understand the content of the published materials to obtain an intent recognition result and a content evaluation result; wherein the published materials include material content in at least two modes; and the multimodal large model is obtained by training based on multimodal sample data; An associated storage module is used to associate and store the intention recognition result and the content evaluation result with the published material.

Images