Content processing system, method, device, equipment and medium

Abstract

The present application discloses a content processing system, method, device, equipment and medium, which relates to the field of computer technology and is used to improve the accuracy of content processing. The content processing system includes: a computing layer, a storage layer and a service layer, wherein: the computing layer is used to extract materials from the acquired content data and user data respectively to obtain various materials, and perform corresponding feature engineering processing on various materials respectively to obtain various features, wherein various features respectively include real-time features and sample features; the storage layer is used to store various materials and various features; the service layer is used to obtain the corresponding target real-time features according to the feature types required for each content processing task, and input the target real-time features into the trained target model to obtain the content processing results, wherein the trained target model is trained according to the target sample features of the same type as the corresponding target real-time features.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to the field of artificial intelligence technology, and provides a content processing system, method, device, equipment and medium. Background Art

[0002] With the continuous development of computer technology, various content processing platforms have gradually emerged. These content processing platforms can provide users with corresponding content services, such as recommending articles and videos to users.

[0003] At present, most content processing platforms use models to implement content service processing tasks. Before using the model, most of them need to be trained offline, that is, the model is trained in advance according to the corresponding sample data. After obtaining the trained model, the trained model is directly applied to the content processing platform to implement the content service processing task. However, this method cannot guarantee that the sample data used in the model training phase and the input data used in the model use phase have the same processing logic, which leads to poor model use effect and reduces the accuracy of the content processing results output by the model. Summary of the invention

[0004] Embodiments of the present application provide a content processing system, method, apparatus, device, and medium for improving the accuracy of content processing results.

[0005] On the one hand, an embodiment of the present application provides a content processing system, including a computing layer, a storage layer, and a service layer, wherein:

[0006] The computing layer is used to extract materials from the acquired content data and user data to obtain various materials, and perform corresponding feature engineering processing on the various materials to obtain various features, wherein the various features include real-time features and sample features, respectively. The real-time features are generated based on the materials in the current time period, and the sample features are generated based on the materials before the current time period;

[0007] The storage layer is used to store the various materials and the various features;

[0008] The service layer is used to obtain corresponding target real-time features according to the feature types required for each content processing task, and input the target real-time features into the trained target model to obtain content processing results, wherein the trained target model is trained based on target sample features of the same type as the corresponding types of the target real-time features.

[0009] On the one hand, an embodiment of the present application provides a content processing method, including:

[0010] Perform material extraction on the acquired content data and user data to obtain various materials, and perform corresponding feature engineering processing on the various materials to obtain various features, wherein the various features include real-time features and sample features, respectively. The real-time features are generated based on the materials in the current time period, and the sample features are generated based on the materials before the current time period;

[0011] storing the various materials and the various characteristics;

[0012] According to the feature types required for each content processing task, corresponding target real-time features are obtained, and the target real-time features are input into a trained target model to obtain content processing results, wherein the trained target model is trained based on target sample features of the same type as the corresponding types of the target real-time features.

[0013] In a possible embodiment, obtaining corresponding target real-time features according to the feature types required by each content processing task includes:

[0014] Determining a feature type corresponding to each content service task from pre-stored configuration information, wherein the configuration information includes a feature type required for each content service task;

[0015] According to the feature type corresponding to each content service task, the corresponding target real-time feature is obtained.

[0016] In a possible embodiment, the configuration information further includes the type of materials required for each content service task; and acquiring the corresponding target real-time feature according to the feature type corresponding to each content service task includes:

[0017] According to the feature type corresponding to each content service task, obtaining the corresponding target real-time feature from the various features;

[0018] If it is determined that there is no real-time feature corresponding to each content processing task among the various features, then acquiring corresponding target materials from the storage layer according to the type of materials required for each content service task;

[0019] Perform feature engineering processing on the target material to obtain target real-time features.

[0020] In a possible embodiment, after performing feature engineering processing on the target material to obtain target real-time features, the method further includes: storing the target real-time features.

[0021] In a possible embodiment, the obtained content data and user data are respectively subjected to material extraction to obtain various materials, including:

[0022] Calling a material calculation package corresponding to each type of data in the content data and the user data, extracting materials from the corresponding data, and obtaining various materials;

[0023] The various materials are subjected to corresponding feature engineering processing to obtain various features, including:

[0024] The feature operator package corresponding to each of the various materials is called, and feature engineering processing is performed on the corresponding materials to obtain various features.

[0025] In a possible embodiment, calling a material calculation package corresponding to each of the content data and the user data, extracting materials from the corresponding data, and obtaining various materials include:

[0026] Extracting materials from the content data according to the material calculation package corresponding to the content data to obtain content materials; and extracting materials from the user data according to the material calculation package corresponding to the user data to obtain user materials;

[0027] The calling of the feature operator package corresponding to each of the various materials performs feature engineering processing on the corresponding materials to obtain various features, including:

[0028] According to the feature operator package corresponding to the content material, feature engineering processing is performed on the content material to obtain content features; and according to the feature operator package corresponding to the user material, feature engineering processing is performed on the user material to obtain user features.

[0029] In a possible embodiment, the method further includes:

[0030] Obtaining an updated material calculation package from a message queue, and loading the updated material calculation package;

[0031] An updated feature operator package is obtained from the message queue, and the updated feature operator package is loaded.

[0032] In a possible embodiment, the method further includes:

[0033] From at least one pre-stored trained model, determine a target model required to process each content processing task.

[0034] In a possible embodiment, the method further includes: adding various sample features among the various features to a message queue, so that the training device performs the following operations:

[0035] Acquire the various sample features, and acquire corresponding sample features according to the feature types required for training each model in at least one model, and train the corresponding model according to the acquired sample features to obtain at least one trained model;

[0036] The at least one trained model is obtained from the training device.

[0037] On the one hand, an embodiment of the present application provides a content processing device, including:

[0038] The material processing module is used to extract materials from the acquired content data and user data to obtain various materials;

[0039] A feature processing module, used to perform corresponding feature engineering processing on the various materials to obtain various features, wherein the various features include real-time features and sample features, the real-time features are generated based on the materials in the current time period, and the sample features are generated based on the materials before the current time period;

[0040] A storage module, used for storing the various materials and the various characteristics;

[0041] The content processing module is used to obtain the corresponding target real-time features according to the feature types required for each content processing task, and input the target real-time features into the trained target model to obtain content processing results, wherein the trained target model is trained based on target sample features of the same type as the corresponding types of the target real-time features.

[0042] On the one hand, an embodiment of the present application provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of any of the above methods when executing the computer program.

[0043] On the one hand, an embodiment of the present application provides a computer-readable storage medium having computer program instructions stored thereon, and the computer program instructions implement the steps of any of the above methods when executed by a processor.

[0044] On the one hand, an embodiment of the present application provides a computer program product, which includes a computer program stored on a computer-readable storage medium, and the computer program includes program instructions, which implement the steps of any of the above methods when executed by a processor.

[0045] The content processing system, method, device, equipment and medium in the embodiment of the present application extract materials from content data and user data respectively to obtain various materials, and process various materials to obtain various features. In the training stage of the model, sample features among various features are used for training. In the stage of using the model, real-time features among various features are used to input the model to obtain content processing results. In this way, the consistency of the data processing logic in the model use and training stages is guaranteed, the use effect of the model is improved, and the accuracy of the content processing results is improved. In addition, the content processing system can classify and store various materials and various features generated during the content processing process to facilitate the subsequent use of these materials and features by other components. In addition, deploying a content processing system can realize multiple content processing services and reduce the cost of system deployment. In addition, the material processing logic and feature processing logic can be updated at any time, which improves the flexibility of the entire content processing system. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] By reading the detailed description below with reference to the accompanying drawings, the above and other purposes, features and advantages of the exemplary embodiments of the present application will become readily understood. In the accompanying drawings, several embodiments of the present application are shown in an exemplary and non-limiting manner, wherein:

[0047] Figure 1 A schematic diagram of an application scenario of the content processing method provided in an embodiment of the present application;

[0048] Figure 2 A schematic diagram of the structure of the content processing system provided in the embodiment of the present application Figure 1 ;

[0049] Figure 3 Structural example of the content processing system provided in the embodiment of the present application Figure 2 ;

[0050] Figure 4 An example diagram of the process of obtaining sample features provided in an embodiment of the present application;

[0051] Figure 5 A flowchart of a content processing method provided in an embodiment of the present application;

[0052] Figure 6 A schematic diagram of the structure of a content processing device provided in an embodiment of the present application;

[0053] Figure 7 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0054] The principles and spirit of the present application will be described below with reference to several exemplary embodiments. It should be understood that these embodiments are provided only to enable those skilled in the art to better understand and implement the present application, and are not intended to limit the scope of the present application in any way. On the contrary, these embodiments are provided to make the present disclosure more thorough and complete, and to fully convey the scope of the present disclosure to those skilled in the art.

[0055] Those skilled in the art know that the embodiments of the present application can be implemented as a system, device, equipment, method or computer program product. Therefore, the present disclosure can be specifically implemented in the following forms, namely: complete hardware, complete software (including firmware, resident software, microcode, etc.), or a combination of hardware and software.

[0056] It should be understood herein that any number of elements in the drawings is for illustration rather than limitation, and any naming is only for distinction rather than having any limiting meaning.

[0057] For ease of understanding, the terms involved in the embodiments of the present application are explained below:

[0058] 1. Artificial Intelligence (AI): It is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results.

[0059] 2. Machine learning: It is a branch of the field of artificial intelligence, focusing on automatically analyzing data to obtain patterns and using the patterns to predict unknown data.

[0060] 3. Deep learning model: A type of artificial intelligence model commonly used in machine learning. Common deep learning models include: Convolutional Neural Networks (CNN), Recurrent Neural Networks (RNN), Long Short-Term Memory Networks (LSTM), Deep Neural Networks (DNN), Deep Belief Nets (DBNs) and other neural networks. There are two ways for data to propagate in a neural network. One is along the path from input to output, which is called forward propagation, and the other is from output back to input, which is called backpropagation. In the forward propagation process, the input information is processed layer by layer by the neural network and transmitted to the output layer. The error between the output value and the expectation is described by the loss function, and then it is transferred to backpropagation. The partial derivative of the loss function with respect to the weight of each neuron is calculated layer by layer, forming the weight gradient data of the loss function with respect to the weight vector, which is used as the basis for updating the weight parameters. The training of the neural network is completed in the process of continuously updating the weight parameters.

[0061] 4. Content: including but not limited to video or graphic content, such as short videos and graphic content such as articles.

[0062] 5. Feature Engineering: Usually, each record in the data to be analyzed contains one or more attribute information, such as the user's age, gender, etc. Some attribute information (such as gender with a string data type) may not be directly applied to machine learning algorithms and need to be converted into numerical features. Sometimes, in order to more accurately characterize the data to be analyzed and train a better model, the features themselves also need to undergo certain conversions and / or combinations into new features. This process is called feature engineering.

[0063] 6. Material: refers to the original input data used for feature engineering. In the embodiment of the present application, materials can be divided into user materials and content materials according to the objects described by the data. User materials are data related to users, such as user static attribute data, user historical behavior data, and user real-time behavior data. Content materials are content-related data, such as content exposed by the system and content operated by users. After feature engineering is performed on the materials, features are obtained. Depending on the time period in which the features are generated, the features can be divided into real-time features and sample features. Sample features are used to train or update the model, while real-time features are used as input to the model to obtain the current content processing results.

[0064] 7. Kafka: A distributed message queue system, which is the core module for constructing distributed computing services across multiple machines.

[0065] 8. Redis: A distributed memory-based key-value storage system, equivalent to a distributed dictionary, which can be used for fast data query.

[0066] 9. User portrait: It can also be called character portrait, which is a variable set used to describe various types of user data. Personalized recommendation, advertising system, content recommendation, etc. are all applications that can be implemented based on user portrait. When you want to select a certain user group for refined operation, you can use user portrait to filter out specific groups. User portrait can be related to various factors such as the user's basic attributes, real-time behavior and historical behavior. Basic attributes include the user's age, gender, etc., and the user's real-time behavior and historical behavior include, for example, the user's click operation on the article, forwarding operation, etc.

[0067] 10. Material calculation package: describes the way to calculate data. A material calculation package can include one or more calculation logics. Each calculation logic can be used to process the corresponding data to obtain the corresponding material, or calculate the intermediate results of the material, etc.

[0068] 11. Feature operator package: describes the way to calculate materials. A feature operator may include one or more operators. Each operator can be used to process the corresponding material to obtain the corresponding features, or calculate the intermediate results of the features.

[0069] 12. Maven tool: It is a software project management and automatic build tool that can build and manage dependencies for Java projects. Maven tool can also be used to build and manage various projects, such as projects written in C#, Ruby, Scala and other languages. SUMMARY OF THE INVENTION

[0071] As discussed above, the model used by the content processing platform needs to be trained with a large amount of sample data before it can be used. In the related art, a large amount of sample data is usually collected in advance, the model is trained based on the sample data, and then the content processing platform collects real-time data and inputs the real-time data into the model to obtain the content processing results. However, the related art cannot guarantee the consistency of the processing logic of the sample data and the real-time data, which in turn affects the use effect of the model and reduces the accuracy of the content processing results output by the model. In addition, in the related art, the collection of sample data and real-time data is usually separated by a long time, that is, the time difference between the sample data and the real-time data is long, which is easy to cause feature crossing and also affects the use effect of the model.

[0072] In view of this, an embodiment of the present application provides a content service processing system, in which services, computing and storage are layered. The computing layer is used to extract materials from content data and user data to obtain various materials, and perform corresponding feature engineering processing on various materials to obtain various features. The storage layer is used to store various materials and various features, and the service layer is specifically used to obtain content processing results with the help of models. The model used by the service layer uses sample features among various features for training during its training phase. During the service layer model usage phase, real-time features among various features are input into the model to obtain content processing results output by the model. In this way, it is ensured that both sample features and real-time features use the same processing logic, thereby ensuring the consistency of online and offline data processing logic, improving the use effect of the model, and further improving the accuracy of the content processing results.

[0073] After introducing the basic principles of the present application, various non-limiting implementation methods of the present application are described in detail below.

[0074] Application Scenario Overview

[0075] refer to Figure 1 , is a schematic diagram of an application scenario of the content processing method provided in the embodiment of the present application. Figure 1 The application scenario shown includes a terminal device 110 and a first server 120. The terminal device 110 and the first server 120 may be connected via a wireless communication network or a wired communication network, and the terminal device 110 includes but is not limited to electronic devices such as desktop computers, mobile phones, mobile computers, tablet computers, media players, smart wearable devices, smart TVs, vehicle-mounted devices, and personal digital assistants (PDAs). The first server 120 may be an independent physical server, or a server cluster or distributed system consisting of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms.

[0076] The first server 120 is used to provide content services, such as content publishing, content recommendation and other services. The terminal device 110 is installed with a client, and the user can obtain the content services provided by the first server 120 through the client, or the user can also access the content website through the browser in the terminal device 110 to obtain the content services provided by the first server 120. For example, any user can upload content to the first server 120 corresponding to the content platform through the client to publish the created content through the content platform, so that other users can view the content published by the user.

[0077] The first server 120 analyzes and identifies the content uploaded by the user, and determines the relevant content based on the content analysis and identification results. The first server 120 determines the relevant content corresponding to the historically viewed content based on the historically viewed content of other users, and determines the content recommended to the user from the obtained related content, and sends the content to the user's terminal device.

[0078] In a possible application scenario, the application scenario may also include a second server 130, which is used to obtain corresponding sample data from the first server 120, train a model, obtain a trained model, and then send the trained model to the first server 120, so that the first server 120 can implement the corresponding content service described above according to the trained model. The specific implementation form of the second server 130 can refer to the implementation form of the first server 120 discussed above, and will not be repeated here.

[0079] Combine the following Figure 1 The content processing method according to the exemplary embodiment of the present application is described by using the application scenario. It should be noted that the above application scenario is only shown to facilitate understanding of the spirit and principle of the present application, and the implementation of the present application is not limited in this respect. On the contrary, the implementation of the present application can be applied to any applicable scenario.

[0080] Exemplary Methods

[0081] refer to Figure 2 , is a schematic diagram of the structure of a content processing system provided in an embodiment of the present application, the content processing system includes a service layer 210, a computing layer 220 and a storage layer 230. The content processing system can be specifically built on Figure 1 In the first server 120, the specific implementation of the first server 120 can refer to the above discussion and will not be repeated here.

[0082] The computing layer 220 is used to extract materials from the obtained content data and user data to obtain various materials, and to perform corresponding feature engineering processing on various materials to obtain various features.

[0083] The computing layer 220 in the embodiment of the present application can process the original data of different time periods by extracting common logic to obtain materials and process the materials, thereby ensuring the consistency of real-time model reasoning and background training materials and feature processing.

[0084] The computing layer 220 can specifically process the common logic involved in material processing into a material calculation package, and process the common logic of feature processing into a feature operator package. The data can be processed through the material calculation package, and the material can be feature engineered through the feature operator package, thereby ensuring the consistency of the processing logic. If the processing logic is changed later, it is only necessary to change the corresponding material calculation package or feature operator package, thereby reducing the difficulty of code changes and avoiding inconsistent processing processes caused by code changes.

[0085] Specifically, the material acquisition process includes acquiring data, which specifically includes user attributes, content data, user real-time behavior data, and user historical behavior data, etc., and performing material extraction processing on the data to obtain materials. The computing layer 220 can subscribe to the user's behavior log stream on the client from Hadoop FLINK to obtain the user's real-time behavior data. Specifically, the computing layer 220 can call the material calculation package to implement material extraction processing on these data to obtain materials. In order to improve processing efficiency, the computing layer 220 can encapsulate the material extraction logic into a material calculation package, and store the material calculation package in the material calculation library 222. When extracting materials, the computing layer 220 can call the corresponding material calculation package from the material calculation library 222.

[0086] For example, the computing layer 220 subscribes to the user's behavior log stream on the client through Hadoop FLINK. The behavior log stream includes the user's real-time behavior log. The user's real-time behavior includes the user's click behavior, feedback behavior (such as collection, follow-up, like, share, dislike, etc.), login behavior and refresh behavior, etc. The computing layer 220 parses the log stream to obtain the user ID, obtains the user's static information through the user ID, and fills in the user's various attributes, including basic information such as gender, age, platform, and network. The computing layer 220 can obtain the user's historical behavior data based on the log stream before the current time period. After obtaining various data, the computing layer 220 can encapsulate the various data into a structure to obtain material data. It should be noted that the log stream contains various behavior data of the user, and the behavior data extracted from the log stream of the current time period can be regarded as real-time behavior data, and the behavior data extracted from the log stream before the current time period can be regarded as historical behavior data.

[0087] Feature processing mainly includes performing feature engineering on materials to obtain features. Similarly, the computing layer 220 can call feature operator packages to perform feature engineering on materials to obtain features. Among them, a feature operator package includes one or more operators, and different operators can be used to process different materials or the same materials separately to obtain corresponding features.

[0088] For example, define feature B = op1(A), or B = op2(A, C). Here, op1 and op2 represent different operators, and op1 and op2 can belong to a feature operator package. Operators can be understood as a way to process materials, such as finding the maximum feature value max, the minimum feature value min, and the feature cross, which are not limited here. A and C represent the original input. After operator op1 processes A, feature B is obtained. After operator op2 processes A and C, feature B is obtained.

[0089] For example, developers use Maven tools to compile one or more corresponding feature operators into jar packages and store them in the feature operator library 221. When the computing layer 220 needs to use them, they can directly call them from the feature operator library 221. The feature operator library 221 can be, for example, a Maven repository.

[0090] As an embodiment, the material may be in the form of a matrix or a sequence, etc., and the feature may be in the form of a matrix, a vector or a single value, etc., without specific limitation.

[0091] In an embodiment of the present application, the material calculation packages used to calculate materials are stored in the material calculation library 222, and the feature operator packages used to calculate features are stored in the feature operator library 221, so that the calculation logic corresponding to the data used online and the data used offline are respectively dependent on the feature operator library 221 and the material calculation library 222 to ensure the consistency of the data processing logic, and then ensure that the same processing logic is used in the online recommendation engine, feature warehouse update, and offline training sample feature construction to ensure the consistency of the data calculation logic.

[0092] As an embodiment, for different materials, the material calculation packages involved may be different, so the calculation layer 220 needs to determine the material calculation package that needs to be called according to the type of material currently required. For the same material, the corresponding material calculation package is the same, thereby ensuring the consistency of the material processing process. Similarly, the feature operator packages involved in the processing of different materials may be different, but the feature operator packages corresponding to the processing of the same material in different time periods are the same, thereby meeting the needs of processing different materials and ensuring the consistency of the processing logic of the same feature.

[0093] For example, the computing layer 220 can obtain user data and content data based on real-time data in the log stream, call the corresponding material computing package, perform material extraction on the content data, obtain content materials, and call the corresponding material computing package to perform material extraction on the user data to obtain user materials. The computing layer 220 can call the feature operator package to perform feature engineering processing on the user material to obtain user features, and can call the corresponding feature operator package to process the content material to obtain content features.

[0094] As an embodiment, the computing logic involved in the computing layer 220 computing various materials and various features can be implemented through platforms such as hadoop, Hadoop flink and hive.

[0095] As an embodiment, the calculation layer 220 may obtain basic features in addition to content features and user features.

[0096] Specifically, the calculation layer 220 may extract the user material and the content material to obtain the basic material, and then perform feature engineering on the basic material to obtain the basic feature. The basic material may be generated by combining part of the user material with part of the content material, and the basic feature may be in vector form.

[0097] As an embodiment, user features, material features and basic features are divided according to the objects described by the features. In the embodiments of the present application, various features can also be divided into real-time features and sample features according to time. Features generated according to materials within the current time period are regarded as real-time features, and features generated according to materials before the current time period are regarded as sample features. The value of the current time period can be set according to demand. Real-time features and sample features each include user features, material features and basic features, etc. It should be noted that real-time features and sample features are relative concepts. For example, real-time features can be updated to sample features after a period of time.

[0098] Correspondingly, the computing layer 220 includes an offline computing framework 223 and an online computing framework 224. The offline computing framework 223 can be used to obtain the materials required for calculating sample features, and can also be used to calculate sample features required for training models, etc. The online computing framework 224 can be used to obtain the materials required for calculating real-time features, and can also be used to calculate real-time features required when using the model.

[0099] In specific implementation, the offline calculation framework 223 can call the material calculation package from the material calculation library 222 to obtain the materials required for calculating the sample features, and can also call the feature operator package from the feature operator library 221 to calculate the corresponding sample features. Similarly, the online calculation framework 224 can call the material calculation package from the material calculation library 222 to calculate the materials required for the real-time features, and can also call the feature operator package from the feature operator library 221 to calculate the corresponding real-time features.

[0100] The storage layer 230 is used to obtain various materials and various characteristics from the computing layer 220 and store the various materials and various characteristics.

[0101] In a specific implementation, in order to facilitate searching, the storage layer 230 can classify and store various materials and various features, store various materials in a material warehouse, and store various features in a feature warehouse.

[0102] Reference Figure 2 As shown, the material warehouse can be specifically divided into a user material warehouse 231 for storing user materials, a content material warehouse 233 for storing content materials, and a basic material warehouse 235 for storing basic materials, etc. The feature warehouse can also be specifically divided into a user feature warehouse 232 for storing user features, a content feature warehouse 234 for storing content features, and a basic feature warehouse 236 for storing basic features, etc. Among them, the storage layer 230 can be implemented using platforms such as hive, redis, myrocks, and hdfs.

[0103] Among them, the service layer 210 is used to obtain the corresponding target real-time features according to the feature types required for each content processing task, and input the target real-time features into the trained target model to obtain content processing results, wherein the trained target model is trained based on target sample features of the same type as the corresponding types of the target real-time features.

[0104] The service layer 210 may specifically include multiple services, and different services are used to process different content processing tasks. For example, the service layer 210 may include user services 211, content services 212, basic services 213, and log services 214. The service layer 210 of the content processing system in the embodiment of the present application can be implemented based on spring boot, and provide a unified calling interface to the recommendation engine through dubbo. The recommendation engine can call the corresponding service of the service layer 210 based on the dubbo interface.

[0105] Among them, user service 211 can be used to manage user materials and user features, including acquisition and monitoring management. User service 211 can be used to encapsulate the acquisition interface for materials and features. User service 211 also provides external monitoring functions to facilitate developers to observe the distribution of materials and features, including the minimum number of various operation behaviors, maximum click volume, mean value and coverage, etc. For example, the user service can call the user material or user feature return in the storage layer 230, and further set the duration. If it exceeds the preset duration, it is determined that the acquisition failed.

[0106] The implementation of content service 212 is different from that of user service. Content service 212 involves updating and acquiring content materials and content features. Content service 212 also provides external monitoring functions, including content distribution, maximum content clicks, minimum content clicks, coverage, etc. For example, when the content is a document, content service 212 can obtain the document identifier (ID) under the current traffic by subscribing to the kafka topic, obtain the corresponding features from the storage device for each document ID, and write them into the storage layer 230. Content service 212 provides an acquisition interface to the outside world, and when receiving a request, it obtains the corresponding materials and features from the storage layer 230 in turn according to the results. Figure 3 As shown, the content service 212 can send the content identifier of the material that has not been obtained to the Kafka message queue, and the computing layer 220 can update the characteristics of the content identifier and write it into the storage layer 230.

[0107] The basic service 213 is used to update and obtain basic materials and basic features, and also provides external monitoring functions. In addition, the log service 214 can be used to store offline required materials persistently, and can also be used to write the generated log stream into the message queue so that the computing layer 220 can obtain the corresponding data.

[0108] In specific implementation, the service layer 210 may receive a content processing request sent by a client to determine a corresponding content processing task. The service layer 210 calls the corresponding real-time features, inputs the real-time features into the corresponding trained model, and obtains the content processing result.

[0109] As an embodiment, the processing logic involved in the service layer 210 can be implemented through platforms such as hadoop, Hadoop flink and hive.

[0110] Combine the following Figure 3 , an example is given to introduce the process of the content processing system in the embodiment of the present application processing content processing tasks.

[0111] As an example, Figure 3As shown, the content processing system may also include a recommendation engine 380, which is used to directly respond to user requests. When receiving a corresponding request from the user, the recommendation engine can send a corresponding request to the service layer 210, and the service layer 210 can determine the corresponding content processing task based on the request sent by the recommendation engine 380.

[0112] For real-time content processing tasks, after determining the content processing tasks, the service layer 210 can determine the type of features required to process the content processing tasks, thereby obtaining the target real-time features. For example, the service layer 210 can determine the type of features required to process the real-time content processing tasks from the configuration information, the configuration information includes the features required to process each content processing task, the configuration information can also include the target materials required to process each content processing task, and the configuration information can also include the material calculation package and feature operator package corresponding to the features required to process each content processing task. The configuration information can be directly stored in the service layer 210, or as Figure 3 As shown, the content processing system may further include a configuration center 310 , and configuration information may be stored in the configuration center 310 in the content processing system.

[0113] After determining the required feature type, the service layer 210 can obtain the corresponding real-time feature from the storage layer 230 as the target real-time feature. Figure 3 As shown, the service layer 210 can specifically obtain the corresponding target real-time features from the feature redis.

[0114] However, in some cases, the computing layer 220 may not process the material in a timely manner, resulting in the storage layer 230 not having the corresponding real-time features. In this case, the service layer 210 may call the material calculation package required for the real-time content processing task through the online computing framework 224 of the computing layer 220, first perform material processing on the corresponding data, and obtain the target material corresponding to the content processing task, and further call the required feature operator package through the online computing framework 224 of the computing layer 220 to process the corresponding material and obtain the target real-time features.

[0115] Furthermore, the service layer 210 can also instruct the computing layer 220 to send the obtained target real-time characteristics and corresponding target materials to the storage layer 230 through the online computing framework 224, so that the storage layer 230 can store the target real-time characteristics and target materials. Figure 3 As shown, the real-time features can be written into the feature redis in the storage layer 230, and the target materials can be written into the material redis.

[0116] As an example, Figure 3As shown, the service layer 210 may also include a click-through rate prediction (PCTR) service 215, which is used to predict the advertisements and other contents that the user may click. In one embodiment, the PCTR service 215 is combined with other services of the service layer to display the recommended advertisements and other contents.

[0117] As an example, Figure 3 As shown, the content processing system may further include a monitoring center 330, which may monitor various parts of the content processing system in real time to avoid abnormalities of the content processing system.

[0118] To improve the processing efficiency of the computing layer 220, please refer to Figure 3 In the structural diagram of the content processing system shown in FIG. 1 , the computing layer 220 may specifically include feature tasks (361, 363) and material tasks 362. The tasks here can be understood as the task components required by the service layer 210 to call the online computing framework 224 or the offline computing framework 223 to implement a certain service.

[0119] Among them, feature tasks (361, 363) are used to obtain various features, and continue as Figure 3 As shown, the feature task (361, 363) may include a sample feature task 361 and a real-time feature task 363. The sample feature task 361 is specifically used to obtain sample features, and the real-time feature task 363 is specifically used to obtain real-time features. The material task 362 is used to obtain various materials. The specific methods of obtaining various features and various materials can refer to the contents discussed above, which will not be repeated here.

[0120] In specific implementation, the material task 362 can call the corresponding material calculation package to process the data and obtain the corresponding material. Similarly, the feature task can call the corresponding feature operator package to process the material and obtain the corresponding feature. For example, the sample feature task 361 can call the corresponding feature operator package to process the material and obtain the sample feature, and the real-time feature task 363 can call the corresponding feature operator package to process the material and obtain the real-time feature.

[0121] In actual implementation, the online computing framework 224 and the offline computing framework 223 separately process related tasks based on sample feature tasks, real-time feature tasks and material tasks, thereby improving the concurrency of the computing layer 220 and improving the efficiency of the content processing system.

[0122] As an example, continue to refer to Figure 3 , the computing layer 220 can obtain corresponding data from Hadoop flink 340 to process the data, thereby obtaining various materials and various features, etc.

[0123] As an embodiment, the computing layer 220 may also update various materials and various features. The update frequencies of various materials may be different, and the update frequencies of various features may also be different.

[0124] The following uses the document material as an example to introduce material updates and feature updates:

[0125] The document material includes the static information of the document and the dynamic information corresponding to the document. For example, in the information flow recommendation scenario, the document material can also have different update frequencies, such as the dynamic click rate of seconds and the static attribute of daily updates. After the document material is updated, the computing layer 220 processes the updated document material to obtain the updated document features, thereby updating the document features.

[0126] As an embodiment, user materials may also have different update time periods, with granularity of day, hour, minute, etc. For example, static attributes are updated at the day level, user behaviors are updated at the hour level, and dynamic materials have minute and hour update frequencies according to the refresh number and session control. Similarly, after the user materials are updated, the corresponding user features can also be calculated based on the updated user materials to implement the update of user features.

[0127] As an embodiment, user materials and document materials may be updated at the same update frequency, so that basic materials may be obtained based on the updated document materials and user materials, and basic features may be obtained based on the basic materials to achieve update of the basic features.

[0128] Furthermore, since there are many types of features, in order to facilitate subsequent processing and searching, each feature can also be stored according to different update frequencies. For example, each feature can be divided into static features, hourly features and real-time features.

[0129] In addition to updating various features, the computing layer 220 can also update feature operator packages and material computing packages, thereby updating the computing logic for processing materials and features.

[0130] The computing layer 220 can obtain the updated feature operator package, store the updated feature operator package, and specifically write the updated feature operator package into the feature operator library 221. The computing layer 220 can also obtain the updated material calculation package, store the updated material calculation package, and write the updated material calculation package into the material calculation library 222. The computing layer 220 can specifically obtain the updated feature operator package and the material calculation package from the message queue, and the updated feature operator package in the message queue can be released by the developer.

[0131] For example, a developer can compile the updated material calculation package into a jar package using the Maven tool, and then publish the compiled jar package to the message queue, so that the calculation layer 220 can store the obtained jar package in the material calculation library 222, which can be implemented by a Maven warehouse. The calculation layer 220 can parse the jar package to obtain the updated material calculation package.

[0132] Similarly, developers can use Maven tools to compile the updated feature operator package into a jar package, and then publish the compiled jar package to the message queue, so that the computing layer 220 can store the acquired jar package in the feature operator library 221, which can be implemented by a Maven repository, for example. The computing layer 220 can parse the jar package to obtain the updated feature operator package.

[0133] In order to facilitate the update of the feature operator package, the computing layer 220 can obtain the feature configuration file, parse the feature configuration file, and obtain the updated feature operator package to achieve the update of the feature operator package. Among them, the feature configuration file is used to describe the feature generation method. Similarly, the computing layer 220 can obtain the material configuration file, parse the material configuration file, and obtain the updated material calculation package to achieve the update of the material calculation package. Among them, the material configuration file is used to describe the material generation method, and specifically can be used to describe the updated material calculation package.

[0134] Feature configuration files and material configuration files can be configured and generated according to the needs of algorithm developers. Feature configuration files describe specified input materials and parameters, feature operator packages, and specified output features. Material configuration files describe specified input data and parameters, material calculation packages, and specified output materials, etc. The computing layer 220 can store and manage feature configuration files and material configuration files in the form of a relational database. Later, feature configuration files can be updated or added to increase the flexibility of generating features, and material configuration files can be updated or added to increase the flexibility of generating materials. In this way, while ensuring the consistency of the processing logic involved in the data used online and offline, flexibility can be greatly increased, and the feature engineering iteration efficiency of algorithm developers can be improved.

[0135] For example, the computing layer 220 can obtain an updated feature configuration file from the message queue, and according to the feature configuration file, load the feature operator jar package, perform feature engineering calculations to generate features, and store the features in the form of key values ​​in the feature warehouse in the storage layer 230, for example, in the format of "environment: feature configuration name: user id: feature" to form a key to complete the update of the features.

[0136] As an example, continue to refer to Figure 3 After processing and obtaining sample features, the sample feature task in the computing layer 220 can write the sample features into the hdfs 371 in the storage layer 230; after obtaining real-time features, the real-time feature task in the computing layer 220 can write the real-time features into the feature redis 372 in the storage layer 230 for use by the service layer 210, etc.; after obtaining various materials, the material task in the computing layer 220 can write various materials into the material redis 373 in the storage layer 230. Figure 3 The specific implementation of the storage layer 230 is merely an example and is not actually limited thereto.

[0137] After the service layer 210 obtains the target real-time features, it can input the target real-time features into the trained target model to obtain the output of the trained target model, thereby obtaining the content processing result. This involves how the service layer 210 specifically obtains the target model, which is described below by example:

[0138] The service layer 210 may be trained according to sample features among various features to obtain a trained model. Alternatively, the service layer 210 may obtain a trained model from a training device, and the training device may be implemented, for example, by the second server 130 mentioned above. When the service layer 210 obtains a trained model from a training device, it involves how the training device obtains sample features, which is described below by way of example:

[0139] After obtaining various features and various materials, the computing layer 220 can add sample features of the various features to the message queue so that the training device can obtain the sample features from the message queue and then train the corresponding model.

[0140] Combine the following Figure 4 The sample feature acquisition process shown in the figure introduces the process of obtaining sample features:

[0141] The computing layer 220 performs log union on the exposure log and the click log, obtains corresponding user data and content data from the unioned logs, writes the user data and content data into the redis cache, and then obtains various corresponding features based on these data.

[0142] In order to facilitate the use of the training device, different types of sample features will have corresponding labels. When the computing layer 220 adds the sample features to the message queue, the labels of the sample features can also be added to the message queue. The message queue can be, for example, Figure 3The kafka message queue 350 shown in the figure makes it easier for the subsequent training device to filter the sample features required by itself according to the label. Among them, the label is set according to the different behaviors of the user, that is, a label is set for each different behavior of the user, such as click, like, favorite, follow, forward, share and other labels. In addition, corresponding labels can also be set according to the different results generated by the user's different behaviors on the content, such as click duration, visit duration, number of favorites, number of forwards and number of shares, etc.

[0143] Since the data volume of the sample features is large, in the embodiment of the present application, the computing layer 220 may filter the sample features to control the data volume of the sample features.

[0144] For example, the configuration center 310 stores sample data filtering rules corresponding to various models, and the computing layer 220 can call the corresponding filtering rules to filter the sample features, such as filtering the corresponding sample features according to the hash value of the user ID, or filtering according to one or more tags to obtain the corresponding sample features.

[0145] Furthermore, the computing layer 220 can also sample sample features to obtain more representative sample features. Sample feature sampling is different from filtering. Sampling sample features can highlight business needs. The computing layer 220 can perform sampling according to a variety of sampling logics, such as positive sample sampling, article type sampling, and user activity sampling.

[0146] After the computing layer 220 obtains the sample features, it can generate data in a corresponding format according to the corresponding requirements and send the sample features to the message queue, so that the training device can obtain the sample features and perform model training based on the sample features. After the training device obtains the sample features, it can use the online optimization algorithm (Follow The Regularized Leader, FTRL) to update the model.

[0147] Similarly, after the training device obtains the trained model, the trained model can be written into a message queue so that the service layer 210 can obtain the trained model from the message queue. The message queue is, for example, a kafka message queue.

[0148] Furthermore, the service layer 210 may obtain multiple trained models, and different models may be suitable for processing different content processing tasks. Therefore, after obtaining multiple trained models, the service layer 210 can determine the model required to process the current content processing task from the multiple trained models as the target model. The specific correspondence between each content processing task and the model can be stored in the configuration center 310, or can be stored in the service layer 210.

[0149] The content processing system in the embodiment of the present application can separate services, storage, and computing, and designs a reasonable architecture. Specifically, a content processing system architecture based on the separation of services, computing, and storage is proposed. The content processing system can split feature operations from online real-time services, and maintain the unity of the computing logic during the model training process and the online use process. It also provides feature operator packages and material calculation packages to ensure the uniformity of the material processing process, avoid machine learning problems such as feature crossing, and improve the accuracy of using models to process content services.

[0150] Moreover, both the model training and model use processes rely on raw data, including historical behavior data generated by users in the client over a period of time, user portraits in various dimensions, real-time external information data of users in the client, and dynamic information of documents in the client. The content processing system in the embodiment of the present application can perform corresponding material extraction processing on the raw data respectively, so as to meet the data requirements in different processes and improve the processing efficiency of the system. Furthermore, feature engineering processing of materials involves a large number of calculations, as well as various types of transformation processing, including crossover, threshold division, value range nonlinear transformation, etc., and in multiple application scenarios, some calculation processes are unchanged and can be shared. Therefore, the content processing system in the embodiment of the present application can extract the common calculation logic to form a feature operator package and a material calculation package, reducing redundant calculations. The optimization of these calculation logics can improve the efficiency of the content processing process, and use the content processing system in business development to improve production efficiency, promote algorithm iteration, and improve business processing effects. In addition, materials or features can also be updated periodically or irregularly to meet the content processing needs of different time periods.

[0151] Moreover, in the embodiment of the present application, the calculation, storage, and service can be split to improve the processing flexibility of each part. In addition, materials and features can be added and changed according to the needs of algorithm developers. It is possible to generate a variety of materials and features to meet the needs of different models, and thus meet the needs of different content processing tasks.

[0152] The content processing system in the embodiment of the present application can also support dynamic testing. For example, the developer can request the corresponding content service through the content processing system at any time, so as to perform an overall test on the content processing system. In another embodiment, the developer can access the corresponding data of any layer in the content processing system through the corresponding interface, and can experiment with different materials and different feature processing methods, thereby improving the efficiency of troubleshooting, and has good flexibility and efficiency in practical applications.

[0153] Based on the same inventive concept as the content processing system, the embodiment of the present application provides a content processing method, which can be applied to the content processing system discussed above, for example, it can be executed by the first server mentioned above. Figure 5 The content processing method shown is a flowchart, the method comprising:

[0154] S501, extract materials from the acquired content data and user data to obtain various materials, and perform corresponding feature engineering processing on various materials to obtain various features.

[0155] Among them, various features include real-time features and sample features respectively. The real-time features are generated based on the materials in the current time period, and the sample features are generated based on the materials before the current time period.

[0156] S501 may be executed by the computing layer. The specific implementation method may refer to the content discussed above and will not be described again here.

[0157] S502, storing various materials and various characteristics.

[0158] S502 may be executed by the storage layer. The specific implementation method may refer to the content discussed above and will not be described again here.

[0159] S503, according to the feature types required for each content processing task, the corresponding target real-time features are obtained, and the target real-time features are input into the trained target model to obtain the content processing results.

[0160] The trained target model is obtained by training based on target sample features of the same type as the target real-time features.

[0161] S503 may be implemented by the service layer. The specific implementation method may refer to the content discussed above and will not be described again here.

[0162] Based on any of the above implementations, S501 specifically includes: extracting materials from the content data according to the material calculation package corresponding to the content data to obtain content materials; and extracting materials from the user data according to the material calculation package corresponding to the user data to obtain user materials;

[0163] According to the feature operator package corresponding to the content material, feature engineering processing is performed on the content material to obtain content features; and, according to the feature operator package corresponding to the user material, feature engineering processing is performed on the user material to obtain user features.

[0164] Furthermore, the content processing method also includes: obtaining an updated material calculation package from the message queue and loading the updated material calculation package, and obtaining an updated feature operator package from the message queue and loading the updated feature operator package.

[0165] Based on any of the above implementations, before executing S503, the content processing method further includes: determining a target model required to process each content processing task from at least one pre-stored trained model.

[0166] As an embodiment, sample features among various features are added to a message queue so that a training device can obtain various sample features, and obtain corresponding sample features according to the feature types required for training at least one model in training, and train the model according to the obtained sample features to obtain at least one trained model, and obtain at least one trained model from the training device.

[0167] Based on any of the above implementations, S503 specifically includes: determining the feature type corresponding to each content service task from pre-stored configuration information, wherein the configuration information includes the feature type required for each content service task, and obtaining the corresponding target real-time feature according to the feature type corresponding to each content service task.

[0168] Furthermore, the configuration information also includes the types of materials required for each content service task. S503 specifically includes: according to the type of features corresponding to each content service task, obtaining the corresponding target real-time features from the storage layer; if it is determined that the real-time features corresponding to each content processing task do not exist in the storage layer, then according to the types of materials required for each content service task, obtaining the corresponding target materials from the storage layer; performing feature engineering processing on the target materials to obtain the target real-time features.

[0169] Furthermore, when the target real-time characteristics are calculated in real time according to the corresponding target material, the content processing method further includes storing the target real-time characteristics.

[0170] The above-mentioned content processing method can be implemented on the basis of the content processing system. Its further implementation methods can refer to the implementation methods of the content processing system and will not be described in detail.

[0171] Exemplary Devices

[0172] Based on the same inventive concept as the above-mentioned content processing method, the embodiment of the present application also provides a content processing device, which can be used to implement the functions of the content processing system discussed above, please refer to Figure 6 , the content processing device comprises:

[0173] The calculation processing module 601 is used to extract materials from the acquired content data and user data to obtain various materials; and perform corresponding feature engineering processing on various materials to obtain various features, wherein various features include real-time features and sample features, respectively. The real-time features are generated based on the materials in the current time period, and the sample features are generated based on the materials before the current time period;

[0174] Storage module 602, used to store various materials and various characteristics;

[0175] The content processing module 603 is used to obtain the corresponding target real-time features according to the feature types required for each content processing task, and input the target real-time features into the trained target model to obtain the content processing results, wherein the trained target model is trained based on the target sample features of the same type as the corresponding target real-time features.

[0176] In a possible embodiment, the content processing module 603 is specifically used for:

[0177] According to the feature type corresponding to each content service task, the corresponding target real-time feature is obtained from the storage layer;

[0178] If it is determined that there is no real-time feature corresponding to each content processing task in the storage layer, then the corresponding target material is obtained from the storage layer according to the type of material required for each content service task;

[0179] Perform feature engineering on the target material to obtain the target real-time features.

[0180] In a possible embodiment, the storage module 602 is further used for:

[0181] Obtain the target real-time features from the service layer and store the target real-time features.

[0182] In a possible embodiment, the calculation processing module 601 is specifically used for:

[0183] Call the material calculation package corresponding to each type of data in the content data and user data, extract materials from the corresponding data, and obtain various materials;

[0184] Call the feature operator package corresponding to each of the various materials, perform feature engineering on the corresponding materials, and obtain various features.

[0185] In a possible embodiment, the calculation processing module 601 is specifically used for:

[0186] Extracting materials from the content data according to the material calculation package corresponding to the content data to obtain content materials; and extracting materials from the user data according to the material calculation package corresponding to the user data to obtain user materials;

[0187] According to the feature operator package corresponding to the content material, feature engineering processing is performed on the content material to obtain content features; and, according to the feature operator package corresponding to the user material, feature engineering processing is performed on the user material to obtain user features.

[0188] In a possible embodiment, the calculation processing module 601 is further used for:

[0189] Obtain the updated material calculation package from the message queue and load the updated material calculation package;

[0190] Get the updated feature operator package from the message queue and load the updated feature operator package.

[0191] In a possible embodiment, the content processing module 603 is further configured to:

[0192] Obtain the updated material calculation package from the message queue and load the updated material calculation package;

[0193] Get the updated feature operator package from the message queue and load the updated feature operator package.

[0194] In a possible embodiment, the content processing module 603 is further configured to:

[0195] From at least one pre-stored trained model, determine a target model required to process each content processing task.

[0196] In a possible embodiment, the calculation processing module 601 is further used to add various sample features among the various features to the message queue, so that the training device performs the following operations:

[0197] Acquire various sample features, and acquire corresponding sample features according to the feature types required for each model in training at least one model, and train the corresponding model according to the acquired sample features to obtain at least one trained model;

[0198] At least one trained model is obtained from a training device.

[0199] As an embodiment, the computing processing module 601 may be arranged in the computing layer discussed above, the storage module 602 may be arranged in the storage layer discussed above, and the content processing module 603 may be arranged in the service layer discussed above.

[0200] It should be noted that Figure 6The device shown can be used to implement any of the content processing methods discussed above, which will not be described in detail here.

[0201] Based on the same inventive concept as the above-mentioned content processing method, an embodiment of the present application also provides an electronic device, which may specifically be a single physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, as well as big data and artificial intelligence platforms.

[0202] like Figure 7 As shown, the electronic device may include at least one processor 701 and at least one memory 702. The memory 702 stores program codes. When the program codes are executed by the processor 701, the processor 701 executes various steps of the content processing method according to various exemplary embodiments of the present application described in the above “Exemplary Method” section of this specification. For example, the processor 701 may execute the following steps: Figure 5 As shown in S501, material extraction is performed on the acquired content data and user data respectively to obtain various materials, and corresponding feature engineering processing is performed on various materials to obtain various features; S502, various materials and various features are stored; S503, according to the feature types required for each content processing task, corresponding target real-time features are obtained, and the target real-time features are input into the trained target model to obtain content processing results.

[0203] Processor 701 can be a general-purpose processor, such as a central processing unit (CPU), a digital signal processor (DigitalSignal Processor, DSP), an application-specific integrated circuit (Application Specific Integrated Circuit, ASIC), a field programmable gate array (Field Programmable Gate Array, FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components, and can implement or execute the various methods, steps and logic block diagrams disclosed in the embodiments of the present application. A general-purpose processor can be a microprocessor or any conventional processor, etc. The steps of the method disclosed in conjunction with the embodiments of the present application can be directly embodied as a hardware processor to be executed, or a combination of hardware and software modules in the processor can be executed.

[0204] The memory 702 is a non-volatile computer-readable storage medium that can be used to store non-volatile software programs, non-volatile computer executable programs and modules. The memory may include at least one type of storage medium, such as a flash memory, a hard disk, a multimedia card, a card-type memory, a random access memory (Random Access Memory, RAM), a static random access memory (Static Random Access Memory, SRAM), a programmable read-only memory (Programmable Read Only Memory, PROM), a read-only memory (Read Only Memory, ROM), an electrically erasable programmable read-only memory (Electrically Erasable Programmable Read-Only Memory, EEPROM), a magnetic memory, a disk, an optical disk, etc. The memory is any other medium that can be used to carry or store a desired program code in the form of an instruction or data structure and can be accessed by a computer, but is not limited thereto. The memory 702 in the embodiment of the present application can also be a circuit or any other device that can realize a storage function, for storing program instructions and / or data.

[0205] As an example, Figure 7 The electronic device in can realize the functions of the content processing system discussed above, and can also be used to realize the functions of the content processing device discussed above.

[0206] Exemplary Program Products

[0207] An embodiment of the present application provides a computer-readable storage medium for storing computer program instructions used by the above-mentioned electronic device, which includes a program for executing the content processing method in any exemplary embodiment of the present application.

[0208] The above-mentioned computer storage media can be any available media or data storage devices that can be accessed by a computer, including but not limited to magnetic storage (such as floppy disks, hard disks, magnetic tapes, magneto-optical disks (MO), etc.), optical storage (such as CDs, DVDs, BDs, HVDs, etc.), and semiconductor storage (such as ROM, EPROM, EEPROM, non-volatile memory (NAND FLASH), solid-state drives (SSDs), etc.).

[0209] In some possible implementations, various aspects of the present application may also be implemented as a computer program product, which includes a program code. When the computer program product is run on a server device, the computer program product is used to enable the server device to execute the steps of the content processing method according to various exemplary implementations of the present application described in the above “Exemplary Method” section of this specification. For example, the server device may execute the following steps: Figure 5 As shown in S501, material extraction is performed on the acquired content data and user data respectively to obtain various materials, and corresponding feature engineering processing is performed on various materials to obtain various features; S502, various materials and various features are stored; S503, according to the feature types required for each content processing task, corresponding target real-time features are obtained, and the target real-time features are input into the trained target model to obtain content processing results.

[0210] The computer program product may use any combination of one or more readable media. The readable medium may be a readable signal medium or a readable storage medium. The readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of readable storage media (a non-exhaustive list) include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.

[0211] The computer program product for the content processing method according to the embodiment of the present application can adopt a portable compact disk read-only memory (CD-ROM) and include program code, and can be run on a server device. However, the program product of the present application is not limited to this. In this document, a computer-readable storage medium can be any tangible medium containing or storing a program, which can be used by or in combination with an instruction execution system, an apparatus or a device.

[0212] A readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, wherein readable program code is carried. Such propagated data signals may take a variety of forms, including, but not limited to, electromagnetic signals, optical signals, or any suitable combination of the foregoing. A readable signal medium may also be any readable medium other than a readable storage medium, which may transmit, propagate, or transfer a program for use by or in conjunction with an instruction execution system, apparatus, or device.

[0213] The program code embodied on the readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wireline, optical fiber cable, RF, etc., or any suitable combination of the foregoing.

[0214] Program code for performing the operations of the present application may be written in any combination of one or more programming languages, including object-oriented programming languages, such as Java, C++, etc., and conventional procedural programming languages, such as "C" or similar programming languages. The program code may be executed entirely on the user computing device, partially on the user computing device, partially on a remote computing device, or entirely on a remote computing device or server. In the case of a remote computing device, the remote computing device may be connected to the user computing device via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computing device (e.g., via the Internet using an Internet service provider).

[0215] It should be noted that, although several units or subunits of the device are mentioned in the above detailed description, this division is merely exemplary and not mandatory. In fact, according to the embodiments of the present application, the features and functions of two or more units described above can be embodied in one unit. Conversely, the features and functions of one unit described above can be further divided into multiple units to be embodied.

[0216] In addition, although the operations of the method of the present application are described in a specific order in the drawings, this does not require or imply that the operations must be performed in this specific order, or that all the operations shown must be performed to achieve the desired results. Additionally or alternatively, some steps may be omitted, multiple steps may be combined into one step, and / or one step may be decomposed into multiple steps.

[0217] Although the spirit and principle of the present application have been described with reference to several specific embodiments, it should be understood that the present application is not limited to the disclosed specific embodiments, and the division of various aspects does not mean that the features in these aspects cannot be combined to benefit, and such division is only for the convenience of expression. The present application is intended to cover various modifications and equivalent arrangements included in the spirit and scope of the attached claims.

Claims

1. A content processing system, characterized in that: It includes computing layer, storage layer and service layer, among which: The computing layer is used to extract materials from the acquired content data and user data to obtain various materials, and perform corresponding feature engineering processing on the various materials to obtain various features, wherein the various features include real-time features and sample features, respectively. The real-time features are generated based on the materials in the current time period, and the sample features are generated based on the materials before the current time period; The storage layer is used to store the various materials and the various features; The service layer is used to obtain the corresponding target real-time features according to the feature types required for each content processing task, and input the target real-time features into the trained target model to obtain content processing results, wherein the trained target model is trained based on target sample features of the same type as the corresponding types of the target real-time features, wherein the service layer is specifically used to: determine the feature type corresponding to each content service task from pre-stored configuration information, wherein the configuration information includes the feature types required for each content service task, and obtain the corresponding target real-time features according to the feature types corresponding to each content service task.

2. The system according to claim 1, characterized in that The configuration information also includes the types of materials required for each content service task; the service layer is specifically used for: According to the feature type corresponding to each content service task, obtaining the corresponding target real-time feature from the storage layer; If it is determined that the storage layer does not contain the real-time feature corresponding to each content processing task, then acquiring the corresponding target material from the storage layer according to the type of material required for each content service task; Perform feature engineering processing on the target material to obtain target real-time features.

3. The system according to claim 2, characterized in that The storage layer is also used to: The target real-time characteristics are obtained from the service layer and stored.

4. The system according to claim 1, characterized in that The computing layer is specifically used for: Calling a material calculation package corresponding to each type of data in the content data and the user data, extracting materials from the corresponding data, and obtaining various materials; The feature operator package corresponding to each of the various materials is called, and feature engineering processing is performed on the corresponding materials to obtain various features.

5. The system according to claim 4, characterized in that The computing layer is specifically used for: Extracting materials from the content data according to the material calculation package corresponding to the content data to obtain content materials; and extracting materials from the user data according to the material calculation package corresponding to the user data to obtain user materials; According to the feature operator package corresponding to the content material, feature engineering processing is performed on the content material to obtain content features; and according to the feature operator package corresponding to the user material, feature engineering processing is performed on the user material to obtain user features.

6. The system according to claim 4 or 5, characterized in that The computation layer is also used to: Obtaining an updated material calculation package from a message queue, and loading the updated material calculation package; An updated feature operator package is obtained from the message queue, and the updated feature operator package is loaded.

7. The system according to claim 1, characterized in that The service layer is also used to: From at least one pre-stored trained model, determine a target model required to process each content processing task.

8. The system according to claim 7, characterized in that The computing layer is further used to add various sample features among the various features to a message queue, so that the training device performs the following operations: Obtaining the various sample features, and obtaining corresponding sample features according to the feature types required for each model in training at least one model; According to the acquired sample features, the corresponding model is trained to obtain at least one trained model, and the at least one trained model is sent to the service layer.

9. A content processing method, characterized in that: include: Perform material extraction on the acquired content data and user data to obtain various materials, and perform corresponding feature engineering processing on the various materials to obtain various features, wherein the various features include real-time features and sample features, respectively. The real-time features are generated based on the materials in the current time period, and the sample features are generated based on the materials before the current time period; storing the various materials and the various characteristics; According to the feature types required by each content processing task, corresponding target real-time features are obtained, and the target real-time features are input into the trained target model to obtain content processing results, wherein the trained target model is trained based on target sample features of the same type as the corresponding type of the target real-time features, wherein the method of obtaining the corresponding target real-time features according to the feature types required by each content processing task comprises: determining the feature types corresponding to each content service task from pre-stored configuration information, wherein the configuration information comprises the feature types required for each content service task, and obtaining the corresponding target real-time features according to the feature types corresponding to each content service task.

10. The method according to claim 9, characterized in that The configuration information also includes the type of materials required for each content service task; the step of obtaining the corresponding target real-time feature according to the feature type corresponding to each content service task includes: According to the feature type corresponding to each content service task, obtaining the corresponding target real-time feature from the various features; If it is determined that there is no real-time feature corresponding to each content processing task among the various features, then obtaining the corresponding target material from the storage layer according to the type of material required for each content service task; Perform feature engineering processing on the target material to obtain target real-time features.

11. The method according to claim 10, characterized in that After performing feature engineering processing on the target material to obtain target real-time features, the method further includes: storing the target real-time features.

12. The method according to claim 9, characterized in that The obtained content data and user data are respectively subjected to material extraction to obtain various materials, including: Calling a material calculation package corresponding to each type of data in the content data and the user data, extracting materials from the corresponding data, and obtaining various materials; The various materials are subjected to corresponding feature engineering processing to obtain various features, including: The feature operator package corresponding to each of the various materials is called, and feature engineering processing is performed on the corresponding materials to obtain various features.

13. The method according to claim 12, characterized in that The calling of the material calculation package corresponding to each of the content data and the user data, extracting materials from the corresponding data, and obtaining various materials, including: Extracting materials from the content data according to the material calculation package corresponding to the content data to obtain content materials; and extracting materials from the user data according to the material calculation package corresponding to the user data to obtain user materials; The calling of the feature operator package corresponding to each of the various materials performs feature engineering processing on the corresponding materials to obtain various features, including: According to the feature operator package corresponding to the content material, feature engineering processing is performed on the content material to obtain content features; and according to the feature operator package corresponding to the user material, feature engineering processing is performed on the user material to obtain user features.

14. The method according to claim 12 or 13, characterized in that The method further comprises: Obtaining an updated material calculation package from a message queue, and loading the updated material calculation package; An updated feature operator package is obtained from the message queue, and the updated feature operator package is loaded.

15. The method according to claim 9, characterized in that The method further comprises: Obtaining an updated material calculation package from a message queue, and loading the updated material calculation package; An updated feature operator package is obtained from the message queue, and the updated feature operator package is loaded.

16. The method according to claim 15, characterized in that The method further includes: adding various sample features among the various features to a message queue, so that the training device performs the following operations: Acquire the various sample features, and acquire corresponding sample features according to the feature types required for training each model in at least one model, and train the corresponding model according to the acquired sample features to obtain at least one trained model; The at least one trained model is obtained from the training device.

17. A content processing device, characterized in that: include: A calculation and processing module, used to extract materials from the acquired content data and user data to obtain various materials; and to perform corresponding feature engineering processing on the various materials to obtain various features, wherein the various features include real-time features and sample features, respectively. The real-time features are generated based on the materials in the current time period, and the sample features are generated based on the materials before the current time period; A storage module, used for storing the various materials and the various characteristics; A content processing module is used to obtain corresponding target real-time features according to the feature types required for each content processing task, and input the target real-time features into a trained target model to obtain content processing results, wherein the trained target model is trained based on target sample features of the same type as the corresponding types of the target real-time features, wherein the content processing model is specifically used to: determine the feature type corresponding to each content service task from pre-stored configuration information, wherein the configuration information includes the feature type required for each content service task, and obtain the corresponding target real-time features according to the feature type corresponding to each content service task.

18. The device according to claim 17, characterized in that The configuration information also includes the types of materials required for each content service task; the content processing model is specifically used for: According to the feature type corresponding to each content service task, obtaining the corresponding target real-time feature from the various features; If it is determined that there is no real-time feature corresponding to each content processing task among the various features, then obtaining the corresponding target material from the storage layer according to the type of material required for each content service task; Perform feature engineering processing on the target material to obtain target real-time features.

19. The device according to claim 18, characterized in that The storage module is also used for: After performing feature engineering processing on the target material to obtain the target real-time features, the target real-time features are stored.

20. The device according to claim 17, characterized in that The calculation processing module is specifically used for: Calling a material calculation package corresponding to each type of data in the content data and the user data, extracting materials from the corresponding data, and obtaining various materials; The various materials are subjected to corresponding feature engineering processing to obtain various features, including: The feature operator package corresponding to each of the various materials is called, and feature engineering processing is performed on the corresponding materials to obtain various features.

21. The device according to claim 20, characterized in that The calculation processing module is specifically used for: Extracting materials from the content data according to the material calculation package corresponding to the content data to obtain content materials; and extracting materials from the user data according to the material calculation package corresponding to the user data to obtain user materials; The calling of the feature operator package corresponding to each of the various materials performs feature engineering processing on the corresponding materials to obtain various features, including: According to the feature operator package corresponding to the content material, feature engineering processing is performed on the content material to obtain content features; And, according to the feature operator package corresponding to the user material, feature engineering processing is performed on the user material to obtain user features.

22. The device according to claim 19 or 20, characterized in that The computing and processing module is also used for: Obtaining an updated material calculation package from a message queue, and loading the updated material calculation package; An updated feature operator package is obtained from the message queue, and the updated feature operator package is loaded.

23. The device according to claim 17, characterized in that The computing and processing module is also used for: Obtaining an updated material calculation package from a message queue, and loading the updated material calculation package; An updated feature operator package is obtained from the message queue, and the updated feature operator package is loaded.

24. The device according to claim 23, characterized in that The computing and processing module is also used for: Various sample features among the various features are added to the message queue, so that the training device performs the following operations: Acquire the various sample features, and acquire corresponding sample features according to the feature types required for training each model in at least one model, and train the corresponding model according to the acquired sample features to obtain at least one trained model; The at least one trained model is obtained from the training device.

25. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the steps of the method according to any one of claims 9 to 16 are implemented.

26. A computer-readable storage medium having computer program instructions stored thereon, characterized in that: When the computer program instructions are executed by a processor, the steps of the method according to any one of claims 9 to 16 are implemented.

Images