Content recommendation method and system based on industry knowledge graph and reinforcement learning

Abstract

The invention discloses a content recommendation method and system based on an industry knowledge graph and reinforcement learning, and relates to the technical field of advertisement recommendation, and the method comprises the steps: collecting original advertisement data and user behavior data, carrying out the entity recognition, relation extraction and attribute extraction, and constructing the industry knowledge graph; a graph attention network is adopted to analyze association strength among different modal entity nodes in the industry knowledge graph, advertisement elements are dynamically recombined according to the association strength, and a personalized advertisement material package is generated; mapping the personalized advertisement material package into a reinforcement learning action space, extracting a user historical behavior path from the industry knowledge graph to construct a state space, and obtaining a reinforcement learning environment; in a reinforcement learning environment, mapping the user behavior nodes into a relation chain of an industry knowledge graph, allocating reward values to intermediate nodes, and constructing an RL strategy model; deep modeling and strategy optimization of user behavior intentions are realized, and relevance of recommended content and intelligence of decision making are enhanced.

Description

Technical Field

[0001] This invention relates to the field of advertising recommendation technology, and in particular to a content recommendation method and system based on industry knowledge graphs and reinforcement learning. Background Technology

[0002] With the rapid development of digital advertising technology, ad content recommendation methods based on user interest modeling and contextual understanding have gradually become a hot topic in research and industrial applications. Existing ad recommendation technologies mainly employ collaborative filtering, content recommendation, and deep learning to mine preference patterns from user behavior data and combine them with ad creative features for matching and recommendation. Among these, knowledge graphs, as a structured semantic modeling tool, have been increasingly applied to ad recommendation scenarios in recent years to model the semantic relationships between users and ads, and between behaviors and attributes. Simultaneously, reinforcement learning methods have also been introduced into ad recommendation systems, dynamically optimizing recommendation strategies through a "state-action-reward" mechanism and continuously improving recommendation accuracy by simulating user interaction feedback.

[0003] Existing recommendation methods that integrate knowledge graphs and reinforcement learning still have some limitations. In particular, the process of constructing reinforcement learning environments and defining action-state spaces often struggles to fully express the heterogeneity and multimodality of the semantic structure of industry advertisements. Furthermore, as the final output of the recommendation system, the personalized combination of advertising creatives often fails to adjust in conjunction with the semantic relationships in the knowledge graph, making it difficult to achieve a refined mapping and generation from semantics to content. Summary of the Invention

[0004] In view of the aforementioned existing problems, the present invention is proposed.

[0005] Therefore, this invention provides a content recommendation method based on industry knowledge graphs and reinforcement learning to solve the problems of insufficient expression of industry semantic structure and imprecise personalized generation of advertising content in reinforcement learning environments.

[0006] To solve the above-mentioned technical problems, the present invention provides the following technical solution:

[0007] In a first aspect, the present invention provides a content recommendation method based on industry knowledge graph and reinforcement learning, which includes collecting raw advertising data and user behavior data, and performing entity recognition, relation extraction and attribute extraction to construct an industry knowledge graph;

[0008] The graph attention network is used to analyze the association strength between different modal entity nodes in the industry knowledge graph, and the advertising elements are dynamically reorganized according to the association strength to generate personalized advertising material packages.

[0009] Personalized advertising creative packages are mapped to the action space of reinforcement learning, while user historical behavior paths are extracted from industry knowledge graphs to construct a state space and obtain a reinforcement learning environment.

[0010] In a reinforcement learning environment, user behavior nodes are mapped to relational chains in an industry knowledge graph, and reward values ​​are assigned to intermediate nodes to construct an RL policy model.

[0011] The user's current request is input into the RL strategy model. Combining the user profile and advertising entity relationships in the industry knowledge graph, the optimal ad combination is selected from the action space and multimodal rendering is performed to output personalized recommendation content.

[0012] As a preferred embodiment of the content recommendation method based on industry knowledge graph and reinforcement learning described in this invention, the original advertising data includes advertising text data, advertising image data, advertising video data, and advertising delivery parameters;

[0013] The user behavior data includes both explicit and implicit user behaviors.

[0014] As a preferred embodiment of the content recommendation method based on industry knowledge graphs and reinforcement learning described in this invention, the specific steps for constructing the industry knowledge graph are as follows:

[0015] The original advertising data and user behavior data are processed for format standardization, correspondence matching, word segmentation and part-of-speech tagging to generate semantically normalized text data;

[0016] Perform entity recognition on semantically normalized text data to extract advertising-related entities and user behavior entities;

[0017] Dependency parsing is performed on advertising-related entities and user behavior entities to obtain entity semantic dependency paths, and entity semantic relationships are identified through relation extraction operations.

[0018] By utilizing the contextual information of raw advertising data and user behavior data, attribute extraction operations are performed on advertising-related entities and user behavior entities to obtain entity attribute information;

[0019] The industry knowledge graph is constructed by organizing advertising-related entities, user behavior entities, and entity semantic relationships into knowledge triples, and using knowledge triples and entity attribute information as nodes and edges of the graph structure.

[0020] As a preferred embodiment of the content recommendation method based on industry knowledge graphs and reinforcement learning described in this invention, the specific steps for generating personalized advertising material packages are as follows:

[0021] A graph attention network is used to encode the multimodal features of entity nodes with different modalities in the industry knowledge graph and extract node feature representations.

[0022] Based on the node feature representation, the features of adjacent nodes are weighted and aggregated through an attention mechanism to generate updated node features, and interaction calculations are performed to output association weight values.

[0023] The association weight value is used to perform semantic fusion on advertising-related entities and user behavior entities to generate multimodal node representations. These representations are then combined with the association weight value to update the association strength between entity nodes in real time, thus obtaining the real-time updated association strength.

[0024] By using real-time updated association strength, ad elements are clustered and filtered to generate personalized ad creative packages.

[0025] As a preferred embodiment of the content recommendation method based on industry knowledge graphs and reinforcement learning described in this invention, the specific steps for obtaining the reinforcement learning environment are as follows:

[0026] The ad type, modal content, style tags, and target user attributes are extracted from the personalized ad creative package to construct a structured ad action representation, and multi-dimensional feature combinations are performed to generate an action space for reinforcement learning.

[0027] Map user explicit behaviors and user implicit behaviors to user behavior nodes;

[0028] Using user identifiers as indexes, we extract the historical user behavior paths between users and advertisements from the industry knowledge graph, and combine user behavior nodes to generate state representations to construct a state space for reinforcement learning.

[0029] Input the action space and state space into the reinforcement learning modeling process, establish the state-action mapping relationship, and generate the reinforcement learning environment.

[0030] As a preferred embodiment of the content recommendation method based on industry knowledge graphs and reinforcement learning described in this invention, the specific steps for constructing the RL policy model are as follows:

[0031] In a reinforcement learning environment, user behavior nodes are matched with MetaPath templates in the industry knowledge graph to generate corresponding multi-hop relationship chains.

[0032] A multi-level attention mechanism is adopted to calculate the semantic importance of each intermediate node in the multi-hop relationship chain in turn, and to assign reward values ​​to each intermediate node according to a preset decreasing weight based on the path position of the intermediate node in the multi-hop relationship chain.

[0033] The reward values ​​of each intermediate node are aggregated according to the link order to obtain the total path-level reward signal;

[0034] The path-level total reward signal and the immediate feedback reward are weighted and merged proportionally to construct the final reward signal;

[0035] The final reward signal is input into the reinforcement learning modeling process, and an RL policy model is constructed through interactive learning of state, action, and reward.

[0036] As a preferred embodiment of the content recommendation method based on industry knowledge graphs and reinforcement learning described in this invention, the specific steps for outputting personalized recommended content are as follows:

[0037] Receive the user's current request and perform semantic parsing to extract request features;

[0038] The request features are fused with user profiles and advertising entity relationships in the industry knowledge graph using multimodal semantics to generate a state input vector;

[0039] Extract candidate ad actions from personalized ad creative packages;

[0040] Input the state input vector into the RL policy model, calculate the value score of each candidate ad action, sort the value scores from high to low, and select the candidate ad action with the highest value as the optimal ad combination;

[0041] Based on the optimal ad mix, retrieve the corresponding personalized ad creative package, perform multimodal fusion and layout design on the personalized ad creative, and output personalized recommendation content.

[0042] Secondly, the present invention provides a content recommendation system based on industry knowledge graph and reinforcement learning, including a data acquisition module for collecting raw advertising data and user behavior data, and performing entity recognition, relation extraction and attribute extraction to construct an industry knowledge graph;

[0043] The data acquisition module is used to collect raw advertising data and user behavior data, and to perform entity recognition, relationship extraction and attribute extraction to build an industry knowledge graph;

[0044] The material generation module is used to analyze the correlation strength between different modal entity nodes in the industry knowledge graph using graph attention network analysis, and dynamically reorganize advertising elements according to the correlation strength to generate personalized advertising material packages.

[0045] The environment construction module is used to map personalized advertising material packages into the action space of reinforcement learning, and at the same time extract user historical behavior paths from the industry knowledge graph to construct the state space and obtain the reinforcement learning environment.

[0046] The strategy decision module is used to map user behavior nodes to relational chains of industry knowledge graphs in a reinforcement learning environment, and to assign reward values ​​to intermediate nodes to build an RL strategy model.

[0047] The graph evolution module takes the user's current request as input to the RL strategy model, combines the user profile and advertising entity relationships in the industry knowledge graph, selects the optimal advertising combination from the action space, performs multimodal rendering, and outputs personalized recommendation content.

[0048] Thirdly, the present invention provides a computer device including a memory and a processor, wherein the memory stores a computer program, wherein: when the computer program is executed by the processor, it implements any step of the content recommendation method based on industry knowledge graph and reinforcement learning as described in the first aspect of the present invention.

[0049] Fourthly, the present invention provides a computer-readable storage medium having a computer program stored thereon, wherein: when the computer program is executed by a processor, it implements any step of the content recommendation method based on industry knowledge graph and reinforcement learning as described in the first aspect of the present invention.

[0050] The beneficial effects of this invention are as follows: By dynamically modeling the semantic associations between multimodal entity nodes in an industry knowledge graph using graph attention networks, personalized advertising material packages are generated, improving the matching accuracy and personalization level of advertising content; at the same time, by mapping user behavior nodes to multi-hop relationship chains in the knowledge graph and allocating semantic rewards, a reinforcement learning strategy model is constructed, realizing deep modeling and strategy optimization of user behavior intent, enhancing the relevance of recommended content and the intelligence of decision-making. Attached Figure Description

[0051] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the following description of the embodiments will be briefly introduced. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0052] Figure 1 This is a flowchart of a content recommendation method based on industry knowledge graphs and reinforcement learning.

[0053] Figure 2 This is a schematic diagram of a content recommendation system based on industry knowledge graphs and reinforcement learning.

[0054] Figure 3 A flowchart for building an industry knowledge graph.

[0055] Figure 4 A flowchart for generating personalized advertising creative packages. Detailed Implementation

[0056] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, the specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings.

[0057] Many specific details are set forth in the following description in order to provide a full understanding of the invention. However, the invention may also be practiced in other ways different from those described herein, and those skilled in the art can make similar extensions without departing from the spirit of the invention. Therefore, the invention is not limited to the specific embodiments disclosed below.

[0058] Secondly, the term "one embodiment" or "embodiment" as used herein refers to a specific feature, structure, or characteristic that may be included in at least one implementation of the present invention. The phrase "in one embodiment" appearing in different places in this specification does not necessarily refer to the same embodiment, nor is it a single or selective embodiment that is mutually exclusive with other embodiments.

[0059] Reference Figures 1-4 This is one embodiment of the present invention, which provides a content recommendation method based on industry knowledge graphs and reinforcement learning, including the following steps:

[0060] S1. Collect raw advertising data and user behavior data, and perform entity recognition, relationship extraction and attribute extraction to build an industry knowledge graph.

[0061] It should be noted that the original advertising data includes advertising text data, advertising image data, advertising video data, and advertising delivery parameters;

[0062] User behavior data includes both explicit and implicit user behavior.

[0063] Explicit user behavior refers to user interactions that can be directly observed, such as clicking, browsing, liking, forwarding, and commenting on advertising content.

[0064] Implicit user behavior refers to latent preferences that cannot be directly expressed but can be inferred from user behavior patterns, such as dwell time, scrolling speed, page pause positions, and swipe trajectories.

[0065] S1.1 Perform format standardization, matching, word segmentation, and part-of-speech tagging on the original advertising data and user behavior data to generate semantically standardized text data.

[0066] It should be noted that data format conversion operations are performed on the original advertising data and user behavior data respectively, and the advertising text data, advertising image data, advertising video data and advertising delivery parameters in the original advertising data are uniformly converted into structured text format;

[0067] Convert both explicit and implicit user behaviors in user behavior data into a structured behavior record text format.

[0068] After format conversion, the advertising text data is semantically associated with the text content in the user's explicit and implicit behaviors. Based on fields such as timestamp, behavior type, advertising identifier, and user identifier in the advertising content and user behavior content, a corresponding matching relationship between advertising content and user behavior is established.

[0069] For structured text content that has completed the corresponding match, word segmentation is performed on the text content based on natural language processing methods. By dividing continuous text into the smallest language units, word sequences are obtained.

[0070] Use existing part-of-speech tagging tools to perform part-of-speech tagging on the segmented word sequence, mark the grammatical role information of each word, and output semantically normalized text data that has undergone format standardization, correspondence matching, word segmentation and part-of-speech tagging.

[0071] S1.2 Perform entity recognition on the semantically normalized text data to extract advertising-related entities and user behavior entities.

[0072] It should be noted that named entity recognition is performed on the advertising text content in the semantically normalized text data to identify entities with independent semantics that can be used to construct advertising knowledge, including the advertising product name, advertising brand, advertising platform, advertising channel, advertising time, advertising location information, advertising content keywords, advertising modality type, and advertising target audience attributes. The identified advertising knowledge is used as advertising-related entities.

[0073] Named entity recognition is performed on user behavior text content in semantically normalized text data to extract information content with behavioral characteristics or behavioral intentions, including user behavior type, behavior occurrence time, behavior location, corresponding advertisement identifier, user interest tag, user active device type, and user preference category. The identified information content is used as user behavior entities.

[0074] S1.3 Perform dependency parsing on advertising-related entities and user behavior entities to obtain entity semantic dependency paths, and identify entity semantic relationships through relation extraction operations.

[0075] It should be noted that, based on advertising-related entities and user behavior entities, sentences containing advertising-related entities and user behavior entities are extracted from semantically normalized text data. Dependency parsing is performed on the sentences, and existing dependency parsing tools are used to parse the grammatical relationships between words in the sentences to generate a syntactic dependency tree structure.

[0076] By analyzing the syntactic dependency tree structure, we can identify the semantic dependency path between advertising-related entities and user behavior entities, and clarify the semantic connection relationships between entities in terms of syntactic modification, subject-verb, and object.

[0077] Based on the acquisition of entity semantic dependency paths, a relation extraction method is adopted to identify the types of semantic connection relationships between entities, including causal relationships, temporal relationships, subordinate relationships, parallel relationships and attribute relationships. For each pair of advertising-related entities and user behavior entities, the semantic relationship between entities is accurately identified by combining dependency syntax structure and context information.

[0078] It should also be noted that the specific steps for generating the syntactic dependency tree structure are as follows: Existing dependency parsing tools (Stanford parser, SpaCy dependency parser, and Tsinghua natural language processing tools, etc.) are used to segment and tag the grammatical relationships between words in the sentence, identify the dependency relationships between words in the sentence, such as subject-predicate, verb-object, and modification relationships, and construct a syntactic dependency tree structure based on these dependency relationships to represent the hierarchy and dependency relationships between words within the sentence, thereby achieving a comprehensive analysis of the sentence's grammatical structure.

[0079] S1.4. Using the contextual information of the original advertising data and user behavior data, perform attribute extraction operations on advertising-related entities and user behavior entities to obtain entity attribute information.

[0080] It should be noted that, by utilizing the contextual information of the original advertising data and user behavior data, entity attribute information is extracted from aspects such as advertising content description, advertising time, advertising channels, advertising creative elements, and advertising target user groups; at the same time, by combining the contextual features of user explicit and implicit behaviors in the user behavior data, such as time, location, frequency of behavior, and device type, attribute information of user behavior entities is extracted; during the attribute extraction process, the text environment in which the entity appears, the behavioral trajectory, and related fields in the contextual information are used to perform multi-dimensional attribute identification of advertising-related entities and user behavior entities, forming a complete set of entity attribute information;

[0081] It should also be noted that the contextual information of the original advertising data and user behavior data is linked and integrated through key fields such as timestamps, location identifiers, user identifiers, and advertising identifiers, and combined with the advertising environment, user access environment, and behavior sequence to form a contextual environment that includes time, space, and behavioral sequence relationships.

[0082] It should be noted that attribute extraction can be performed using a combination of rule matching and statistical features, incorporating dictionary, regular expressions, and contextual feature extraction rules.

[0083] S1.5 Organize advertising-related entities, user behavior entities, and entity semantic relationships into knowledge triples, and use the knowledge triples and entity attribute information as nodes and edges of the graph structure to establish an industry knowledge graph.

[0084] It should be noted that, based on advertising-related entities, user behavior entities, and identified entity semantic relationships, all entity pairs are traversed. Combining the semantic relationships between advertising-related entities and user behavior entities, each entity pair and its corresponding relationship is combined to form a knowledge triple of the form <Entity 1, Relationship, Entity 2>. By collecting all knowledge triples, a complete set of knowledge triples is constructed. Advertising-related entities and user behavior entities are used as nodes in the knowledge graph, and entity semantic relationships are used as edges connecting the nodes. The attribute information corresponding to each entity is then appended to the corresponding node as its attribute features. By summarizing the set of all knowledge triples, a complete graph structure containing nodes and edges is formed, i.e., the industry knowledge graph.

[0085] S2. Use graph attention network analysis to analyze the association strength between different modal entity nodes in the industry knowledge graph, and dynamically reorganize advertising elements according to the association strength to generate personalized advertising material packages.

[0086] S2.1. A graph attention network is used to encode the multimodal features of different modal entity nodes in the industry knowledge graph and extract node feature representations.

[0087] It should be noted that entity nodes containing multimodal information are extracted from the industry knowledge graph. These entity nodes correspond to multiple modal features such as advertising text, advertising images, advertising videos, and user behavior.

[0088] The multimodal features of each entity node are vectorized. The text content is converted into a semantic vector by a language encoder. The image content is input into a convolutional neural network. Low-level features such as edges, textures and shapes of local regions are extracted through multiple convolutional layers. Then, the dimensionality is reduced by pooling layers while retaining important structural and change information. Finally, global visual features with semantic representation capabilities are obtained in high-level convolutional layers. The output is a fixed-dimensional visual feature vector. The video content is analyzed frame by frame to extract significant change information in each frame. Combined with the dynamic evolution trend between frames, the temporal change pattern of continuous frames is compressed into a time-dependent vector representation to form a temporal feature vector for expressing the video content.

[0089] The visual feature vectors and temporal feature vectors mentioned above are fused according to their corresponding entity nodes to form a multimodal feature vector;

[0090] The multimodal feature vectors are input into the graph attention network. The graph attention mechanism is used to weight and aggregate the features of each entity node with those of its neighboring nodes. By calculating the attention weights between entity nodes and their neighbors, the semantic relationships and structural information between nodes are captured. Finally, the graph attention network outputs the node feature representation of each entity node, expressed as:

[0091] ;

[0092] In the formula, It is a physical node The node feature representation updated by the graph attention network. It is a non-linear activation function. It is a physical node and neighboring nodes Attention weights between them It is a linear transformation weight matrix. Neighboring nodes The original feature vector, It is the index variable of the neighboring nodes. It is a physical node The set of adjacent nodes, It is the index variable of the entity node;

[0093] It should be noted that the training of graph attention networks involves calculating the attention weight of each entity node through forward propagation and aggregating the features of neighboring nodes to generate new node representations. Then, backpropagation is used to minimize the loss function and update the network parameters (such as linear transformation weights and attention weights), thereby enabling graph attention networks to perform better on specific tasks.

[0094] The training process of a convolutional neural network includes using labeled training image data, calculating the error between the output of the convolutional neural network and the real label through forward propagation, using the backpropagation algorithm to pass the error gradient back to each convolutional layer and fully connected layer, using optimization algorithms (such as stochastic gradient descent) to adjust the weights and bias parameters of the convolutional kernel in the convolutional neural network, iterating multiple times until the error converges, and completing the training of the convolutional neural network.

[0095] S2.2 Based on the node feature representation, the features of adjacent nodes are weighted and aggregated through an attention mechanism to generate updated node features, and interaction calculations are performed to output the association weight value.

[0096] It should be noted that feature interaction calculation operations are performed on the node feature representation of the entity node and the node feature representation of each neighboring node to obtain the attention score between each pair of nodes, as expressed in the following expression:

[0097] ;

[0098] In the formula, It is a physical node with neighboring nodes Attention score It is a learnable parameter vector. It is a transpose operator. It is a physical node The original feature vector, It is a non-linear activation function;

[0099] Normalize all attention scores to obtain a set of normalized attention weights. Using this set, perform a weighted sum of the node feature representations of the corresponding neighboring nodes to generate updated node feature representations for the entity nodes. Using the updated entity node feature representations and the updated node feature representations of all neighboring nodes as input, perform feature interaction calculations to output the association weight values ​​between the entity node and each neighboring node. The expression is:

[0100] ;

[0101] In the formula, It is a physical node with neighboring nodes The correlation weight value, It is the updated neighbor node The node feature representation, It is a non-linear activation function.

[0102] S2.3. Use association weight values ​​to perform semantic fusion on advertising-related entities and user behavior entities to generate multimodal node representations. Combine these representations with association weight values ​​to update the association strength between entity nodes in real time and obtain the real-time updated association strength.

[0103] It should be noted that, based on the association weight value, the multimodal features of advertising-related entities and user behavior entities are weighted and fused. By fusing information from different modalities, the multidimensional semantic features of the entities are comprehensively reflected, thereby generating a multimodal node representation containing rich semantic information.

[0104] Multimodal node representations and association weights work together to reflect the semantic connection strength between entity nodes across different modalities. Based on this connection strength, the association relationships between entity nodes are continuously adjusted and optimized, thereby achieving dynamic strengthening and real-time updating of entity semantic relationships after multimodal feature fusion.

[0105] S2.4 uses real-time updated association strength to cluster and filter advertising elements, generating personalized advertising creative packages.

[0106] It should be noted that by utilizing the real-time updated association strength, comparing the multimodal feature expressions between advertising elements, and combining the semantic connection reflected by the association strength, the similarity of advertising elements in terms of content and presentation is judged, thereby identifying semantically related advertising elements, realizing similarity analysis, and classifying advertising elements with high semantic relevance into the same category, thus achieving clustering of advertising elements.

[0107] Based on the clustering results, the advertising elements are filtered, prioritizing the retention of advertising elements with strong semantic connections and rich multimodal information, while removing advertising elements with weak semantic connections or redundancy, ultimately generating a personalized advertising material package that meets individual needs and is rich in content.

[0108] It should be noted that the advertising elements are derived from the multimodal feature information contained in the advertising-related entities and user behavior entities. They are extracted from the node feature representation obtained by graph attention network encoding and used to reflect the multidimensional attributes of the advertising content, such as text, images, videos, and delivery parameters.

[0109] S3. Map personalized advertising material packages to the action space of reinforcement learning, and extract user historical behavior paths from the industry knowledge graph to construct the state space and obtain the reinforcement learning environment.

[0110] S3.1 Extract ad type, modal content, style tags and target user attributes from personalized ad creative packages, construct structured ad action representations, and combine multi-dimensional features to generate an action space for reinforcement learning.

[0111] It should be noted that each advertising element in the advertising material package is analyzed one by one. The advertising type is identified by analyzing the advertising text content, such as brand promotion, promotional activities, etc.; the multimodal information in the advertisement is analyzed, including text, image and video content, to clarify the modal content category of the advertisement, such as plain text, mixed text and image, video advertisement, etc.; advertising creative style tags are extracted, covering multiple dimensions of advertising expression, color style, visual effects and other characteristics, reflecting the overall style attributes of the advertisement.

[0112] Simultaneously, target user attribute information, including gender, age group, and interest preferences, is obtained from advertising strategies and historical user response data. The extracted ad types, modal content, style tags, and target user attributes are each vectorized and encoded, transforming discrete categories into numerical feature representations. Feature combination operations are then performed on multi-dimensional features, forming a comprehensive structured advertising action representation through feature concatenation or weighted fusion. This structured advertising action representation is ultimately used as the action space for reinforcement learning.

[0113] It should be noted that the advertising placement strategy is based on historical advertising data and user behavior data. It analyzes advertising performance metrics (such as click-through rate, conversion rate, and impressions) for different time periods, channels, and audiences, and combines these with the advertiser's goals and budget constraints to form an advertising placement plan and configuration.

[0114] Historical user response data is a record of user interaction behavior during the ad delivery process, including explicit behaviors such as clicking, browsing, liking, forwarding, and commenting on ads, as well as implicit behaviors such as dwell time, scrolling speed, and swipe trajectory.

[0115] S3.2 Map user explicit behaviors and user implicit behaviors to user behavior nodes.

[0116] It should be noted that for explicit user behavior, each explicit user behavior is mapped to a user behavior node based on the behavior type, occurrence time, associated advertising identifier, and user identifier, including the behavior category, timestamp, and corresponding advertising information;

[0117] For implicit user behavior, behavioral characteristics such as dwell time, scrolling speed, and swipe path are transformed into structured representations through behavioral trajectory analysis and time series processing, forming implicit user behavior nodes that cover information such as time, location, and behavior intensity. The explicit and implicit behavior nodes are then represented as a unified set of user behavior nodes.

[0118] S3.3 Using user identifiers as indexes, extract the user's historical behavior path between the user and the advertisement from the industry knowledge graph, combine the user behavior nodes to generate state representations, and construct the state space of reinforcement learning.

[0119] It should be noted that, using user identifiers as indexes, path information containing user behavior nodes and advertising-related entities is retrieved from the industry knowledge graph, and the historical behavior path between users and advertisements is extracted. The historical behavior path is composed of multiple user behavior nodes connected in chronological order.

[0120] The extracted historical user behavior paths are combined with a set of user behavior nodes, and a state representation representing the evolution of user behavior and the relationship between user behavior and advertising is formed through serialization. The state representation includes attribute information such as time, type, and location of user behavior nodes, reflecting the multi-dimensional behavioral characteristics of users and their association with advertising entities. Finally, a reinforcement learning state space is constructed based on the above state representation.

[0121] S3.4 Input the action space and state space into the reinforcement learning modeling process, establish the state-action mapping relationship, and generate the reinforcement learning environment.

[0122] It should be noted that the action space and state space are used as input data and fed into the reinforcement learning modeling process. The reinforcement learning modeling process includes gradually establishing a state-action mapping relationship based on the user behavior characteristics and advertising interaction information represented in the state space, combined with the advertising action characteristics in the action space. The specific steps are as follows: Encode the state space and action space into computable vector forms to facilitate subsequent policy updates; Analyze the value or benefit in the current state based on historical interaction data or environmental feedback to form a value assessment of the state-action pair; Adjust the action selection strategy based on the value assessment results to optimize the action response to the state; Finally, through continuous iterative updates, gradually improve the state-action mapping relationship and complete the construction of the reinforcement learning environment.

[0123] S4. In a reinforcement learning environment, user behavior nodes are mapped to relational chains in an industry knowledge graph, and reward values ​​are assigned to intermediate nodes to construct an RL policy model.

[0124] S4.1 In a reinforcement learning environment, user behavior nodes are matched with the MetaPath template in the industry knowledge graph to generate corresponding multi-hop relationship chains.

[0125] It should be noted that a predefined set of MetaPath templates is obtained, which describes the multi-hop relationship path pattern between user behavior nodes and ad-related entities. Subsequently, the set of user behavior nodes is traversed, and for each user behavior node, adjacent nodes that meet the conditions are identified and connected according to the relationship type and path length in the MetaPath template, thus gradually constructing a multi-hop relationship chain;

[0126] Each multi-hop relationship chain starts with a user behavior node, passes through intermediate multi-hop nodes, and finally connects to the target ad-related entity node. All nodes and edges along the path satisfy the semantic and structural constraints of the MetaPath template. Through matching and connection operations, a set of multi-hop relationship chains containing multiple nodes and edges is generated.

[0127] It should also be noted that the MetaPath template set is obtained by analyzing entity types and their semantic relationships in industry knowledge graphs, combined with prior knowledge in advertising and user behavior domains, and extracting typical multi-hop path patterns.

[0128] S4.2. Employ a multi-level attention mechanism to sequentially calculate the semantic importance of each intermediate node in the multi-hop relationship chain, and assign reward values ​​to each intermediate node according to a preset decreasing weight based on the path position of the intermediate node in the multi-hop relationship chain.

[0129] It should be noted that the feature representation of each intermediate node is extracted from the multi-hop relationship chain, and combined with the contextual information between nodes, the importance weight of each intermediate node relative to the entire relationship chain is obtained through weighted summation and normalization. Subsequently, a preset decreasing weight is set according to the path position of the intermediate node in the multi-hop relationship chain; for example, the weight of an intermediate node farther from the starting node is lower. A comprehensive reward value is assigned to each intermediate node by combining the obtained semantic importance weight and the preset decreasing weight.

[0130] It should also be noted that the decreasing weight is set based on the path position of the intermediate node in the multi-hop relationship chain. Combined with the distance order of the intermediate node in the path, the weight is gradually reduced according to the decreasing rule. For example, the farther away from the starting node, the lower the weight, to ensure that important nodes close to the starting point in the path receive higher weights.

[0131] S4.3 Aggregate the reward values ​​of each intermediate node according to the link order to obtain the path-level total reward signal.

[0132] It should be noted that the reward values ​​allocated to each intermediate node in the multi-hop relation chain are accumulated or weighted summed sequentially according to the order of the nodes in the link to form a path-level total reward signal representing the semantic contribution of the entire path.

[0133] S4.4. Combine the path-level total reward signal with the immediate feedback reward in a weighted manner to construct the final reward signal.

[0134] It should be noted that the total path-level reward signal and the immediate feedback reward are processed using a unified scale. By normalization or standardization methods, the total path-level reward signal and the immediate feedback reward are adjusted to the same numerical range to ensure that the reward values ​​are comparable when fused.

[0135] Based on the weight ratio, for example, weight Used for path-level total reward signal, weight For immediate feedback rewards, a weighted summation operation is performed on the normalized path-level total reward signal and the immediate feedback reward to generate the final reward signal;

[0136] It should be noted that the instant feedback reward is a reward signal generated based on the user's real-time interaction with the advertisement, including explicit behaviors such as user clicks, views, likes, shares, and comments, as well as implicit behaviors such as dwell time and scrolling speed.

[0137] Instant feedback rewards are obtained by analyzing the user's interaction data corresponding to the current request, combined with information such as behavior type, timestamp, and ad identifier, to quantify the user's instant response to the ad;

[0138] The weighting ratio is usually determined based on the importance assessment of the path-level total reward signal and the immediate feedback reward, combined with design goals and business needs. This ensures that the long-term behavioral impact and immediate response effect can be reasonably balanced during the integration, and that the final reward signal reflects both the historical path contribution and the current interaction performance.

[0139] S4.5 Input the final reward signal into the reinforcement learning modeling process, and construct the RL policy model through interactive learning of state-action-reward.

[0140] It should be noted that the selected state-action pair is value-evaluated based on the final reward signal to determine the effectiveness of the action within the given state. Subsequently, the action selection strategy is adjusted based on the value evaluation results, optimizing the direction of subsequent action selections to improve overall reward performance. Through multiple state-action-reward interactions, the mapping relationship between states and actions is continuously corrected and improved, gradually forming a reinforcement learning strategy model that can effectively guide ad recommendations.

[0141] S5. Input the user's current request into the RL strategy model, combine the user profile and advertising entity relationships in the industry knowledge graph, select the optimal advertising combination from the action space and perform multimodal rendering, and output personalized recommendation content.

[0142] S5.1 Receive the user's current request and perform semantic parsing to extract request features.

[0143] It should be noted that after receiving the user's current request, the request content is segmented and tagged with parts of speech to identify intent expressions and semantic entities. Combined with contextual information, user intent features, interests and preferences, and explicit and implicit needs in the request are extracted to form a structured request feature representation.

[0144] S5.2. Perform multimodal semantic fusion between the request features and user profiles and advertising entity relationships in the industry knowledge graph to generate a state input vector.

[0145] It should be noted that the request features are converted into semantic vector representations through a language encoder, and the user profiles and advertising entity relationships in the industry knowledge graph are used to extract corresponding multimodal feature vectors, including text features, image features and behavioral features.

[0146] Semantic-level fusion of request features and user profile features is performed, and information such as user interests and behavioral habits reflected in the user profile are associated and matched with request features. At the same time, multimodal features in advertising entity relationships are cross-fused with request features to explore the relationship between request intent and advertising content.

[0147] During the fusion process, a multi-head attention mechanism is used to weight and integrate features from different modalities, capture the interaction relationships and semantic associations between features, and finally output the fused state input vector.

[0148] S5.3 Extract each candidate ad action from the personalized ad creative package.

[0149] It should be noted that each advertising element in the personalized advertising creative package is analyzed one by one to extract advertising type information, such as brand promotion, promotional activities, etc.; the modal content of the advertisement is identified, including text, image and video formats; the style tags of the advertisement are obtained, covering multi-dimensional features such as presentation, color style and visual effects; at the same time, the target user attributes of the advertisement, such as gender, age group and interest preferences, are extracted, and the extracted features are structured to form a complete set of candidate advertising actions.

[0150] S5.4 Input the state input vector into the RL strategy model, calculate the value score of each candidate ad action, sort the value scores from high to low, and select the candidate ad action with the highest value as the optimal ad combination.

[0151] It should be noted that the fused state input vector is input into the reinforcement learning policy model to evaluate each candidate advertising action in the action space and generate a corresponding value score, expressed as:

[0152] ;

[0153] In the formula, In the state input vector Next candidate ad action Value rating It is the policy function corresponding to the reinforcement learning policy model. It is the state input vector generated by fusion. These are candidate ad actions in the action space;

[0154] The reinforcement learning strategy model evaluates the potential of each action to achieve the expected advertising effect in the current state based on the state input vector and the structured representation of the action, outputs a specific value score, sorts all candidate advertising action values ​​according to their numerical values, and selects the candidate advertising action with the highest value score as the optimal advertising combination.

[0155] S5.5. Based on the optimal ad mix, retrieve the corresponding personalized ad creative package, perform multimodal fusion and layout design on the personalized ad creative, and output personalized recommendation content.

[0156] It should be noted that, based on the optimal ad combination selected by the reinforcement learning strategy model, the corresponding ad elements are retrieved from the personalized ad creative package, and multimodal material information such as text, images and videos that match the optimal ad action are obtained one by one;

[0157] Feature fusion is performed on the retrieved multimodal advertising creatives. Text content is converted into semantic vectors using a language encoder, image content is extracted for visual features using a convolutional neural network, and video content is converted into temporal feature vectors using a temporal feature extraction method. The information from each modality is then combined to form a unified multimodal feature representation. Based on these multimodal features, the layout design of the advertising content is performed, including layout adjustments, visual style coordination, and element hierarchy optimization, ultimately generating personalized recommendation content that matches user preferences and the display environment.

[0158] This embodiment also provides a content recommendation system based on industry knowledge graph and reinforcement learning, including: a data acquisition module, used to collect raw advertising data and user behavior data, and perform entity recognition, relationship extraction and attribute extraction to construct an industry knowledge graph;

[0159] The data acquisition module is used to collect raw advertising data and user behavior data, and to perform entity recognition, relationship extraction and attribute extraction to build an industry knowledge graph;

[0160] The material generation module is used to analyze the correlation strength between different modal entity nodes in the industry knowledge graph using graph attention network analysis, and dynamically reorganize advertising elements according to the correlation strength to generate personalized advertising material packages.

[0161] The environment construction module is used to map personalized advertising material packages into the action space of reinforcement learning, and at the same time extract user historical behavior paths from the industry knowledge graph to construct the state space and obtain the reinforcement learning environment.

[0162] The strategy decision module is used to map user behavior nodes to relational chains of industry knowledge graphs in a reinforcement learning environment, and to assign reward values ​​to intermediate nodes to build an RL strategy model.

[0163] The graph evolution module takes the user's current request as input to the RL strategy model, combines the user profile and advertising entity relationships in the industry knowledge graph, selects the optimal advertising combination from the action space, performs multimodal rendering, and outputs personalized recommendation content.

[0164] This embodiment also provides a computer device applicable to content recommendation methods based on industry knowledge graphs and reinforcement learning, comprising: a memory and a processor; the memory is used to store computer-executable instructions, and the processor is used to execute the computer-executable instructions to implement the content recommendation method based on industry knowledge graphs and reinforcement learning as proposed in the above embodiment.

[0165] The computer device can be a terminal, comprising a processor, memory, communication interface, display screen, and input devices connected via a system bus. The processor provides computing and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The communication interface is used for wired or wireless communication with external terminals; wireless communication can be achieved through Wi-Fi, carrier networks, NFC (Near Field Communication), or other technologies. The display screen can be an LCD screen or an e-ink screen. The input devices can be a touch layer covering the display screen, buttons, a trackball, or a touchpad on the computer device's casing, or an external keyboard, touchpad, or mouse.

[0166] This embodiment also provides a storage medium storing a computer program that, when executed by a processor, implements the content recommendation method based on industry knowledge graphs and reinforcement learning as proposed in the above embodiments. The storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as Static Random Access Memory (SRAM), Electrically Erasable Programmable Read-Only Memory (EEPROM), Erasable Programmable Read Only Memory (EPROM), Programmable Red-Only Memory (PROM), Read-Only Memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk.

[0167] In summary, this invention improves the matching accuracy and personalization level of advertising content by dynamically modeling the semantic relationships between multimodal entity nodes in an industry knowledge graph using graph attention networks to generate personalized advertising material packages. Simultaneously, by mapping user behavior nodes to multi-hop relationship chains in the knowledge graph and assigning semantic rewards, a reinforcement learning strategy model is constructed, achieving deep modeling and strategy optimization of user behavior intentions, thus enhancing the relevance of recommended content and the intelligence of decision-making.

[0168] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the technical solutions of the present invention, and all such modifications or substitutions should be covered within the scope of the claims of the present invention.

Claims

1. A content recommendation method based on industry knowledge graphs and reinforcement learning, characterized in that: include, Collect raw advertising data and user behavior data, and perform entity recognition, relationship extraction and attribute extraction to build an industry knowledge graph; The graph attention network is used to analyze the association strength between different modal entity nodes in the industry knowledge graph, and the advertising elements are dynamically reorganized according to the association strength to generate personalized advertising material packages. Personalized advertising creative packages are mapped to the action space of reinforcement learning, while user historical behavior paths are extracted from industry knowledge graphs to construct a state space and obtain a reinforcement learning environment. In a reinforcement learning environment, user behavior nodes are mapped to relational chains in an industry knowledge graph, and reward values ​​are assigned to intermediate nodes to construct an RL policy model. The user's current request is input into the RL strategy model. Combining the user profile and advertising entity relationships in the industry knowledge graph, the optimal ad combination is selected from the action space and multimodal rendering is performed to output personalized recommendation content.

2. The content recommendation method based on industry knowledge graphs and reinforcement learning as described in claim 1, characterized in that: The original advertising data includes advertising text data, advertising image data, advertising video data, and advertising delivery parameters; The user behavior data includes both explicit and implicit user behaviors.

3. The content recommendation method based on industry knowledge graphs and reinforcement learning as described in claim 1, characterized in that: The specific steps for constructing the industry knowledge graph are as follows. The original advertising data and user behavior data are processed for format standardization, correspondence matching, word segmentation and part-of-speech tagging to generate semantically normalized text data; Perform entity recognition on semantically normalized text data to extract advertising-related entities and user behavior entities; Dependency parsing is performed on advertising-related entities and user behavior entities to obtain entity semantic dependency paths, and entity semantic relationships are identified through relation extraction operations. By utilizing the contextual information of raw advertising data and user behavior data, attribute extraction operations are performed on advertising-related entities and user behavior entities to obtain entity attribute information; The industry knowledge graph is constructed by organizing advertising-related entities, user behavior entities, and entity semantic relationships into knowledge triples, and using knowledge triples and entity attribute information as nodes and edges of the graph structure.

4. The content recommendation method based on industry knowledge graphs and reinforcement learning as described in claim 1, characterized in that: The specific steps for generating personalized advertising creative packages are as follows: A graph attention network is used to encode the multimodal features of entity nodes with different modalities in the industry knowledge graph and extract node feature representations. Based on the node feature representation, the features of adjacent nodes are weighted and aggregated through an attention mechanism to generate updated node features, and interaction calculations are performed to output association weight values. The association weight value is used to perform semantic fusion on advertising-related entities and user behavior entities to generate multimodal node representations. These representations are then combined with the association weight value to update the association strength between entity nodes in real time, thus obtaining the real-time updated association strength. By using real-time updated association strength, ad elements are clustered and filtered to generate personalized ad creative packages.

5. The content recommendation method based on industry knowledge graphs and reinforcement learning as described in claim 1, characterized in that: The specific steps for obtaining the reinforcement learning environment are as follows. The ad type, modal content, style tags, and target user attributes are extracted from the personalized ad creative package to construct a structured ad action representation, and multi-dimensional feature combinations are performed to generate an action space for reinforcement learning. Map user explicit behaviors and user implicit behaviors to user behavior nodes; Using user identifiers as indexes, we extract the historical user behavior paths between users and advertisements from the industry knowledge graph, and combine user behavior nodes to generate state representations to construct a state space for reinforcement learning. Input the action space and state space into the reinforcement learning modeling process, establish the state-action mapping relationship, and generate the reinforcement learning environment.

6. The content recommendation method based on industry knowledge graphs and reinforcement learning as described in claim 1, characterized in that: The specific steps for constructing the RL policy model are as follows. In a reinforcement learning environment, user behavior nodes are matched with MetaPath templates in the industry knowledge graph to generate corresponding multi-hop relationship chains. A multi-level attention mechanism is adopted to calculate the semantic importance of each intermediate node in the multi-hop relationship chain in turn, and to assign reward values ​​to each intermediate node according to a preset decreasing weight based on the path position of the intermediate node in the multi-hop relationship chain. The reward values ​​of each intermediate node are aggregated according to the link order to obtain the total path-level reward signal; The path-level total reward signal and the immediate feedback reward are weighted and merged proportionally to construct the final reward signal; The final reward signal is input into the reinforcement learning modeling process, and an RL policy model is constructed through interactive learning of state, action, and reward.

7. The content recommendation method based on industry knowledge graphs and reinforcement learning as described in claim 1, characterized in that: The specific steps for outputting personalized recommendations are as follows: Receive the user's current request and perform semantic parsing to extract request features; The request features are fused with user profiles and advertising entity relationships in the industry knowledge graph using multimodal semantics to generate a state input vector; Extract candidate ad actions from personalized ad creative packages; Input the state input vector into the RL strategy model, calculate the value score of each candidate ad action, sort the value scores from high to low, and select the candidate ad action with the highest value as the optimal ad combination; Based on the optimal ad mix, retrieve the corresponding personalized ad creative package, perform multimodal fusion and layout design on the personalized ad creative, and output personalized recommendation content.

8. A content recommendation system based on industry knowledge graphs and reinforcement learning, based on the content recommendation method based on industry knowledge graphs and reinforcement learning as described in any one of claims 1 to 7, characterized in that: include, The data acquisition module is used to collect raw advertising data and user behavior data, and to perform entity recognition, relationship extraction and attribute extraction to build an industry knowledge graph; The content generation module uses graph attention network analysis to analyze the correlation strength between different modal entity nodes in the industry knowledge graph, and dynamically reorganizes advertising elements based on the correlation strength to generate personalized advertising content packages. The environment construction module is used to map personalized advertising material packages into the action space of reinforcement learning, and at the same time extract user historical behavior paths from the industry knowledge graph to construct the state space and obtain the reinforcement learning environment. The strategy decision module is used to map user behavior nodes to relational chains of industry knowledge graphs in a reinforcement learning environment, and to assign reward values ​​to intermediate nodes to build an RL strategy model. The graph evolution module is used to input the user's current request into the RL strategy model, combine the user profile and advertising entity relationships in the industry knowledge graph, select the optimal advertising combination from the action space, perform multimodal rendering, and output personalized recommendation content.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that: When the processor executes the computer program, it implements the steps of the content recommendation method based on industry knowledge graph and reinforcement learning as described in any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by the processor, it implements the steps of the content recommendation method based on industry knowledge graph and reinforcement learning as described in any one of claims 1 to 7.

Images