Information recommendation method and device, electronic equipment, computer readable storage medium and computer program product

Abstract

The invention provides an information recommendation method and device, electronic equipment, a computer readable storage medium and a computer program product. The method comprises the steps that graph data are constructed based on first recommendation information in first data of an object, and the graph data are used for determining recommendation probabilities of different types of recommendation information; feature extraction is carried out on the second data of the object to obtain multiple modal features of the object, the multiple modal features are fused to obtain a first fusion feature, and the first fusion feature is used for classifying recommendation information in a recommendation information base; and generating a recommendation information sequence for the object based on the first fusion feature and the graph data, and taking recommendation information in the recommendation information sequence as second recommendation information sent to the object. Through the information recommendation method and device, the information recommendation accuracy can be effectively improved.

Description

Technical Field

[0001] The present application relates to the field of data processing, and in particular to an information recommendation method, device, electronic device, computer-readable storage medium, and computer program product. Background Art

[0002] In related technologies, it is difficult to effectively integrate user behavior data, advertising content data and other auxiliary data (such as social data and video data) in advertising recommendation scenarios, and there is a hierarchical relationship between the label data of advertisements, which leads to the recommendation system being unable to fully utilize the available information, thereby making the accuracy of information recommendations low. Summary of the Invention

[0003] The embodiments of the present application provide an information recommendation method, apparatus, electronic device, computer-readable storage medium, and computer program product, which can effectively improve the accuracy of information recommendation.

[0004] The technical solution of the embodiment of the present application is implemented as follows:

[0005] This embodiment of the present application provides an information recommendation method, including:

[0006] constructing graph data based on first recommendation information in the first data of the object, wherein the graph data is used to determine recommendation probabilities of different types of recommendation information;

[0007] performing feature extraction on the second data of the object to obtain multiple modal features of the object, and fusing the multiple modal features to obtain a first fused feature, wherein the first fused feature is used to classify the recommendation information;

[0008] A recommendation information sequence for the object is generated based on the first fused feature and the graph data, and the recommendation information in the recommendation information sequence is used as second recommendation information sent to the object.

[0009] The present invention provides an information recommendation device, including:

[0010] a construction module, configured to construct graph data based on first recommendation information in first data of the object, wherein the graph data is used to determine recommendation probabilities of different types of recommendation information;

[0011] a processing module, configured to perform feature extraction on the second data of the object to obtain multiple modal features of the object, and fuse the multiple modal features to obtain a first fused feature, wherein the first fused feature is used to classify the recommendation information;

[0012] A recommendation module is configured to generate a recommendation information sequence for the object based on the first fusion feature and the graph data, and use the recommendation information in the recommendation information sequence as second recommendation information sent to the object.

[0013] An embodiment of the present application provides an electronic device, including:

[0014] a memory for storing computer-executable instructions or computer programs;

[0015] The processor is configured to implement the information recommendation method provided in the embodiment of the present application when executing the computer-executable instructions or computer program stored in the memory.

[0016] An embodiment of the present application provides a computer-readable storage medium storing a computer program or computer-executable instructions, which, when executed by a processor, implements the information recommendation method provided in the embodiment of the present application.

[0017] An embodiment of the present application provides a computer program product, including a computer program or computer-executable instructions. When the computer program or computer-executable instructions are executed by a processor, the information recommendation method provided in the embodiment of the present application is implemented.

[0018] The embodiments of the present application have the following beneficial effects:

[0019] By constructing graph data based on the first recommendation information in the first data of the object, it is possible to make more effective use of multiple data information; at the same time, feature extraction is performed on the second data of the object to obtain multiple modal features of the object, so as to effectively capture the hierarchical relationship between different data, and multiple modal features are fused to obtain the first fused feature, which can effectively integrate data information of different modalities and enhance the expressive ability of the feature; finally, based on the first fused feature and the graph data, the second recommendation information to be sent to the object is determined. In this way, by fusing multiple data information, the accuracy of information recommendation can be effectively improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] Figure 1 1 is a schematic diagram of the architecture of the information recommendation system 100 provided in an embodiment of the present application;

[0021] Figure 2 is a structural diagram of an electronic device 500 provided in an embodiment of the present application;

[0022] Figure 3 Schematic diagram of the flow of the information recommendation method provided in the embodiment of the present application;

[0023] Figure 4 Schematic diagram of the flow of the information recommendation method provided in the embodiment of the present application;

[0024] Figure 5 Schematic diagram of the flow of the information recommendation method provided in the embodiment of the present application;

[0025] Figure 6 This is a schematic diagram of the principle of the information recommendation method provided in an embodiment of the present application. DETAILED DESCRIPTION

[0026] In order to make the purpose, technical solutions and advantages of this application clearer, the application will be further described in detail below with reference to the accompanying drawings. The described embodiments should not be regarded as limiting this application. All other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of this application.

[0027] In the following description, reference is made to “some embodiments”, which describes a subset of all possible embodiments, but it will be understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0028] It is understandable that in the embodiments of the present application, when user information and other related data (such as user behavior data) are involved, when the embodiments of the present application are applied to specific products or technologies, user permission or consent is required, and the collection, use and processing of relevant data must comply with relevant laws, regulations and standards.

[0029] In the embodiments of the present application, the term "module" or "unit" refers to a computer program or a part of a computer program that has a predetermined function and works together with other related parts to achieve a predetermined goal, and can be implemented in whole or in part by using software, hardware (such as processing circuits or memories) or a combination thereof. Similarly, a processor (or multiple processors or memories) can be used to implement one or more modules or units. In addition, each module or unit can be part of an overall module or unit that includes the function of the module or unit.

[0030] In the following description, the terms "first\second\..." are merely used to distinguish similar objects and do not represent a specific ordering of the objects. It is understandable that "first\second\..." can be interchanged with a specific order or sequence where permitted, so that the embodiments of the present application described herein can be implemented in an order other than that illustrated or described herein.

[0031] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by those skilled in the art to which this application pertains. The terms used herein are for the purpose of describing the embodiments of this application only and are not intended to limit this application.

[0032] Before further describing the embodiments of the present application in detail, the nouns and terms involved in the embodiments of the present application are explained. The nouns and terms involved in the embodiments of the present application are subject to the following interpretations.

[0033] 1) Learning to Rank (LTR): LTR is a machine learning method that focuses on modeling the ranking order of items (such as web pages, advertisements, and products). Unlike traditional classification or regression tasks, LTR directly optimizes the ranking order rather than predicting the probabilities of one or more labels. It has widespread applications in recommender systems, information retrieval, search engines, and e-commerce.

[0034] 2) Long Short-Term Memory (LSTM): LSTM is a special type of recurrent neural network (RNN) designed to address the vanishing or exploding gradient problems encountered by traditional RNNs when dealing with long-distance dependencies. Its ability to process sequential data with long-term dependencies makes LSTM a very useful tool.

[0035] 3) Graph Neural Network (GNN): GNN is a deep learning model specifically designed for processing graph-structured data. Since graph-structured data is common in many application areas, such as social networks, recommendation systems, bioinformatics, and transportation networks, GNN models have been widely studied and applied.

[0036] 4) Stochastic Graph Generation: Stochastic graph generation uses stochastic processes to create graph-structured data. In this approach, the nodes and edges of the graph are not generated by explicitly defining relationships, but rather by simulating random interactions between nodes. Such stochastic processes can capture the dynamic properties of many complex systems in the real world, such as link formation in social networks and protein interactions in biological networks.

[0037] Information recommendation technology involves a variety of background technologies. Its main purpose is to accurately predict and recommend information that users may be interested in through data analysis and machine learning methods, thereby increasing information click-through rate and return on investment. The background technologies for information recommendation include the following main aspects:

[0038] (1) Data collection and processing:

[0039] User behavior data collection: including user browsing history, search history, click behavior, etc., achieved through cookies or other technologies; advertising content data collection: collecting advertising text, images, videos and other content data; multimodal data fusion: integrating data from different media (text, images, videos) to improve recommendation accuracy and personalization.

[0040] (2) Feature extraction and representation:

[0041] Text feature extraction: Use natural language processing techniques such as the bag-of-words model and word embedding to convert text information into feature vectors that can be used for machine learning. Image and video feature extraction: Utilize deep learning methods such as convolutional neural networks (CNN) to extract key features from advertising images and videos. User behavior feature extraction: Analyze user historical behavior patterns to extract user interests, preferences, and behavioral characteristics.

[0042] (3) Recommendation model establishment:

[0043] Collaborative filtering: Recommends ads that interest similar users based on their historical behavior and the behavior of similar users; Content filtering: Recommends ads related to user interests based on the text content, image, or video features of the ads; Hybrid recommendation: Combines collaborative filtering and content filtering, leveraging multiple features and algorithms to improve recommendation effectiveness.

[0044] (4) Personalized optimization and real-time recommendation:

[0045] Personalized optimization: Dynamically adjust recommendation strategies based on users' real-time behavior and feedback to improve personalized recommendation effects; Real-time recommendation: Through real-time processing and response, quickly adjust recommendation results to adapt to users' instantly changing needs and interests.

[0046] (5) Evaluation and optimization:

[0047] Recommendation system evaluation: Use indicators such as click-through rate (CTR) and conversion rate to evaluate recommendation effectiveness and optimize recommendation algorithms and model parameters; A / B testing: Use experimental methods such as A / B testing to verify the effectiveness of new algorithms or strategies and further optimize the recommendation system.

[0048] (6) Privacy protection and compliance:

[0049] Data privacy protection: comply with relevant laws and regulations to protect the security and legal use of user privacy data; transparency and explainability: improve the transparency of the recommendation system and explain the basis and principles of the recommendation to users.

[0050] In the solutions provided by related technologies, it is difficult to effectively integrate user behavior data, advertising content data and other auxiliary data (such as social data, video data) in information recommendation scenarios, resulting in the recommendation system being unable to fully utilize the available information. In addition, there is noise in the label data, and there is a hierarchical relationship between the labels. Traditional models find it difficult to handle these complex relationships. There are few data samples for some recommendation types, making it difficult to train an effective recommendation model.

[0051] In view of this, embodiments of the present application provide an information recommendation method, apparatus, electronic device, computer-readable storage medium, and computer program product that can effectively improve the accuracy of information recommendations. The electronic device provided in embodiments of the present application can be implemented as a server, or can be implemented collaboratively by a server and a terminal. The following description uses the information recommendation method provided in embodiments of the present application, implemented collaboratively by a server and a terminal, as an example.

[0052] For example, see Figure 1 , Figure 1 This is a schematic diagram of the architecture of the information recommendation system 100 provided in the embodiment of the present application, which is used to support an information recommendation application, such as Figure 1 As shown, the information recommendation system 100 includes: a server 200, a network 300, a terminal 400 and a database 600. The terminal 400 is connected to the server 200 via the network 300, and the server 200 is connected to the database 600. The network 300 can be a local area network or a wide area network, or a combination of the two; the database 600 is used to store structured data (such as user behavior data, recommendation information, multiple modal data of users, and other data).

[0053] In some embodiments, a user initiates an information browsing operation through terminal 400, and terminal 400 transmits the information browsing operation to server 200 via network 300. Server 200 extracts first recommendation information from the first data of the object from database 600, and constructs graph data based on the first recommendation information in the first data of the object, wherein the graph data is used to determine the recommendation probability of different types of recommendation information; at the same time, server 200 also extracts second data of the object from database 600, and performs feature extraction on the second data of the object to obtain multiple modal features of the object; then, server 200 fuses the multiple modal features to obtain a first fused feature, wherein the first fused feature is used to classify the recommendation information; then, server 200 generates a recommendation information sequence for the object based on the first fused feature and graph data, and uses the recommendation information in the recommendation information sequence as the second recommendation information sent to the object; finally, server 200 selects recommendation information for the user from the second recommendation information, and returns it to terminal 400 via network 300 for display on terminal 400, that is, displays the corresponding recommendation information (such as advertisement) to the user.

[0054] In other embodiments, the embodiments of the present application can also be implemented with the help of cloud technology. Cloud technology refers to a hosting technology that unifies a series of resources such as hardware, software, and network within a wide area network or local area network to realize data calculation, storage, processing, and sharing.

[0055] Cloud technology is a general term for network, information, integration, management platform, and application technologies used in the cloud computing business model. It can form a resource pool that can be used flexibly and conveniently on demand. Cloud computing technology will become a key support. The backend services of technical network systems require a large amount of computing and storage resources.

[0056] For example, Figure 1 The server 200 in the example can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms. The terminal 400 can be a smart phone, tablet computer, laptop computer, desktop computer, smart speaker, smart watch, car terminal, etc., but is not limited to these. The terminal 400 and the server 200 can be directly or indirectly connected via wired or wireless communication, which is not limited in the embodiments of the present application.

[0057] It should be noted that the information recommendation method provided in the embodiment of the present application can be applied to different information recommendation application scenarios such as advertising recommendation, news recommendation, game recommendation, and entertainment information recommendation, and can effectively improve the accuracy of information recommendation.

[0058] The following continues to describe the structure of the electronic device provided in the embodiment of the present application. Take the electronic device as an example, see Figure 2 , Figure 2 is a structural diagram of an electronic device 500 provided in an embodiment of the present application, Figure 2 The electronic device 500 shown includes: at least one processor 510, a memory 540, and at least one network interface 520. The various components in the electronic device 500 are coupled together via a bus system 530. It is understood that the bus system 530 is used to achieve connection and communication between these components. In addition to the data bus, the bus system 530 also includes a power bus, a control bus, and a status signal bus. However, for the sake of clarity, the bus system 530 is not described in detail. Figure 2 Various buses are labeled as bus system 530 .

[0059] The processor 510 can be an integrated circuit chip with signal processing capabilities, such as a general-purpose processor, a digital signal processor (DSP), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc., where the general-purpose processor can be a microprocessor or any conventional processor, etc.

[0060] The memory 540 may be removable, non-removable, or a combination thereof. Exemplary hardware devices include solid-state memory, hard drives, optical drives, etc. The memory 540 may optionally include one or more storage devices that are physically remote from the processor 510.

[0061] The memory 540 includes volatile memory or non-volatile memory, or may include both volatile and non-volatile memory. The non-volatile memory may be a read-only memory (ROM), and the volatile memory may be a random access memory (RAM). The memory 540 described in the embodiments of the present application is intended to include any suitable type of memory.

[0062] In some embodiments, the memory 540 can store data to support various operations, examples of which include programs, modules, and data structures, or a subset or superset thereof, as exemplified below.

[0063] Operating system 541, including system programs for processing various basic system services and performing hardware-related tasks, such as the framework layer, core library layer, and driver layer, which are used to implement various basic services and process hardware-based tasks;

[0064] A network communication module 542 for reaching other computing devices via one or more (wired or wireless) network interfaces 520 , exemplary network interfaces 520 including Bluetooth, Wireless LAN (WiFi), and Universal Serial Bus (USB);

[0065] In some embodiments, the apparatus provided in the embodiments of the present application may be implemented in software. Figure 2 The information recommendation device 543 stored in the memory 540 is shown. It can be software in the form of a program or plug-in, and includes the following software modules: a construction module 5431, a processing module 5432, a recommendation module 5433, a training module 5434, an expansion module 5435, and an interpretation module 5436. These modules are logical and can be arbitrarily combined or further divided according to the functions implemented. It should be noted that in Figure 2For the sake of convenience, all the above modules are shown at once, but this should not be considered as excluding the implementation of the information recommendation device 543 that only includes the construction module 5431, the processing module 5432 and the recommendation module 5433. The functions of each module will be explained below.

[0066] The information recommendation method provided in the embodiment of the present application will be specifically described below in combination with the exemplary application and implementation of the server provided in the embodiment of the present application.

[0067] See also Figure 3 , Figure 3 This is a flow chart of the information recommendation method provided by the embodiment of the present application, which will be combined with Figure 3 The steps shown are explained.

[0068] In step 101 , graph data is constructed based on first recommendation information in first data of an object.

[0069] Here, graph data is used to determine the recommendation probabilities of different types of recommendation information.

[0070] In some embodiments, first, first recommendation information (e.g., advertisements that a user has clicked on or viewed by a user) in first data of an object (e.g., user behavior data, including user browsing data, purchase data, etc.) is obtained, where the first data of the object can be the object's behavior (e.g., click, purchase, rating, etc.), object information (e.g., product, content, etc.), and timestamp, etc., and the first recommendation information can be recommendation information related to the object's behavior (e.g., product information that a user clicks on and views); then, preprocessing operations such as cleaning and formatting are performed on the collected first data to ensure data quality. Here, the preprocessing operations can be removing invalid data, processing missing values, standardizing data formats, etc.; then, in a graph data model, entities and relationships between entities are defined, for example, entities can be users, product information (e.g., advertisements for recommending the product), types, brands, etc., which can be used as nodes in the graph data, and relationships between entities can be users' behaviors such as clicks, purchases, and evaluations of product information, as well as associations between product information, which can be used as edges between nodes; finally, graph data is constructed based on the definition of the relationships between entities, for example, entities such as users and product information are mapped as nodes in the graph, and relationships between user behaviors and product information are mapped as edges between nodes.

[0071] For example, suppose there is user-related data on an e-commerce platform, including users, product information, and users' purchasing behavior. First, define entities and the relationships between entities, where entities are users and product information (such as advertisements used to recommend the product), and the relationship between entities is purchase; then, create user and product information nodes. For example, user 1 and product information 1 can be used as nodes. When user 1 purchases the product recommended by product information 1, an edge from user 1 to product information 1 can be created; in addition, more user and product information, as well as the purchasing relationships between them, can be added, such as user ratings of product information, product information types, etc.

[0072] In some embodiments, the above-mentioned construction of graph data based on the first recommendation information in the first data of the object can be achieved in the following way: taking the object and the first recommendation information as the first node to construct the first-level graph data, and taking the first recommendation information and the type of the first item as the second node to construct the second-level graph data, wherein the first item is the item recommended by the first recommendation information.

[0073] It should be noted that the graph data may include first-level graph data, such as a user-advertisement graph; and may also include second-level graph data, such as an advertisement-molecule graph, where a molecule may refer to the type of product related to the advertisement.

[0074] For example, suppose that in an e-commerce platform, the nodes of the first-level graph data can be users and product information; then, the interactive behavior data between users and product information is obtained, such as interactive behaviors such as clicks, purchases, and ratings; then, based on the interactive behavior data, users and product information with interactive behaviors are used as nodes, and edges are added between the user and product information nodes, where the type of edge represents the interactive behavior data between users and product information. For example, suppose user 1 clicks on product information 1, that is, a click-type edge is added between user 1 and product information 1, thereby constructing the first-level graph data; similarly, in the e-commerce platform, the second-level The nodes of the graph data can be the types of product information in the nodes representing the first-level graph data; then, the association relationship data between the product information and the types of product information is obtained, such as the similarity between the product information, the correlation between the types of product information, etc.; then, according to the association relationship data, the object types with association relationships are used as nodes, and edges are added between nodes of different object types. The type of edge represents the association relationship between different nodes. For example, if product information 1 and product information 2 both belong to the type electronic products, an edge connecting product information 1 and product information 2 is added under the electronic product node, thereby constructing the second-level graph data.

[0075] In step 102, feature extraction is performed on the second data of the object to obtain multiple modal features of the object.

[0076] Here, the second data refers to data of multiple modalities.

[0077] In some embodiments, first, data of different modalities of the object are obtained, such as text data, image data, audio data, and video data; then, the data of each modality are preprocessed, such as cleaning, standardization, normalization, segmentation, etc.; finally, the corresponding feature extraction method is applied to the data of each modality to obtain the features of the current modality.

[0078] For example, assuming that the multimedia content analysis system includes data in two modalities, text modality and image modality, first, the text data of the news article (i.e., text modal data) and the image data corresponding to the news article (i.e., image modal data) are obtained; then, the text data of the news article is preprocessed by word segmentation, stop word removal, part-of-speech tagging, etc., and the image data corresponding to the news article is preprocessed by resizing, grayscale, denoising, etc.; finally, the preprocessed data is subjected to feature extraction operations to obtain corresponding modal features, for example, the bag of words model (Bag of Words), term frequency-inverse document frequency (TF-IDF), word vector model (Word2Vec), etc. are used to extract text features from the text data of the news article, and the image features from the image data corresponding to the news article are extracted by using convolutional neural networks (CNN), color histograms, edge detection, etc.

[0079] In step 103, multiple modal features are fused to obtain a first fused feature.

[0080] Here, the first fused feature is used to classify the recommendation information.

[0081] In some embodiments, see Figure 4 , Figure 4 This is a flow chart of the information recommendation method provided in the embodiment of the present application. Figure 4 As shown, Figure 3 Step 103 shown can be performed by Figure 4 Steps 1031 to 1033 shown are implemented by combining Figure 4 The steps shown are explained.

[0082] In step 1031, dimensionality reduction processing is performed on the multiple modal features to obtain multiple dimensionality reduction features.

[0083] Here, one dimensionality reduction feature corresponds to one modal feature.

[0084] In some embodiments, first, the features of each modality are preprocessed, such as standardization, normalization, noise removal, etc., to ensure data quality; then, feature dimensionality reduction techniques (such as principal component analysis, autoencoder, etc.) are used to reduce the dimension of the features; finally, feature encoding techniques (such as hash coding, sparse coding, quantization, etc.) are used to convert the reduced features into a more compact representation.

[0085] For example, assuming that a multimodal music recognition system includes two modal features, audio modal features and text modal features, first, the audio modal features of the music (such as spectrum, Mel-frequency cepstral coefficients, etc.) and the text modal features in the music lyrics (such as bag-of-words model, etc.) are obtained; then, the Mel-frequency cepstral coefficients of the audio modality are reduced in dimension using principal component analysis, and the word frequency-inverse document frequency features of the text modality are reduced in dimension using t-distributed stochastic neighbor (t-SNE) embedding; finally, the reduced-dimensional audio features are processed using hash coding, and the reduced-dimensional text features are processed using quantization coding to obtain the reduced-dimensional features of the corresponding modality.

[0086] In step 1032, attention processing is performed on the multiple dimensionality reduction features to obtain the weight of each dimensionality reduction feature.

[0087] Here, an attention mechanism (such as a self-attention mechanism, a multi-head attention mechanism, etc.) can be used to determine the first weight of each dimensionality reduction feature.

[0088] For example, assuming there are five dimensionality reduction features F1, F2, F3, F4 and F5, the weight of each dimensionality reduction feature is obtained through attention calculation. For example, the weight of the dimensionality reduction feature F1 is 0.3, the weight of the dimensionality reduction feature F2 is 0.2, the weight of the dimensionality reduction feature F3 is 0.2, the weight of the dimensionality reduction feature F4 is 0.15, and the weight of the dimensionality reduction feature F5 is 0.15.

[0089] In step 1033, based on the weight of each dimensionality reduction feature, a weighted sum is performed on the multiple dimensionality reduction features, and the weighted sum result is used as the first fusion feature.

[0090] In some embodiments, the above-mentioned step 103 can also be implemented in the following manner: performing dimensionality reduction processing on multiple modal features to obtain multiple dimensionality reduction features; performing attention processing on the multiple dimensionality reduction features to obtain the weight of each dimensionality reduction feature, and based on the weight of each dimensionality reduction feature, performing weighted summation on the multiple dimensionality reduction features to obtain a weighted summation result; dividing the weighted summation result into multiple sub-features, and determining the weight of each sub-feature; based on the weight of each sub-feature, fusing the multiple sub-features to obtain a first fused feature.

[0091] In some embodiments, first, according to the execution process of the above steps 1031 to 1033, the weighted summation result of the dimensionality reduction features is obtained, where the weight of each dimensionality reduction feature can be based on the importance of the feature, information gain, model coefficient, etc., and each dimensionality reduction feature is weighted according to the weight of the dimensionality reduction feature. This can be achieved by multiplying the vector of each feature with the corresponding weight vector, and all weighted feature vectors are added to obtain a comprehensive fusion feature vector, i.e., the weighted summation result; then, the weighted summation result is divided into multiple sub-features, where different segmentation strategies can be used, such as proportional segmentation, segmentation by feature type, etc., and the weight of each sub-feature is determined, where it can be based on the uniqueness of the sub-feature, the contribution to the overall feature, etc.; finally, each sub-feature is weighted according to the weight of the sub-feature, and all weighted sub-feature vectors are added to obtain a comprehensive fusion feature vector, i.e., the first fusion feature.

[0092] For example, assume that three dimensionality reduction features F1, F2, and F3 are obtained by dimensionality reduction processing of multimodal data. The weights of each dimensionality reduction feature are W1, W2, and W3 respectively. F1, F2, and F3 are weighted summed to obtain the weighted summation result F', which is W1*F1+W2*F2+W3*F3; then, the weighted summation result F' is divided into two sub-features F'1 and F'2, and the weights corresponding to the two sub-features are determined to be W'1 and W'2 respectively; finally, the two sub-features F'1 and F'2 are weighted fused to obtain the first fused feature F" as W'1*F'1+W'2*F'2. In this way, the number of features can be reduced while retaining the importance of the features, making the feature representation more efficient.

[0093] In step 104 , a recommendation information sequence for the object is generated based on the first fused features and the graph data, and the recommendation information in the recommendation information sequence is used as second recommendation information sent to the object.

[0094] In some embodiments, see Figure 5 , Figure 5 This is a flow chart of the information recommendation method provided in the embodiment of the present application. Figure 5 As shown, Figure 3 In step 104, the recommendation information sequence for the object is generated based on the first fusion feature and the graph data, which can be achieved by Figure 5 Steps 1041 to 1044 shown are implemented by combining Figure 5 The steps shown are explained.

[0095] In step 1041 , the recommendation information in the recommendation information database is classified according to the first fusion feature to obtain multiple recommendation information sets.

[0096] For example, suppose an e-commerce platform analyzes the user's browsing history, purchase history, favorite products and other behavioral data, and concludes that a user's interests are concentrated in "sports equipment" and "electronic products". Here, "sports equipment" and "electronic products" constitute the first fusion feature related to the user's interests; the e-commerce platform's recommendation information library contains a variety of product recommendation information, such as basketball, football, sports shoes, mobile phones, tablets, headphones, etc.; then, by filtering out the recommendation information related to sports equipment in the recommendation information library, a recommendation information set is formed. This set may contain recommendation information for products such as basketballs, footballs, sports shoes of different brands and models, such as "basketball shoes", "basketball", etc., and the recommendation information related to electronic products is classified into another recommendation information set, such as "mobile phones", "computers", etc.

[0097] In another example, by analyzing the user's reading history, likes, comments and other behaviors, it was found that a certain user was more interested in "technology news" and "sports news". These two aspects are the first fusion features related to the user's interests; the recommended information library of the news information platform contains recommended information on news articles from various fields, such as technology, sports, entertainment, finance, etc.; then, the news recommendation information in the field of technology is filtered out from the recommended information library to obtain a recommended information set, such as news recommendation information including "artificial intelligence technology application" and "tram news release", and the sports-related news recommendation information is classified into another recommended information set, such as "football news" and "basketball news".

[0098] In some embodiments, classification processing is implemented through a classification network, and probability prediction is implemented through a graph neural network. Before executing the above step 1041, the following processing can also be performed: obtaining a sample data set, the sample data set includes graph data samples and multiple modal data samples; training the initialized graph neural network based on the graph data samples to obtain a trained graph neural network; training the initialized classification network based on multiple modal data samples to obtain a trained classification network.

[0099] In some embodiments, first, a graph data sample (such as a social network graph, a knowledge graph, etc.) and multiple modal data samples (such as text, image, audio, etc.) are obtained; then, the graph data sample and the multiple modal data samples are preprocessed respectively, such as performing node feature extraction, edge attribute processing, graph structure normalization and other operations on the graph data sample, and performing text segmentation, image size adjustment, audio denoising and other operations on the data of each modality; then, the initialized model framework is determined, such as selecting Graph Neural Networks (GCN) or Graph Attention Networks (GAT) as the initialized graph neural network, and selecting Convolutional Neural Networks (CNN) or Recurrent Neural Networks (RNN) Network, RNN) as the initialized classification network; finally, the graph neural network and the classification network are trained respectively according to the sample data to obtain the trained model. For example, the graph data samples are divided into training set, validation set and test set, the training set is trained with the graph neural network, and the parameters of the graph neural network (such as learning rate, number of layers, etc.) are adjusted using the validation set, and the performance of the trained graph neural network is evaluated using the validation set; multiple modal data samples are divided into training set, validation set and test set, the training set is trained with the classification network, and the parameters of the classification network (such as number of iterations, etc.) are adjusted using the validation set, and the performance of the trained classification network is evaluated using the validation set.

[0100] For example, assume that in a social network recommendation system, first, a sample data set is obtained, where the sample data set includes a social network graph and advertising data of multiple modalities; then, the social network graph is preprocessed to extract user features, such as user activity and participation, and the advertising data of multiple modalities are preprocessed to extract advertising features, such as advertising type and price; then, GCN is selected as the initialized graph neural network, and CNN is selected as the initialized classification network; finally, the graph data is divided into training set, validation set and test set, the training set is trained using the GCN model, the parameters are adjusted using the validation set, and the performance of the GCN model is evaluated using the test set to obtain the trained graph neural network; at the same time, the advertising data of multiple modalities are divided into training set, validation set and test set, the training set is trained using the CNN model, the parameters are adjusted using the validation set, and the performance of the CNN model is evaluated using the test set to obtain the trained classification network.

[0101] In some embodiments, before executing the above-mentioned training of the initialized classification network based on the graph data samples to obtain the trained classification network, the following processing can also be performed: statistics are performed on the data volume of each type of data sample; for the type with a data volume less than a data volume threshold, a first data sample of the type is obtained; the first data sample is masked by a pre-trained model to obtain a mask template of the first data sample; based on the mask template of the first data sample, a generation process is performed by a language model to obtain a second data sample; the second data sample is added to multiple modal data samples to obtain multiple expanded modal data samples.

[0102] In some embodiments, first, all types contained in the data set are identified, and the number of data samples in each type is counted; then, a data volume threshold is set, where the data volume threshold can be a preset value or dynamically set according to the size and type distribution of the data set, and is not specifically limited here; then, types with data volumes less than the threshold are screened out, and a first data sample of the current type is obtained; then, a pre-trained model (such as BERT, GPT, etc.) is used to mask the first data sample to obtain a mask template of the first data sample, where the mask template of the first data sample refers to replacing certain parts of the first data sample with a mask symbol [MASK]; then, a second data sample is generated according to the mask template through a language model; finally, the second data sample is added to multiple modal data samples to expand the types with insufficient data volume.

[0103] For example, suppose there is a dataset containing types A, B, and C. Statistics show that type A has 1,000 samples, type B has 500 samples, and type C has 200 samples. The set data volume threshold is 500. Types A and B both meet the threshold requirement and do not need to be processed. However, the data volume of type C is less than the data volume threshold. In this case, data samples from type C are obtained, such as "I walk every day for exercise." The BERT model is used to mask the data samples of type C, replacing "walking" with the [MASK] symbol. The BERT model is used to generate a second data sample based on the mask template, such as "I run every day for exercise." Finally, "I run every day for exercise" is added to type C, thereby expanding type C, which has insufficient data. In this way, by performing data augmentation on types with insufficient data, the generalization and robustness of the model can be effectively improved.

[0104] In step 1042 , feature extraction is performed on the first node in the graph data of the first level to obtain first node features, and feature extraction is performed on the second node in the graph data of the second level to obtain second node features.

[0105] For example, first, feature extraction is performed on the first node in the first-level graph data to obtain the first node feature, and feature extraction is performed on the second node in the second-level graph data to obtain the second node feature. These node features can be node attributes, structural information (such as the number of adjacent nodes), or other node-related information.

[0106] In step 1043 , the first node feature and the second node feature are fused to obtain a second fused feature, and probability prediction is performed based on the second fused feature to obtain recommendation probabilities of different types of recommendation information.

[0107] In some embodiments, a graph neural network can be used to learn the weights of node features in each level through training. Here, the graph neural network can automatically learn the weights of node features, and these weights reflect the importance of node features in the prediction task; then, the node features in each level are weighted fused according to the weights. Here, this can be achieved by multiplying the vector of each feature with the corresponding weight vector, or by using the aggregation function in the graph neural network (such as sum, average, maximum, etc.) to aggregate the feature information of adjacent nodes to obtain a second fused feature; finally, a prediction model is constructed using the graph neural network, and the second fused feature is used as the input of the model. The model performs probability prediction based on the input second fused feature to obtain the recommendation probability of different types of recommendation information. Here, the graph neural network can be a classifier, a regressor or other prediction model.

[0108] For example, assume the first node feature is A and the second node feature is B. Then, using graph neural network training, the weight of each node feature is automatically learned. For example, the weight of the first node feature A is 0.6, and the weight of the second node feature B is 0.4. Subsequently, the first and second node features are weighted and fused to obtain the second fused feature 0.6*A+0.4*B. Finally, using the trained graph neural network, the second fused feature is used as the model input for click prediction, and the output is the recommendation probability of different types of recommendation information. In this way, extracting and fusing node features from graph data through graph neural networks can effectively improve the accuracy of model predictions.

[0109] In step 1044 , a recommendation information sequence for the object is generated based on the multiple recommendation information sets and the recommendation probabilities of different types of recommendation information.

[0110] In some embodiments, the above-mentioned step 1044 can be implemented in the following manner: taking the type with a recommendation probability greater than or equal to the recommendation probability threshold as the type to be recommended; screening out a first recommendation information set from multiple recommendation information sets, and generating a recommendation information sequence for the object based on the recommendation information included in the first recommendation information set, wherein the first recommendation information set is a recommendation information set that meets the type to be recommended.

[0111] In some embodiments, first, the recommendation probability of the object for different types of recommendation information is determined; then, a recommendation probability threshold is set, where the recommendation probability threshold can be a preset value or a value dynamically determined according to different methods such as business needs and expert experience, and is not specifically limited here; then, the recommendation probability of the recommendation information of each recommendation type is compared with the recommendation probability threshold, and the recommendation information type with a recommendation probability greater than or equal to the recommendation probability threshold is retained to obtain the type to be recommended; finally, the multiple recommendation information sets obtained by the classification network processing are screened to obtain a first recommendation information set, and a recommendation information sequence for the object is generated based on the recommendation information included in the first recommendation information set.

[0112] For example, assuming that in an online shopping platform, the product information to be recommended to the user is determined, first, according to the recommendation probabilities of different types of recommendation information obtained by graph neural network processing, it can be determined that the user's recommendation probability for electronic products is 0.6, the recommendation probability for sports equipment is 0.4, and the recommendation probability for food is 0.8; then, the recommendation probability threshold is determined to be 0.5 according to the recommendation probability requirements; then, the recommendation probability of electronic products, sports equipment and videos for the user is compared with the recommendation probability threshold respectively, and the types with recommendation probabilities greater than or equal to the recommendation probability threshold are electronic products and food; finally, the multiple recommendation information sets obtained by the classification network processing are screened to obtain a first recommendation information set corresponding to electronic products and food, and a recommendation information sequence for the user is generated based on the recommendation information included in the first recommendation information set.

[0113] In some embodiments, the above-mentioned generation of a recommendation information sequence for an object based on the recommendation information included in the first recommendation information set may also perform the following processing: determining the degree of match between the recommendation information included in the first recommendation information set and the first recommendation information; sorting the recommendation information included in the first recommendation information set in descending order according to the degree of match from high to low to obtain a sorting result; and using the recommendation information sequence composed of the recommendation information ranked higher in the sorting result as the recommendation information sequence for the object.

[0114] In some embodiments, first, the matching degree between the recommendation information included in the first recommendation information set and the first recommendation information is determined (for example, Pearson correlation coefficient, similarity calculation, etc.); then, the recommendation information included in the first recommendation information set is sorted in descending order according to the matching degree to obtain a recommendation information sequence for the object.

[0115] For example, suppose that in a movie recommendation system, first, the Pearson coefficient is selected as the matching indicator; then, the first recommendation information set includes 5 candidate movies, movie A, movie B, movie C, movie D and movie F; then, the matching degrees between the 5 candidate movies and the first recommendation information (movie M) are calculated respectively, and it can be obtained that the matching degree of movie A is 0.6, the matching degree of movie B is 0.8, the matching degree of movie C is 0.5, the matching degree of movie D is 0.7, and the matching degree of movie F is 0.9; then, the candidate movies are sorted according to the calculated matching degrees, and the resulting movie recommendation sequence is [movie F, movie B, movie D, movie A, movie C].

[0116] In some embodiments, after executing step 104 , the following processing may also be performed: determining the tag type of the second recommendation information; and generating an explanation text of the second recommendation information based on the tag type and the first data.

[0117] In some embodiments, first, business needs are analyzed to determine the tag types contained in the recommendation information, such as product information type, user interest points, usage scenarios, etc.; then, first data (such as user browsing history, purchase history, search history, etc.) and the tag type of the second recommendation information are collected; then, an association rule mining algorithm (such as Apriori, Eclat, FP-growth, etc.) is used to analyze the relationship between the behavioral data and the second recommendation information tag, and the similarity between the behavioral data and the second recommendation information tag is calculated (such as cosine similarity, etc.); then, a template for the explanatory text is constructed based on the tag type and similarity, and the corresponding explanatory text is generated through the explanatory text template based on the tag type and behavioral data; finally, the explanatory text is optimized to improve the quality of the explanatory text.

[0118] For example, assume that in an online shopping website, the label types of product information include product information type, price range, brand, etc. First, collect user A's behavioral data, including user A recently browsing sports shoes, user A recently purchased sportswear, and user A recently searched for "running" related product information, and obtain relevant product information labels, including recommended product information of sports shoes, price range of 100-200 yuan, and brand "Brand A"; then, use the Apriori algorithm to analyze the relationship between user A's behavioral data and recommended product information labels, and calculate that the similarity between user A's behavioral data and the "sports shoes" label is 0.8; then, A template for the explanatory text is constructed based on the tag type and similarity: "Based on your recent behavior, we recommend you the following product information: {product information type}, the price range is {price range} yuan, and the brand is {brand}"; then, based on user A's behavioral data and the template of the explanatory text, the generated explanatory text is "Based on your recent behavior, we recommend you the following product information: sports shoes, the price range is 100-200 yuan, and the brand is brand A"; finally, the explanatory text is optimized, and the optimized explanatory text is "We noticed that you have recently been interested in sports-related product information, so we recommend brand A's sports shoes, with a price range of 100-200 yuan."

[0119] The following describes an exemplary application of the embodiment of the present application in a practical application scenario, which describes the specific implementation process of the information recommendation method in an advertisement recommendation scenario.

[0120] Related technologies consume significant computing resources when calculating the similarity between the eigenvalues ​​of multimodal data fusion and the eigenvalues ​​of multiple ad vectors. This can lead to low recommendation efficiency, especially in large-scale datasets and high-dimensional feature spaces. Furthermore, current ad recommendation scenarios struggle to effectively integrate user behavior data, ad content data, and other auxiliary data (such as social data and video data), preventing recommendation systems from fully utilizing all available information. Furthermore, label data contains noise, and labels have hierarchical relationships. Traditional models struggle to handle these complex relationships, and data samples for certain ad types are limited, making it difficult to train effective recommendation models. In view of this, this application optimizes the advertising recommendation effect through a multi-level, multi-stream architecture and graph neural network (GNN) model. First, user data and advertising data are preprocessed through standardization, and text data is encoded using pre-trained language models such as BERT to construct user-advertising graphs and advertising-molecule graphs. Node embeddings are obtained through random walks and GNNs to further extract user and advertising features. Then, an LSTM sequence model is used to capture the time series dependency between user behavior and advertising. At the same time, content and social data are combined in the graph data to perform comprehensive reasoning of cross-level attention mechanisms. A multi-stream architecture is used to connect video, text, and new position semantic streams. Relevant information is amplified through proxy attention fusion and token fusion to achieve hierarchical classification of HTC label-aware representation. Moreover, during the reasoning process, knowledge is acquired from the expert model by propagating uncertain tokens, and mask templates are mined and filled using LLM-based logits to generate a few types of text. Afterwards, candidate advertisements to be recommended are determined based on the reasoning results of different structures. Finally, the candidate advertisements are ranked by Learning to Rank, a recommendation list is generated based on user interests and advertising relevance, and explanatory text is generated to help users understand the reasons for the recommendation. In this way, through flexible model combination and multi-level information fusion, the accuracy of advertising recommendations and user satisfaction are effectively improved, and the recommendation model has stronger explanatory power and adaptability.

[0121] In some embodiments, see Figure 6 , Figure 6 This is a schematic diagram of the principle of the information recommendation method provided by the embodiment of the present application. Figure 6 Describe the specific implementation process.

[0122] like Figure 6As shown, the information recommendation method provided by this application may include seven stages: data preparation, data preprocessing, feature extraction, model construction, model reasoning, recommendation result generation, and recommendation result interpretation. In the data preparation stage, user data (i.e., first data) and data from various sources such as advertising data, text data, and molecular data (i.e., second data) are obtained, wherein user data includes user's historical clicks, browsing, purchase records, social media interactions and other data; advertising data (i.e., first recommendation information) includes data such as the content, type, delivery time, and target audience of the advertisement; text data includes data such as advertisement text description and user comments; and molecular data includes data such as product ingredient information related to the advertisement; finally, data from different sources are integrated to form a complete data set.

[0123] In the data preprocessing stage, word segmentation, stop word removal, stemming, and pre-trained models such as BERT can be used to vectorize words in text data. Based on user data, advertising data, and molecule data, multi-layer graph data such as user-advertising graph (i.e., first-level graph data) and advertising-molecule graph (i.e., second-level graph data) can be constructed. First, click, browse, purchase and other data are extracted from user behavior logs, and user comments are segmented, stop words are removed, and word vectorization is performed to encode the user's historical behavior into a feature vector. Secondly, the advertising text is preprocessed and encoded using pre-trained models such as BERT, and the structural information of the advertisement (such as type, target audience, etc.) is encoded into a feature vector. Thirdly, a user-advertising graph is constructed, with nodes representing users and advertisements and edges representing interactive behaviors. An advertising-molecule graph is constructed, with nodes representing advertisements and molecules and edges representing association relationships.

[0124] During the feature extraction phase, methods like node2vec can be used to simulate random walks to process multi-layer graph data, generating more expressive graph representations. Long-term dependencies can be captured using LSTMs and GRUs, combined with the BERT pre-trained model to capture semantic information in text sequences. First, the BERT pre-trained model is used to encode user comments and extract user preferences. A GNN graph representation learning method is then used to extract user embeddings from the user-advertisement graph. Second, the BERT pre-trained model is used to encode ad text and extract ad semantic features. A graph representation learning method is then used to extract ad embeddings from the ad-molecule graph.

[0125] During the model construction stage, a GNN model can be built based on multi-layer graph data, integrating content and social data to construct a multi-level relationship graph. The Transformer model can be combined to obtain the global dependency of text and graph, connect video, text, and position semantic streams (i.e., multiple modalities), and perform cross-stream agent attention fusion. In addition, during the model construction stage, the loss function needs to be designed. In this application, multiple loss functions such as structural similarity preservation loss, masked language modeling loss, hierarchical classification loss, divergent contrastive learning loss, and uncertainty propagation loss are used.

[0126] Here, in the structural similarity preserving loss, the relationship between molecule-text pairs is enhanced by the structural similarity preserving loss. The molecular data and the corresponding text description should maintain a high similarity to capture the intrinsic connection between product ingredients and descriptions. The calculation of the structural similarity preserving loss L1 can be achieved by formula (1).

[0127] L1=∑ i,j y i,j *d(z i ,z j )+(1-y i,j )*max(0,md(z i ,z j ) (1)

[0128] Among them, y i,j is the label of the molecule-text pair, ranging from 0 (dissimilar) to 1 (similar), d(z i ,z j ) is the Euclidean distance between node embeddings, and m is the length of the graph boundary.

[0129] In the hierarchical classification loss, the hierarchical classification loss is used to capture the hierarchical relationship between multi-level label classifications involved in advertising recommendation, such as advertising types, audience groups, etc. For each level l, the hierarchical classification loss L2 can be calculated using formula (2).

[0130]

[0131] Among them, α1 is the weight of the lth layer, y l is the true label, is the predicted label, It is to calculate the cross entropy between the true label and the predicted label.

[0132] In the masked language modeling loss, the BERT pre-trained language model is used for continuous training, and the masked language modeling loss is used to improve the model's understanding of the advertising text. The calculation of the masked language modeling loss L3 can be achieved through formula (3).

[0133] L3=-∑t∈maskedtokens logP(x t |x \t ) (3)

[0134] Among them, x t is the masked word, x \t For other words.

[0135] In the divergent contrastive learning loss, by minimizing the distance between adjacent semantically similar labels, the model's ability to distinguish similar advertisements can be improved. The calculation of the masked language modeling loss L4 can be achieved through formula (4).

[0136] L4=∑ i,j d(e i ,e j ) (4)

[0137] Among them, d(e i ,e j ) is the Euclidean distance between node embeddings.

[0138] In the uncertainty propagation loss, during the model fine-tuning process, uncertain tokens are propagated to experts to improve the generalization ability of the model. For each uncertain token, the uncertainty propagation loss L5 can be calculated using formula (5).

[0139]

[0140] Among them, w t The weight of the uncertain token.

[0141] Finally, according to formula (6), multiple loss functions are weightedly fused to obtain the final loss function L.

[0142] L=λ1L1+λ2L2+λ3L3+λ4L4+λ5L5 (6)

[0143] Among them, λ i is the weight coefficient of each part of the loss function, which will be adjusted accordingly according to the actual situation.

[0144] In the model inference stage, model inference is performed through a multi-layer graph structure and a multi-stream architecture. In the multi-layer graph structure, content and social data are combined to perform multi-level relational reasoning. The cross-layer attention mechanism is used to integrate information at different levels to predict the results. In the multi-stream architecture, the multi-stream architecture is used to connect the video, text and location semantic streams. Through cross-stream agent attention fusion and Token fusion, the most relevant information in each stream is amplified. First, multi-stream data (i.e., multiple modal data) is obtained, including video streams (including video content watched by users, video feature vector extraction), text streams (including text content read by users, text feature vector extraction) and location semantic streams (including geographic location and semantic information of user interaction with advertisements, location feature vector extraction); then, feature extraction is performed independently on each stream to obtain video features, text features and location features. Each feature vector represents the important information in the stream; then, tokens are generated for each stream. These tokens serve as representative features of the stream. Tokens are represented by attention. The mechanism extracts from the original features, such as video tokens, text tokens, and location tokens, performs cross-stream attention fusion on tokens of different streams, calculates the attention weights between different streams, amplifies the correlation between streams, and calculates the attention weights of video tokens to text tokens and location tokens to capture the information interaction between streams. According to the cross-stream attention weights, tokens of different streams are weightedly fused to form a cross-stream feature vector. The cross-stream feature vector integrates important information from each stream and contains the most relevant information in the video, text, and location streams, which can more comprehensively represent the user's behavior and preferences. In addition, a multi-head attention mechanism can be used to further fuse the features after cross-stream fusion to further extract and amplify the most relevant information in the feature vector; then, a pre-trained language model with continuous training is used to realize HTC label perception representation, and HTC labels are used for hierarchical classification reasoning to capture the hierarchical relationship between labels and realize hierarchical classification of labels.

[0145] In addition, during the model inference process, logits based on a large language model are used to mine and fill mask templates to generate minority type texts and expand the text data. First, the types with fewer data samples in the advertising recommendation system (such as specific advertising types or user groups) are determined; then, the original text data related to the target minority types (such as advertising descriptions, user comments, social media posts, etc.) are collected; then, the ChatGLM pre-trained language model is selected and fine-tuned to adapt to the text data in the specific advertising recommendation field; then, the ChatGLM pre-trained language model is selected and fine-tuned to adapt to the text data in the specific advertising recommendation field; then, logits based on a large language model are used to extract the text data from the original language. High-potential mask templates are mined from the data library. These templates insert mask [MASK] positions in the text to generate possible replacement words or phrases, such as "This [MASK] product is suitable for a healthy lifestyle"; then, using the pre-trained large language model, text belonging to the minority type is generated according to the mask position in the template. The large language model generates high-quality replacement words or phrases based on the context, such as filling the mask in "This [MASK] product is suitable for a healthy lifestyle" with "healthy drinks"; then, based on the optimized mask template and filling results, a large amount of new text data belonging to the target minority type is generated, and the generated minority type text data is added to the original training dataset to increase the sample size of the minority type and balance the data distribution.

[0146] For example, assuming that the target minority type is "healthy product" advertisements, we collect advertisement descriptions and user comments related to health products, and mine the template "This [MASK] product is suitable for a healthy lifestyle" from the corpus. We use the large language model to generate new texts such as "This [organic food] product is suitable for a healthy lifestyle" and "This [vitamin supplement] product is suitable for a healthy lifestyle". We then add the new texts to the original training dataset to expand the training dataset and improve the model's ability to recommend health product advertisements.

[0147] During the recommendation generation phase, Learning to Rank is used to rank candidate ads (recommended information for an object). The final recommendation list (i.e., recommendations for the object) is generated by combining user interest characteristics and ad relevance. During the recommendation explanation phase, explanatory text is generated to help users understand the reasons for the recommendation. This approach uses a multi-layer graph structure and multi-stream architecture, combined with the results of cross-stream agent attention fusion, to provide explanations for recommendations.

[0148] For example, assuming that the user IDs are 101, 102, 103, and 104, the user behavior data includes: User 101: browsed ad A001 (2024-07-01), clicked ad A002 (2024-07-05); User 102: browsed ad A003 (2024-07-03), clicked ad A001 (2024-07-06); User 103: browsed ad A002 (2024-07-02), clicked ad A003 (2024-07-07); User 104: browsed ad A001 (2024-07-04), clicked ad A002 (2024-07-08); User text data includes: User 101: likes technology and game-related content; User 102: is interested in health and fitness advertisements; User 103: pays attention to fashion and beauty products; User 104: Loves traveling and outdoor activities; the advertising data is A001, A002, and A003. The text data of the advertisement includes Advertisement A001: Release of the latest technology products; Advertisement A002: Recommendation of healthy lifestyle; Advertisement A003: Launch of new fashion products; the structural data of the advertisement includes: Advertisement A001: Type: Technology, Target audience: Technology enthusiasts; Advertisement A002: Type: Health, Target audience: Fitness enthusiasts; Advertisement A003: Type: Fashion, Target audience: Fashion experts; the molecular data is M001, M002, and M003; the molecular component embedding includes molecule M001: technology component vector [0.1, 0.2, 0.3, 0.4]; molecule M002: health component vector [0.5, 0.6, 0.7, 0.8]; molecule M003: fashion component vector [0.9, 1.0, 1.1, 1.2].

[0149] Continuing with the above example, feature extraction is performed on the above data, and the user text embedding features are: User 101 text embedding: [0.1, 0.2, 0.3, 0.4]; User 102 text embedding: [0.5, 0.6, 0.7, 0.8]; User 103 text embedding: [0.9, 1.0, 1.1, 1.2]; User 104 text embedding: [1.3, 1.4, 1.5, 1.6]; User behavior features are: User 101 behavior features: [Browse A 001, click A002]; User 102 behavior characteristics: [browse A003, click A001]; User 103 behavior characteristics: [browse A002, click A003]; User 104 behavior characteristics: [browse A001, click A002]; Advertisement text embedding characteristics are Advertisement A001 text embedding: [0.2, 0.3, 0.4, 0.5]; Advertisement A002 text embedding: [0.6, 0.7, 0.8, 0.9]; Advertisement A003 text embedding: [0.1, 0.2, 0.3, 0.4, 0.5]; Advertisement A003 text embedding: [0.2, 0.3, 0.4, 0.5]; Advertisement A003 text embedding: [0.6, 0.7, 0.8, 0.9]; Advertisement A003 text embedding: [0.1, 0.2, 0.3, 0.4, 0.5 ... This embedding: [1.0, 1.1, 1.2, 1.3]; the structural data features of the advertisement are Ad A001 structural data: type: technology, target audience: technology enthusiasts; Ad A002 structural data: type: health, target audience: fitness enthusiasts; Ad A003 structural data: type: fashion, target audience: fashion experts; the constructed user-advertisement graph is user 101-advertisement A001, user 101-advertisement A002, user 102-advertisement A003 03, user 102-advertisement A001, user 103-advertisement A002, user 103-advertisement A003, user 104-advertisement A001, user 104-advertisement A002; the advertisement-molecule graph is advertisement A001-molecule M001, advertisement A002-molecule M002, advertisement A003-molecule M003; the node sequence generated by random walk is user 101-advertisement A001-molecule M001-advertisement A002-user 101.

[0150] Subsequently, continuing with the above example, the user historical behavior features are used as model input through the LSTM model for modeling to obtain user behavior features: User 101 behavior features: [browse A001, click A002]; User 102 behavior features: [browse A003, click A001]; User 103 behavior features: [browse A002, click A003]; User 104 behavior features: [browse A001, click A002]; The user behavior features and advertising features are integrated through a multi-layer graph structure, and the integrated multi-layer graph and node sequence are input into the GNN model for hierarchical training. The cross-level attention mechanism is used for information transmission and fusion, and the user behavior prediction results are output. The video, text, and location semantic streams (i.e., the location information of the advertisement) are connected through a multi-stream architecture. The cross-stream agent attention fusion and token fusion mechanism are used to integrate different modal data to enhance the feature expression capability. At the same time, combined with the HTC label perception representation, the pre-trained language model is used for continuous training to obtain the HTC label perception representation. , through masked language modeling, hierarchical classification and divergent contrast learning objectives, the hierarchical relationship between labels is captured to obtain the label classification results of advertisements. Through uncertainty processing, uncertain tokens are identified and propagated, necessary knowledge is obtained from the expert model, and mask template mining and filling based on the logits of LLM are performed to generate a few types of texts to alleviate the small sample problem; then, the trained GNN model is used for reasoning to obtain the prediction results: the advertisements that user 101 may click on are predicted to be: A001, A002, A003; the advertisements that user 102 may click on are predicted to be: A001, A002, A003; the advertisements that user 103 may click on are predicted to be: A001, A002, A003; the advertisements that user 104 may click on are predicted to be: A001, A002, A003; the classification labels of advertisements in the prediction results are determined through the multi-stream architecture combined with the HTC label perception representation, and then the candidate advertisements are determined by combining the prediction results and the classification labels of advertisements in the prediction results; finally, the Learning to Rank, combining the user's interest characteristics and the relevance of the advertisement, to generate the final recommendation list: User 101 recommendation list: [A002, A001, A003]; User 102 recommendation list: [A001, A003, A002]; User 103 recommendation list: [A003, A002, A001]; User 104 recommendation list: [A002, A001, A003]; at the same time, generate explanatory text of the recommendation information: For user 101: Recommend advertisement A002 because you have clicked on similar health advertisements before; For user 102: Recommend advertisement A001 because you are interested in technology products; For user 103: Recommend advertisement A003 because you pay attention to new fashion products; For user 104: Recommend advertisement A002 because you love a healthy lifestyle.

[0151] The following continues to describe the exemplary structure of the information recommendation device 543 provided in the embodiment of the present application as a software module. In some embodiments, such as Figure 2 As shown, the software modules stored in the information recommendation device 543 of the memory 540 may include: a construction module 5431 , a processing module 5432 and a recommendation module 5433 .

[0152] Construction module 5431 is used to construct graph data based on the first recommendation information in the object's first data. The graph data is used to determine the recommendation probabilities of different types of recommendation information. Processing module 5432 is used to extract features from the object's second data to obtain multiple modal features of the object, and fuse these multiple modal features to obtain a first fused feature. The first fused feature is used to classify the recommendation information. Recommendation module 5433 is used to generate a sequence of recommendation information for the object based on the first fused feature and the graph data, and use the recommendation information in the sequence as the second recommendation information sent to the object.

[0153] In some embodiments, the construction module 5431 is further used to construct a first-level graph data using the object and the first recommendation information as the first node, and to construct a second-level graph data using the first recommendation information and the type of the first item as the second node, wherein the first item is the item recommended by the first recommendation information.

[0154] In some embodiments, the processing module 5432 is also used to perform dimensionality reduction processing on multiple modal features to obtain multiple dimensionality reduction features; perform attention processing on the multiple dimensionality reduction features to obtain the weight of each dimensionality reduction feature; based on the weight of each dimensionality reduction feature, perform weighted summation on the multiple dimensionality reduction features, and use the weighted summation result as the first fusion feature.

[0155] In some embodiments, the processing module 5432 is also used to perform dimensionality reduction processing on multiple modal features to obtain multiple dimensionality reduction features; perform attention processing on the multiple dimensionality reduction features to obtain the weight of each dimensionality reduction feature, and based on the weight of each dimensionality reduction feature, perform weighted summation on the multiple dimensionality reduction features to obtain a weighted summation result; divide the weighted summation result into multiple sub-features, and determine the weight of each sub-feature; based on the weight of each sub-feature, fuse the multiple sub-features to obtain a first fused feature.

[0156] In some embodiments, the recommendation module 5433 is also used to classify the recommendation information in the recommendation information library according to the first fusion feature to obtain multiple recommendation information sets; perform feature extraction on the first node in the first-level graph data to obtain the first node feature, and perform feature extraction on the second node in the second-level graph data to obtain the second node feature; fuse the first node feature and the second node feature to obtain the second fusion feature, and perform probability prediction based on the second fusion feature to obtain the recommendation probability of different types of recommendation information; generate a recommendation information sequence for the object based on multiple recommendation information sets and the recommendation probabilities of different types of recommendation information.

[0157] In some embodiments, the recommendation module 5433 is also used to select a type whose recommendation probability is greater than or equal to a recommendation probability threshold as a type to be recommended; filter out a first recommendation information set from multiple recommendation information sets, and generate a recommendation information sequence for the object based on the recommendation information included in the first recommendation information set, wherein the first recommendation information set is a recommendation information set that meets the type to be recommended.

[0158] In some embodiments, the recommendation module 5433 is also used to determine the degree of match between the recommendation information included in the first recommendation information set and the first recommendation information; sort the recommendation information included in the first recommendation information set in descending order according to the degree of match from high to low to obtain a sorting result; and use the recommendation information sequence composed of the recommendation information ranked higher in the sorting result as the recommendation information sequence for the object.

[0159] In some embodiments, classification processing is implemented through a classification network, and probability prediction is implemented through a graph neural network. The information recommendation device 543 also includes a training module 5434 for obtaining a sample data set, the sample data set includes a graph data sample and multiple modal data samples; the initialized graph neural network is trained based on the graph data samples to obtain a trained graph neural network; the initialized classification network is trained based on multiple modal data samples to obtain a trained classification network.

[0160] In some embodiments, the information recommendation device 543 also includes an expansion module 5435, which is used to count the data volume of each type of data sample; for a type whose data volume is less than a data volume threshold, obtain a first data sample of the type; mask the first data sample through a pre-trained model to obtain a mask template of the first data sample; based on the mask template of the first data sample, generate and process it through a language model to obtain a second data sample; add the second data sample to multiple modal data samples to obtain multiple expanded modal data samples.

[0161] In some embodiments, the information recommendation device 543 further includes an interpretation module 5436 for determining a tag type of the second recommended information; and generating an interpretation text for the second recommended information based on the tag type and the behavior data.

[0162] It should be noted that the description of the device in the embodiment of the present application is similar to the description of the method embodiment above, and has similar beneficial effects as the method embodiment, so it will not be repeated here. Figure 3 、 Figure 4 ,or Figure 5 The present invention should be understood by referring to the description of any one of the accompanying drawings.

[0163] The present invention provides a computer program product including a computer program or computer-executable instructions stored in a computer-readable storage medium. A processor of an electronic device reads the computer-executable instructions from the computer-readable storage medium and executes the computer-executable instructions, causing the electronic device to perform the information recommendation method described in the present invention.

[0164] The embodiment of the present application provides a computer-readable storage medium in which computer-executable instructions or computer programs are stored. When the computer-executable instructions or computer programs are executed by a processor, the processor will execute the information recommendation method provided by the embodiment of the present application, for example, Figure 3 、 Figure 4 ,or Figure 5 The information recommendation method shown.

[0165] In some embodiments, the computer-readable storage medium may be a ferroelectric random access memory (FRAM), ROM, programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), flash memory, magnetic surface memory, optical disk, or compact disc read-only memory (CD-ROM); or various devices including one or any combination of the above memories.

[0166] In some embodiments, computer-executable instructions may be in the form of a program, software, software module, script, or code, written in any form of programming language (including compiled or interpreted languages, or declarative or procedural languages), and may be deployed in any form, including as a stand-alone program or as a module, component, subroutine, or other unit suitable for use in a computing environment.

[0167] As an example, computer-executable instructions may, but need not, correspond to a file in a file system, may be stored as part of a file that stores other programs or data, such as in one or more scripts in a HyperText Markup Language (HTML) document, in a single file dedicated to the program in question, or in multiple coordinating files (e.g., files storing one or more modules, subroutines, or code portions).

[0168] By way of example, computer-executable instructions may be deployed to be executed on one electronic device, or on multiple electronic devices located at one site, or on multiple electronic devices distributed across multiple sites and interconnected by a communication network.

[0169] In summary, the embodiments of the present application have the following beneficial effects:

[0170] (1) Through a multi-stream architecture, we integrate video, text, and location semantic streams, and use cross-stream agent attention fusion and token fusion mechanisms to effectively integrate different modal data and enhance feature expression capabilities;

[0171] (2) Using HTC label-aware representation, continuous training is performed through a pre-trained language model, combined with masked language modeling (MLM), hierarchical classification (CLS), and divergent contrastive learning (DCL) to capture the hierarchical relationship between labels and the semantic similarity of adjacent labels, thereby improving the model's ability to process labeled data;

[0172] (3) LLM-based logits mask template mining and filling are used to generate a small number of types of close-distributed texts, and data synthesis technology is used to alleviate the small sample problem and improve the generalization ability of the recommendation model;

[0173] (4) Generate recommendation explanation text and use knowledge graph to enhance recommendation explanation, helping users understand the reasons and basis for the recommendation, and improving user trust and satisfaction;

[0174] (5) Through multi-layer graph structure and GNN, effective cross-domain knowledge transfer is achieved, improving the effectiveness of advertising recommendations in different domains.

[0175] The above description is merely an embodiment of the present application and is not intended to limit the scope of protection of the present application. Any modifications, equivalent replacements, and improvements made within the spirit and scope of the present application are included in the scope of protection of the present application.

Claims

1. An information recommendation method, characterized in that: The method comprises: constructing graph data based on first recommendation information in the first data of the object, wherein the graph data is used to determine recommendation probabilities of different types of recommendation information; performing feature extraction on the second data of the object to obtain multiple modal features of the object, and fusing the multiple modal features to obtain a first fused feature, wherein the first fused feature is used to classify recommendation information in a recommendation information library; A recommendation information sequence for the object is generated based on the first fused feature and the graph data, and the recommendation information in the recommendation information sequence is used as second recommendation information sent to the object.

2. The method according to claim 1, characterized in that The constructing graph data based on the first recommendation information in the first data of the object includes: The object and the first recommendation information are used as first nodes to construct first-level graph data, and the first recommendation information and the type of the first item are used as second nodes to construct second-level graph data, wherein the first item is the item recommended by the first recommendation information.

3. The method according to claim 2, characterized in that The generating a recommendation information sequence for the object based on the first fusion feature and the graph data includes: classifying the recommendation information in the recommendation information database according to the first fusion feature to obtain multiple recommendation information sets; Performing feature extraction on a first node in the graph data of the first level to obtain a first node feature, and performing feature extraction on a second node in the graph data of the second level to obtain a second node feature; fusing the first node feature and the second node feature to obtain a second fused feature, and performing probability prediction based on the second fused feature to obtain recommendation probabilities of different types of recommendation information; A recommendation information sequence for the object is generated based on the multiple recommendation information sets and the recommendation probabilities of the different types of recommendation information.

4. The method according to claim 3, characterized in that Generating a recommendation information sequence for the object based on the multiple recommendation information sets and the recommendation probabilities of the different types of recommendation information includes: The type with a recommendation probability greater than or equal to the recommendation probability threshold is regarded as the type to be recommended; A first recommendation information set is screened out from the multiple recommendation information sets, and a recommendation information sequence for the object is generated based on the recommendation information included in the first recommendation information set, wherein the first recommendation information set is a recommendation information set that meets the type to be recommended.

5. The method according to claim 4, characterized in that Generating a recommendation information sequence for the object based on the recommendation information included in the first recommendation information set includes: determining a degree of matching between the recommendation information included in the first recommendation information set and the first recommendation information; Sorting the recommended information included in the first recommended information set in descending order according to the order of the matching degree from high to low to obtain a sorting result; The recommendation information sequence composed of the recommendation information ranked higher in the ranking result is used as the recommendation information sequence for the object.

6. The method according to claim 3, characterized in that The classification process is implemented by a classification network, and the probability prediction is implemented by a graph neural network; Before classifying the recommendation information in the recommendation information database according to the first fusion feature, the method further includes: Acquire a sample data set, where the sample data set includes a graph data sample and a plurality of modal data samples; Training the initialized graph neural network based on the graph data sample to obtain the trained graph neural network; The initialized classification network is trained based on the multiple modal data samples to obtain the trained classification network.

7. The method according to claim 6, characterized in that Before training the initialized classification network based on the multiple modality data samples, the method further includes: Count the amount of data for each type of data sample; For the type whose data volume is less than the data volume threshold, obtaining a first data sample of the type; Performing mask processing on the first data sample using a pre-trained model to obtain a mask template of the first data sample; Generate a second data sample by using a language model based on the mask template of the first data sample; The second data sample is added to the multiple modal data samples to obtain a plurality of expanded modal data samples.

8. The method according to claim 1, characterized in that The fusing the multiple modal features to obtain a first fused feature includes: Performing dimensionality reduction processing on the multiple modal features to obtain multiple dimensionality reduction features; Performing attention processing on the multiple dimensionality reduction features to obtain a weight of each of the dimensionality reduction features; Based on the weight of each of the dimensionality reduction features, a weighted sum is performed on the multiple dimensionality reduction features, and the weighted sum result is used as the first fusion feature.

9. The method according to claim 1, characterized in that The fusing the multiple modal features to obtain a first fused feature includes: Performing dimensionality reduction processing on the multiple modal features to obtain multiple dimensionality reduction features; Performing attention processing on the multiple dimensionality reduction features to obtain a weight of each of the dimensionality reduction features, and performing weighted summation on the multiple dimensionality reduction features based on the weight of each of the dimensionality reduction features to obtain a weighted summation result; Dividing the weighted summation result into multiple sub-features, and determining the weight of each sub-feature; Based on the weight of each of the sub-features, the multiple sub-features are fused to obtain a first fused feature.

10. The method according to any one of claims 1 to 9, characterized in that After using the recommendation information in the recommendation information sequence as second recommendation information sent to the object, the method further includes: Determining a tag type of the second recommendation information; Based on the tag type and the first data, an explanatory text of the second recommendation information is generated.

11. An information recommendation device, characterized in that: The device comprises: a construction module, configured to construct graph data based on first recommendation information in first data of the object, wherein the graph data is used to determine recommendation probabilities of different types of recommendation information; a processing module, configured to perform feature extraction on the second data of the object to obtain multiple modal features of the object, and fuse the multiple modal features to obtain a first fused feature, wherein the first fused feature is used to classify the recommendation information; A recommendation module is configured to generate a recommendation information sequence for the object based on the first fusion feature and the graph data, and use the recommendation information in the recommendation information sequence as second recommendation information sent to the object.

12. An electronic device, characterized in that: include: a memory for storing computer-executable instructions or computer programs; The processor is configured to implement the information recommendation method according to any one of claims 1 to 10 when executing the computer-executable instructions or computer programs stored in the memory.

13. A computer-readable storage medium storing computer-executable instructions or a computer program, characterized in that: When the computer-executable instructions or computer program are executed by a processor, the information recommendation method according to any one of claims 1 to 10 is implemented.

14. A computer program product comprising computer executable instructions or a computer program, characterized in that When the computer executable instructions or computer program are executed by a processor, the information recommendation method according to any one of claims 1 to 10 is implemented.

Images