Information recommendation method and device, equipment and storage medium

Abstract

The invention relates to the technical field of information processing, and provides an information recommendation method and device, equipment and a storage medium, and the method comprises the steps: obtaining to-be-recommended information, the to-be-recommended information comprising first type data and second type data; based on a first feature extraction model, extracting a first feature vector of the first type of data; based on a second feature extraction model, extracting a second feature vector of the second type of data; performing splicing fusion and dimension reduction processing on the first feature vector and the second feature vector to obtain multi-modal data of the to-be-recommended information; obtaining interest feature data of the user; and according to the interest feature data of the user and the multi-modal data of the to-be-recommended information, performing information recommendation on the user. Taking the financial field as an example, the to-be-recommended information comprises different types of data such as an insurance product image, a stock explanation video, a certain fund archive book and the like, the to-be-recommended information can be recommended to users with insurance, financial management and other requirements through the method and the device, and the personalized requirements of the users are met.

Description

Technical Field

[0001] The present application relates to the field of information processing technology, and in particular to an information recommendation method, device, equipment and storage medium. Background Art

[0002] Information recommendation means that when a user browses a certain information or finishes browsing the information, the system automatically recommends other information related to or similar to the information currently browsed by the user. With the development of science and technology, various information in the financial field has grown explosively. Faced with a large amount of information, it is difficult for investors to find the useful information they need and make investment decisions quickly based on this information. In order to enable investors to quickly obtain useful financial information, information recommendation methods can be used to recommend information that meets investors' preferences.

[0003] Related technologies focus on single-mode information. For example, they generally only focus on single-mode information such as images browsed by users or search records, which makes it impossible to meet users' personalized needs and results in a poor user experience. Summary of the invention

[0004] The main purpose of the present application is to provide a method, device, equipment and storage medium for recommending information, to fuse the feature vectors corresponding to different types of data, and to obtain the user's interest feature data so that the recommended news is more in line with the user's different interests and meets the user's personalized needs, thereby improving the accuracy of information recommendation and the user's experience. Taking the financial field as an example, the information to be recommended includes different types of data such as insurance product images, stock commentary videos, and a fund archive book. Through the information recommendation method provided by the present application, feature vectors corresponding to different types of data can be extracted, for example, image feature vectors corresponding to insurance product images, image feature vectors corresponding to stock commentary videos, text feature vectors, and text feature vectors corresponding to a fund archive book, etc.; fusing different feature vectors can obtain multimodal data of the information to be recommended, so as to more accurately recommend the information to be recommended to users with insurance, financial management and other needs, thereby meeting the user's personalized needs and improving the accuracy of information recommendation and the user's experience.

[0005] In a first aspect, the present application provides an information recommendation method, comprising:

[0006] Acquire information to be recommended, where the information to be recommended includes first type data and second type data;

[0007] Extracting a first feature vector of the first type of data based on a first feature extraction model;

[0008] Extracting a second feature vector of the second type of data based on a second feature extraction model;

[0009] The first feature vector and the second feature vector are concatenated, fused, and dimensionally reduced to obtain multimodal data of the information to be recommended;

[0010] Obtain user interest feature data;

[0011] Recommend information to the user based on the user's interest feature data and the multimodal data of the information to be recommended.

[0012] In a second aspect, the present application also provides an information recommendation device, comprising:

[0013] A first acquisition module, used to acquire information to be recommended, where the information to be recommended includes first type data and second type data;

[0014] A first feature extraction module, used for extracting a first feature vector of the first type of data based on a first feature extraction model;

[0015] A second feature extraction module, used for extracting a second feature vector of the second type of data based on a second feature extraction model;

[0016] A fusion module, used for concatenating and fusing the first feature vector and the second feature vector and performing dimension reduction processing to obtain multimodal data of the information to be recommended;

[0017] The second acquisition module is used to acquire the user's interest feature data;

[0018] The recommendation module is used to recommend news information to the user based on the user's interest feature data and the multimodal data of the information to be recommended.

[0019] In a third aspect, the present application further provides a computer device, the computer device comprising a memory and a processor;

[0020] The memory is used to store computer programs;

[0021] The processor is used to execute the computer program and implement the above-mentioned information recommendation method when executing the computer program.

[0022] In a fourth aspect, the present application further provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of the information recommendation method as described above are implemented.

[0023] The present application provides a method, device, equipment and storage medium for recommending information, wherein the method includes: obtaining information to be recommended, the information to be recommended includes a first type of data and a second type of data; extracting a first feature vector of the first type of data based on a first feature extraction model; extracting a second feature vector of the second type of data based on a second feature extraction model; splicing, fusing and reducing the dimension of the first feature vector and the second feature vector to obtain multimodal data of the information to be recommended; obtaining user interest feature data; and recommending information to the user based on the user interest feature data and the multimodal data of the information to be recommended. The present application first obtains information to be recommended of different types of data, then extracts feature vectors of different types of data based on different feature extraction models, and then fuses the feature vectors corresponding to different types of data to effectively obtain multimodal data of the information to be recommended; furthermore, by obtaining the user's interest feature data, the recommended news can be more in line with the user's different interests and meet the user's personalized needs, thereby improving the accuracy of information recommendation and the user's experience. Taking the financial field as an example, the information to be recommended includes different types of data such as insurance product images, stock commentary videos, and a fund archive book. Through the information recommendation method provided in this application, feature vectors corresponding to different types of data can be extracted, for example, image feature vectors corresponding to insurance product images, image feature vectors corresponding to stock commentary videos, text feature vectors, and text feature vectors corresponding to a fund archive book, etc.; by fusing different feature vectors, multimodal data of the information to be recommended can be obtained, so as to more accurately recommend the information to be recommended to users with insurance, financial management and other needs, thereby meeting the personalized needs of users and improving the accuracy of information recommendation and the user experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0025] Figure 1 A flowchart of a method for recommending information provided by an embodiment of the present application;

[0026] Figure 2 A schematic diagram of the connection between the server and the terminal provided in the embodiment of the present application;

[0027] Figure 3 A schematic block diagram of an information recommendation device provided in an embodiment of the present application;

[0028] Figure 4A schematic block diagram of the structure of a computer device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0029] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0030] The flowcharts shown in the accompanying drawings are only examples and do not necessarily include all the contents and operations / steps, nor must they be executed in the order described. For example, some operations / steps may also be decomposed, combined or partially merged, so the actual execution order may change according to actual conditions.

[0031] The embodiments of the present application provide a method, device, equipment and storage medium for recommending information. The information recommendation method can be applied to a terminal device, which can be a mobile phone, a tablet computer, a laptop computer, a desktop computer and the like. It can also be applied to a server, which can be a separate server or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms.

[0032] In conjunction with the accompanying drawings, some embodiments of the present application are described in detail below. In the absence of conflict, the following embodiments and features in the embodiments can be combined with each other.

[0033] See also Figure 1 , Figure 1 This is a flow chart of a method for recommending information provided in an embodiment of the present application. It should be noted that the method for recommending information provided in an embodiment of the present application can be used in a terminal device, and of course can also be used in a server.

[0034] like Figure 2 As shown, the information recommendation method is applied to a server, the server and the terminal device are in communication connection, and the information information obtained by the information recommendation method can be sent to the terminal device through the server. Of course, it is not limited to this and is not limited here.

[0035] In specific implementation, the terminal device includes but is not limited to: any one of a mobile phone, a tablet computer, a laptop computer, and a desktop computer; the server can be a single server or a server cluster, or a cloud server that provides cloud computing services.

[0036] like Figure 1 As shown, the news information recommendation method includes steps S101 to S106.

[0037] Step S101: Acquire information to be recommended, where the information to be recommended includes first type data and second type data.

[0038] For example, the information to be recommended generally includes real-time news, historical news related to real-time news, or information of the same type as real-time news. Taking the financial field as an example, the information to be recommended may include information related to insurance, stocks, financial management, funds, securities, etc. Taking the medical field as an example, the information to be recommended may include information related to medical devices, diseases, hospitals, medical staff, epidemic viruses, vaccines, etc.

[0039] Exemplarily, the information to be recommended may include different types of data, such as images, videos, audios, texts, etc. The first type of data in the embodiment of the present application may include images included in images and videos, and the second type of data may include texts, texts converted from voices in videos, and texts converted from voices in audios. For example, the first type of data may include architectural images of medical institutions, images of stock trends, medical images, etc. The second type of data may include texts in hospital department brochures, texts of disease prevention propaganda content, texts of fund profile information, etc.

[0040] Step S102: extracting a first feature vector of the first type of data based on a first feature extraction model.

[0041] For example, different types of data need to be extracted through different feature extraction models to obtain corresponding feature vectors. For example, image data can be processed by a feature extraction model that mainly extracts image features. In an embodiment of the present application, a first feature vector of a first type of data can be extracted by a first feature extraction model.

[0042] In an exemplary embodiment, the first type of data includes images; the first feature extraction model includes a multimodal pre-trained neural network; and step S102 includes step S1021A and step S1022A.

[0043] Step S1021A, inputting the first type of data into an image encoder of a multimodal pre-trained neural network to obtain a multi-layer feature vector;

[0044] Step S1022A: Select the last layer of feature vectors from the multiple layers of feature vectors as the image feature vector of the first type of data.

[0045] For example, the first type of data in the embodiment of the present application may include images of financial products, images of stock trends in stock commentary videos, images of medical devices, images in medical forum videos, etc. Videos generally include multiple images, so processing the video through the first feature extraction model is equivalent to processing the images in the video.

[0046] Exemplarily, the first feature extraction model in the embodiment of the present application may include a CLIP (Contrastive Language-Image Pre-training) model, i.e., a multimodal pre-trained neural network, which is pre-trained by a large amount of paired data of images and texts to learn the alignment relationship between images and texts. It should be noted that the CLIP model includes an image encoder, a text encoder, and a multimodal fusion module. Generally, the image encoder uses models such as a convolutional neural network or a variational autoencoder to encode the image into a vector. The text encoder uses a recurrent neural network to encode the text into a vector. The multimodal fusion module usually uses models such as an attention mechanism or a fully connected layer to fuse the image vector and the text vector. A multimodal fusion system based on the CLIP model is implemented.

[0047] By inputting images of financial products, stock trend images in stock commentary videos, images of medical devices, images in medical forum videos, etc. into a multimodal pre-trained neural network, the corresponding image feature vector can be obtained. Specifically, the images in the images and videos are input into the image encoder in the CLIP model to obtain a multi-layer feature vector, and then the last layer of feature vectors is selected from the multi-layer feature vectors as the image feature vector through the Squeeze function that maps the multi-dimensional input to the low-dimensional output. The Squeeze function can effectively reduce the dimension of the feature space to extract useful features, thereby greatly improving the accuracy and reliability of the model. Taking the news video of "A Hospital's Remote Outpatient Clinic Settled in B Hospital" as an example, the news video includes images. The image encoder in the CLIP model can be used to extract features of the image to obtain the first feature vector corresponding to the image.

[0048] In an exemplary embodiment, the first type of data includes multiple images of the same news type; the first feature extraction model includes a multimodal pre-trained neural network; and step S102 includes step S1021B and step S1022B.

[0049] Step S1021B: input multiple images of the same news type into the multimodal pre-trained neural network to obtain multiple first image feature vectors corresponding to the multiple images of the same news type.

[0050] Step S1022B: Calculate the average value of the multiple first image feature vectors to obtain image feature vectors corresponding to multiple images of the same news type.

[0051] Exemplarily, the information to be recommended may include images of different news types, and each news type may include multiple images. For example, images of insurance type, hospital type, stock type, financial type, and other images of different news types. Multiple images of the same news type are input into the multimodal pre-trained neural network to obtain multiple first image feature vectors corresponding to the images of the same news type; then the average value of the multiple first image feature vectors is calculated by the Average function to obtain the image feature vector corresponding to the images of the same news type, thereby ensuring the accuracy of the obtained image feature vector.

[0052] Specifically, taking the news video of "A Hospital's remote outpatient clinic settled in B Hospital" as an example, the news video includes 11 hospital images. The image encoder in the CLIP model can be used to extract features of the 11 hospital images to obtain 11 first feature vectors corresponding to the hospital images; then the average value of the 11 first image feature vectors is calculated through the Average function to obtain image feature vectors corresponding to 11 hospital images of the same news type.

[0053] Step S103: extracting a second feature vector of the second type of data based on a second feature extraction model.

[0054] For example, different types of data need to be feature extracted by different feature extraction models to obtain corresponding feature vectors. For example, text data can be processed by a feature extraction model that mainly extracts text features. The embodiment of the present application can extract a second feature vector of the second type of data by a second feature extraction model.

[0055] In an exemplary embodiment, the second type of data includes text; the second feature extraction model includes a first pre-trained language model and a second pre-trained language model; and step S103 includes step S1031 and step S1032.

[0056] Step S1031: input the second type of data into the first pre-trained language model to obtain a sentence vector of the second type of data;

[0057] Step S1032: input the sentence vector into the second pre-trained language model to obtain a text feature vector of the second type of data.

[0058] For example, the second type of data may include text, text converted from speech in a video, or text converted from audio, etc. For speech and audio in a video, the embodiment of the present application may first convert the speech and audio in the video into text, and then perform feature extraction on the text through a second feature extraction model.

[0059] The second feature extraction model in the embodiment of the present application may include a first pre-trained language model and a second pre-trained language model to process the text differently through the first pre-trained language model and the second pre-trained language model. The first pre-trained language model and the second pre-trained language model in the embodiment of the present application both include a RoBERTa model.

[0060] It should be noted that the RoBERTa model is a natural language processing model that uses larger-scale unlabeled text data (such as news, community questions and answers, encyclopedias, etc.) for pre-training, so that the RoBERTa model can learn richer language representations. Furthermore, the RoBERTa model uses a dynamic masking strategy, that is, a new masking pattern is generated each time a sequence is input, so that it can adapt to different masking strategies and learn richer language representations. Therefore, the embodiment of the present application can more accurately extract the second feature vector of the text through the RoBERTa model.

[0061] Exemplarily, the embodiments of the present application can embed text into the RoBERTa model through a pipeline (instruction pipeline in the computer field), such as a classic RISC pipeline, which can be applied to a central processing unit and other microprocessors to allow overlapping execution of multiple instructions in the same circuit.

[0062] Specifically, the text is first split into sentences, and then the sentences are input into the first pre-trained language model, i.e., the first RoBERTa model, for training to obtain sentence vectors; then the sentence vectors are input into the second pre-trained language model, i.e., the second RoBERTa model, for training to obtain the text feature vectors corresponding to the text. By processing the second type of data separately with two independent RoBERTa models, the text feature vectors of the second type of data can be extracted more accurately.

[0063] Take the news "A Hospital's Remote Outpatient Clinic Settled in B Hospital" as an example. There are eight paragraphs of text, including "On April 26, A Hospital in a certain province and city...", "National Clinical Medical Research Center for Neurological Diseases...", "A Hospital's Remote Outpatient Clinic...", "For 27 years, hundreds of experts in A Hospital have never stopped..." through word segmentation into a token list: ["4", "month", "2", "6", "day"...]; ["country", "nerve", "system", "disease", "clinical"...]; ["A", "hospital"...] and other tokens. These tokens are input into the first RoBERTa model to obtain the sentence vector [CLS]token, and then multiple sentence vectors are input into the second RoBERTa model to obtain the new [CLS]token, which is the text feature vector of the entire news.

[0064] Step S104: concatenate and fuse the first feature vector and the second feature vector and perform dimension reduction processing to obtain multimodal data of the information to be recommended.

[0065] In the embodiment of the present application, the first feature vector is an image feature vector, and the second feature vector is a text feature vector. After the image feature vector and the text feature vector are extracted, the image feature vector and the text feature vector can be fused to obtain richer and more accurate multimodal data.

[0066] It should be noted that the feature fusion methods include splicing fusion, weighted fusion and post-fusion feature selection. Splicing fusion refers to directly splicing different types of features together to form a new feature vector; weighted fusion refers to adding different types of features according to a certain weight to obtain a new feature vector; post-fusion feature selection refers to combining different types of features and then performing feature selection to select the most valuable features.

[0067] The embodiment of the present application can splice and fuse the image feature vector and the text feature vector to obtain a new feature vector, and then reduce the dimension of the new feature vector to obtain multimodal data of the information to be recommended with a vector dimension of 1. By splicing and fusing the image feature vector and the text feature vector, the data of the information to be recommended can be extracted more effectively, thereby improving the accuracy and coverage of information recommendation.

[0068] Step S105: Obtain user interest feature data.

[0069] Users generally have their own types of information that they are interested in. For example, sports news, medical news, financial news, travel news, etc. Correspondingly, the user's interest feature data may include basketball, running, fitness, food, insurance, medical care, stock market, securities, technology, travel, etc. Based on this, by obtaining the user's interest feature data, it is possible to determine what type of information the user is interested in, so as to improve the accuracy of information recommendation.

[0070] In an exemplary implementation, step S105 includes steps S1051 to S1054.

[0071] Step S1051: Obtain a preset interest vector set, where the preset interest vector set includes multiple interest indicators.

[0072] Step S1052: Obtain the user's browsing history news.

[0073] Step S1053: Based on the first calculation formula, according to the multiple interest indicators and the historical browsed news, determine the attention weight of the user's historical browsed news.

[0074] Step S1054: determining the user's interest feature data based on the second calculation formula and the attention weight of historically browsed news;

[0075] The first calculation formula is:

[0076]

[0077] The second calculation formula is:

[0078]

[0079] Among them, h Dj Multimodal data for historical browsing news; c i is the vector of interest indicators, i.e., learnable parameters; For dense networks; is the attention weight of the jth news; e i is the i-th interest indicator in the user's interest feature data; i and j are both natural numbers from 1 to K; K is the number of interest indicators.

[0080] For example, the preset interest vector set refers to a set of interest indicators including a large number of interest indicators. The interest vector set includes a variety of interest indicators, such as military, education, government affairs, basketball, running, fitness, food, insurance, medical care, stock market, securities, technology, travel, fashion, events, film and television, etc.

[0081] Generally, the user's interest index can be learned from the user's historical browsing news. Specifically, the news browsed by the user in the previous hour, the previous six hours or the previous day can be obtained under the condition of obtaining the user's authorized recommendation message, without having to obtain the news browsed by the user all the time to avoid infringing the user's privacy.

[0082] It should be noted that the attention mechanism enables the model to focus on the important parts of the input data, thereby improving the overall performance and efficiency. In the field of deep learning, especially in sequence models such as natural language processing (NLP) and computer vision, the attention mechanism plays a key role. It mimics the human attention process, enabling the model to focus on the key information in the input data while ignoring the less relevant parts. By assigning different weights (or attention scores) to different parts of the input data, the model is able to identify the most important information.

[0083] The embodiment of the present application uses the attention mechanism to assign different attention weights to historically browsed news based on a first calculation formula according to multiple interest indicators, and then determines the interest indicators that meet the user's interests based on the attention weights of historically browsed news based on a second calculation formula, thereby obtaining the user's interest feature data.

[0084] Specifically, the user's interest feature data may be determined according to the following formula:

[0085]

[0086] Among them, W h is a fully connected layer; h Dj Multimodal data for historical browsing news; c i is the vector of interest indicators, i.e., learnable parameters; is the transpose of the learnable parameters; For dense networks; is the attention weight of the jth news; e i is the i-th interest indicator in the user's interest feature data; i and j are both natural numbers from 1 to K; K is the number of interest indicators.

[0087] It is understandable that the multimodal data h of historical browsing news obtained Dj and the fully connected layer W h Multiply; then activate through the activation function tanh to get the contextual representation of the news; then, input the contextual representation of the news into the Attention layer of the attention mechanism to get the attention weight of the news Finally, the multimodal data h of each news Dj With attention weight By doing the dot product, we can get the user's interest feature data.

[0088] Step S106: Recommend information to the user based on the user's interest feature data and the multimodal data of the information to be recommended.

[0089] After obtaining the user's interest feature data, the embodiment of the present application can understand what type of news the user is interested in. For example, if the user's interest feature data includes interest indicators such as sports, securities, stocks, diseases, insurance, etc., the information corresponding to the aforementioned interest indicators in the recommended information can be recommended to the user to meet the user's personalized needs and improve the user's experience.

[0090] In an exemplary implementation, step S106 includes step S1061 and step S1062.

[0091] Step S1061: Determine the degree of matching between the user's interest feature data and the multimodal data of the information to be recommended.

[0092] Step S1062: recommending to the user the to-be-recommended information corresponding to the multimodal data whose matching degree is greater than or equal to the preset matching value.

[0093] The embodiment of the present application can determine the degree of match between each interest indicator in the user's interest feature data and the information to be recommended; if the degree of match is small, it means that the user is not interested in the news and it is not necessary to recommend it to the user; if the degree of match is large, it means that the user is interested in the news and it can be recommended to the user. Specifically, a matching value can be preset first, and by comparing the matching degree with the preset matching value, news suitable for recommendation to the user can be selected to meet the user's needs.

[0094] Exemplarily, the preset matching values ​​corresponding to different interest indicators may be different. For example, the user's interest feature data includes four interest features: stocks, insurance, food, and travel, and the user's interest in these four interest features is from large to small: travel, food, stocks, and insurance. The corresponding preset matching values ​​can be set to 30%, 45%, 60%, and 80%, respectively. By setting different preset matching values ​​for different interest indicators, the embodiment of the present application can recommend information to the user to prevent the user from frequently browsing news with rough insurance content by setting different preset matching values ​​for different interest indicators. The information to be recommended may include news about insurance that the user is not very interested in. Information about a certain insurance in the information to be recommended can be recommended to the user, so as to avoid the user from frequently browsing news with rough insurance content, thereby improving the user's experience and the accuracy of information recommendation.

[0095] Before recommending news to users, the recommended information can be processed to reduce model training time. When the recommended information only has a single modality, the other modality can be initialized to a 0 vector as an expression, and then the input model can be spliced ​​for inference; when the text of the recommended information is too long, the length of the text can be reduced by extracting key sentences without reducing the value of the news text. For recommended information with different types of images, in order to improve the accuracy, the image feature vectors of the recommended information can be clustered first, the image feature vectors far away from the cluster center can be removed, and the image feature vectors at the cluster center can be selected as multimodal data.

[0096] In an exemplary implementation, step S1061 specifically includes: determining the degree of matching between the user's interest feature data and the multimodal data of the information to be recommended based on a first matching formula.

[0097] The first matching formula is:

[0098]

[0099] Among them, H Newi is the multimodal data of the information to be recommended; j is the jth interest indicator in the user's interest feature data; i is a natural number from 1 to n; n is the number of multimodal data of the information to be recommended; j is a natural number from 1 to K; K is the number of interest indicators in the user's interest feature data.

[0100] The embodiment of the present application can accurately calculate the matching degree between each interest indicator in the user's interest feature data and the information to be recommended through the first matching formula, so as to further improve the accuracy of information recommendation.

[0101] The information recommendation method provided in the above embodiment includes: obtaining information to be recommended, the information to be recommended includes a first type of data and a second type of data; extracting a first feature vector of the first type of data based on a first feature extraction model; extracting a second feature vector of the second type of data based on a second feature extraction model; splicing, fusing and reducing the dimension of the first feature vector and the second feature vector to obtain multimodal data of the information to be recommended; obtaining user interest feature data; and recommending information to the user based on the user interest feature data and the multimodal data of the information to be recommended. The embodiment of the present application first obtains information to be recommended of different types of data, then extracts feature vectors of different types of data based on different feature extraction models, and then fuses the feature vectors corresponding to different types of data to effectively obtain multimodal data of the information to be recommended; furthermore, by obtaining the user's interest feature data, the recommended news can be more in line with the user's different interests and meet the user's personalized needs, thereby improving the accuracy of information recommendation and the user's experience. Taking the financial field as an example, the information to be recommended includes different types of data such as insurance product images, stock commentary videos, and a fund archive book. Through the information recommendation method provided in the embodiment of the present application, feature vectors corresponding to different types of data can be extracted, for example, image feature vectors corresponding to insurance product images, image feature vectors corresponding to stock commentary videos, text feature vectors, and text feature vectors corresponding to a fund archive book, etc.; by fusing different feature vectors, multimodal data of the information to be recommended can be obtained, so as to more accurately recommend the information to be recommended to users with insurance, financial management and other needs, thereby meeting the personalized needs of users and improving the accuracy of information recommendation and the user experience.

[0102] See also Figure 3 , Figure 3 This is a schematic block diagram of an information recommendation device provided in an embodiment of the present application. The information recommendation device can be configured in a server or a terminal device to execute the aforementioned information recommendation method.

[0103] like Figure 3 As shown, the news information recommendation device includes: a first acquisition module 110, a first feature extraction module 120, a second feature extraction module 130, a fusion module 140, a second acquisition module 150 and a recommendation module 160.

[0104] The first acquisition module 110 is used to acquire information to be recommended, where the information to be recommended includes first type data and second type data.

[0105] The first feature extraction module 120 is used to extract a first feature vector of the first type of data based on a first feature extraction model.

[0106] The second feature extraction module 130 is used to extract a second feature vector of the second type of data based on a second feature extraction model.

[0107] The fusion module 140 is used to concatenate and fuse the first feature vector and the second feature vector and perform dimension reduction processing to obtain multimodal data of the information to be recommended.

[0108] The second acquisition module 150 is used to acquire the user's interest feature data.

[0109] The recommendation module 160 is used to recommend news information to users based on the user's interest feature data and the multimodal data of the information to be recommended.

[0110] In an exemplary embodiment, the second acquisition module 150 includes a first acquisition submodule, a second acquisition submodule, a weight determination submodule, and an interest determination submodule.

[0111] The first acquisition submodule is used to acquire a preset interest vector set, where the preset interest vector set includes a plurality of interest indicators.

[0112] The second acquisition submodule is used to acquire the user's historical browsing news.

[0113] The weight determination submodule is used to determine the attention weight of the user's historical browsing news based on the first calculation formula, according to multiple interest indicators and historical browsing news.

[0114] An interest determination submodule, for determining the user's interest characteristic data based on the second calculation formula and the attention weight of historically browsed news;

[0115] The first calculation formula is:

[0116]

[0117] The second calculation formula is:

[0118]

[0119] Among them, h Dj Multimodal data for historical browsing news; c i is the vector of interest indicators, i.e., learnable parameters; For dense networks; is the attention weight of the jth news; e i is the i-th interest indicator in the user's interest feature data; i and j are both natural numbers from 1 to K; K is the number of interest indicators.

[0120] In an exemplary implementation, the recommendation module 160 includes a matching submodule and a recommendation submodule.

[0121] The matching submodule is used to determine the matching degree between the user's interest feature data and the multimodal data of the information to be recommended.

[0122] The recommendation submodule is used to recommend to the user the to-be-recommended information corresponding to the multimodal data whose matching degree is greater than or equal to a preset matching value.

[0123] In an exemplary embodiment, the matching submodule is specifically used to determine the matching degree between the user's interest feature data and the multimodal data of the information to be recommended based on the first matching formula;

[0124] The first matching formula is:

[0125]

[0126] Among them, H Newi is the multimodal data of the information to be recommended; j is the jth interest indicator in the user's interest feature data; i is a natural number from 1 to n; n is the number of multimodal data of the information to be recommended; j is a natural number from 1 to K; K is the number of interest indicators in the user's interest feature data.

[0127] In an exemplary embodiment, the first type of data includes images; the first feature extraction model includes a multimodal pre-trained neural network; and the first feature extraction module 120 specifically includes an input submodule and a selection submodule.

[0128] An input submodule, used for inputting the first type of data into an image encoder of a multimodal pre-trained neural network to obtain a multi-layer feature vector;

[0129] The selection submodule is used to select the last layer of feature vectors from the multi-layer feature vectors as the image feature vector of the first type of data.

[0130] In an exemplary embodiment, the first type of data includes multiple images of the same news type; the first feature extraction model includes a multimodal pre-trained neural network; and the first feature extraction module 120 specifically includes an image extraction submodule and a calculation submodule.

[0131] An image extraction submodule, used for inputting a plurality of images of the same news type into a multimodal pre-trained neural network to obtain a plurality of first image feature vectors corresponding to the plurality of images of the same news type;

[0132] The calculation submodule is used to calculate the average value of multiple first image feature vectors to obtain image feature vectors corresponding to multiple images of the same news type.

[0133] In an exemplary embodiment, the second type of data includes text; the second feature extraction module 130 includes.

[0134] The sentence extraction submodule is used to input the second type of data into the first pre-trained language model to obtain the sentence vector of the second type of data.

[0135] The text extraction submodule is used to input the sentence vector into the second pre-trained language model to obtain a text feature vector of the second type of data.

[0136] It should be noted that technicians in the relevant field can clearly understand that for the convenience and simplicity of description, the specific working processes of the above-described devices and modules and units can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0137] The method of the present application can be used in many general or special computer system environments or configurations. For example: personal computers, server computers, handheld devices or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronic devices, network PCs, minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, etc. The present application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. The present application can also be practiced in distributed computing environments, in which tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules can be located in local and remote computer storage media including storage devices.

[0138] Exemplarily, the above method and apparatus may be implemented in the form of a computer program, which may be run on a computer device.

[0139] See also Figure 4 , Figure 4 1 is a schematic block diagram of a computer device provided in an embodiment of the present application. The computer device may be a server or a terminal device.

[0140] like Figure 4 As shown, the computer device includes a processor, a memory, and a network interface connected via a system bus, wherein the memory may include a storage medium and an internal memory.

[0141] The storage medium can store an operating system and a computer program. The computer program includes program instructions, and when the program instructions are executed, the processor can execute the steps of any information recommendation method.

[0142] The processor is used to provide computing and control capabilities and support the operation of the entire computer equipment.

[0143] The internal memory provides an environment for the operation of the computer program in the storage medium. When the computer program is executed by the processor, the processor can execute the steps of any information recommendation method.

[0144] The network interface is used for network communication, such as sending assigned tasks.

[0145] Those skilled in the art will understand that Figure 4 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0146] It should be understood that the processor may be a central processing unit (CPU), and the processor may also be other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. Among them, the general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc.

[0147] In one embodiment, the processor is used to execute a computer program and implement the following steps when executing the computer program:

[0148] Acquire information to be recommended, where the information to be recommended includes first type data and second type data;

[0149] Extracting a first feature vector of the first type of data based on a first feature extraction model;

[0150] Extracting a second feature vector of the second type of data based on the second feature extraction model;

[0151] The first eigenvector and the second eigenvector are concatenated, fused, and dimensionally reduced to obtain multimodal data of the information to be recommended;

[0152] Obtain user interest feature data;

[0153] Recommend information to users based on their interest feature data and the multimodal data of the information to be recommended.

[0154] In an exemplary embodiment, obtaining user interest feature data includes:

[0155] Obtaining a preset interest vector set, where the preset interest vector set includes multiple interest indicators;

[0156] Get the user's historical browsing news;

[0157] Based on the first calculation formula, according to the multiple interest indicators and the historically browsed news, the attention weight of the user's historically browsed news is determined;

[0158] Based on the second calculation formula, the user's interest feature data is determined according to the attention weight of historical browsing news;

[0159] The first calculation formula is:

[0160]

[0161] The second calculation formula is:

[0162]

[0163] Among them, h Dj Multimodal data for historical browsing news; c i is the vector of interest indicators, i.e., learnable parameters; For dense networks; is the attention weight of the jth news; e i is the i-th interest indicator in the user's interest feature data; i and j are both natural numbers from 1 to K; K is the number of interest indicators.

[0164] In an exemplary embodiment, recommending information to a user based on the user's interest feature data and the multimodal data of the information to be recommended includes:

[0165] Determine the matching degree between the user's interest feature data and the multimodal data of the information to be recommended;

[0166] Recommend to the user the information to be recommended corresponding to the multimodal data whose matching degree is greater than or equal to the preset matching value.

[0167] In an exemplary embodiment, determining the matching degree between the user's interest feature data and the multimodal data of the information to be recommended includes:

[0168] Based on the first matching formula, determining the matching degree between the user's interest feature data and the multimodal data of the information to be recommended;

[0169] The first matching formula is:

[0170]

[0171] Among them, H Newiis the multimodal data of the information to be recommended; j is the jth interest indicator in the user's interest feature data; i is a natural number from 1 to n; n is the number of multimodal data of the information to be recommended; j is a natural number from 1 to K; K is the number of interest indicators in the user's interest feature data.

[0172] In an exemplary embodiment, the first type of data includes an image; the first feature extraction model includes a multimodal pre-trained neural network; based on the first feature extraction model, extracting a first feature vector of the first type of data includes:

[0173] Inputting the first type of data into an image encoder of a multimodal pre-trained neural network to obtain a multi-layer feature vector;

[0174] The last layer of feature vectors is selected from the multi-layer feature vectors as the image feature vector of the first type of data.

[0175] In an exemplary embodiment, the first type of data includes a plurality of images of the same news type; the first feature extraction model includes a multimodal pre-trained neural network; based on the first feature extraction model, extracting a first feature vector of the first type of data includes:

[0176] Inputting a plurality of images of the same news type into a multimodal pre-trained neural network to obtain a plurality of first image feature vectors corresponding to the plurality of images of the same news type;

[0177] An average value of the plurality of first image feature vectors is calculated to obtain image feature vectors corresponding to a plurality of images of the same news type.

[0178] In an exemplary embodiment, the second type of data includes text; the second feature extraction model includes a first pre-trained language model and a second pre-trained language model; based on the second feature extraction model, extracting a second feature vector of the second type of data includes:

[0179] Inputting the second type of data into the first pre-trained language model to obtain a sentence vector of the second type of data;

[0180] The sentence vector is input into the second pre-trained language model to obtain a text feature vector of the second type of data.

[0181] It should be noted that those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working process of the above-described information recommendation can refer to the corresponding process in the embodiment of the aforementioned information recommendation method, and will not be repeated here.

[0182] The embodiment of the present application also provides a computer-readable storage medium, on which a computer program is stored. The method implemented when the computer program is executed by a processor can refer to the various embodiments of the information recommendation method of the present application.

[0183] The computer-readable storage medium may be an internal storage unit of the computer device of the aforementioned embodiment, such as a hard disk or memory of the computer device. The computer-readable storage medium may also be an external storage device of the computer device, such as a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), etc. equipped on the computer device.

[0184] It should be understood that the terms used in this application specification are only for the purpose of describing specific embodiments and are not intended to limit the application. As used in this application specification and the appended claims, unless the context clearly indicates otherwise, the singular forms "a", "an" and "the" are intended to include plural forms.

[0185] It should also be understood that the term "and / or" used in the specification of this application and the appended claims refers to any combination of one or more of the associated listed items and all possible combinations, including these combinations. It should be noted that, in this article, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or system including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or system. In the absence of further restrictions, an element defined by the sentence "including a..." does not exclude the presence of other identical elements in the process, method, article or system including the element.

[0186] The serial numbers of the embodiments of the present application are for description only and do not represent the advantages and disadvantages of the embodiments. The above description is only a specific implementation mode of the present application, but the protection scope of the present application is not limited thereto. Any technician familiar with the technical field can easily think of various equivalent modifications or replacements within the technical scope disclosed in the present application, and these modifications or replacements should be included in the protection scope of the present application. Therefore, the protection scope of the present application shall be based on the protection scope of the claims.

Claims

1. A method for recommending information, characterized in that: include: Acquire information to be recommended, where the information to be recommended includes first type data and second type data; Extracting a first feature vector of the first type of data based on a first feature extraction model; Extracting a second feature vector of the second type of data based on a second feature extraction model; Concatenate, fuse and reduce the dimension of the first feature vector and the second feature vector to obtain multimodal data of the information to be recommended; Obtain user interest feature data; Recommend information to the user based on the user's interest feature data and the multimodal data of the information to be recommended.

2. The information recommendation method according to claim 1, characterized in that: The obtaining of user's interest characteristic data includes: Acquire a preset interest vector set, where the preset interest vector set includes a plurality of interest indicators; Obtain the user's browsing history news; Based on the first calculation formula, determining the attention weight of the user's historical browsing news according to the plurality of interest indicators and the historical browsing news; Based on the second calculation formula, the interest feature data of the user is determined according to the attention weight of the historical browsing news; The first calculation formula is: The second calculation formula is: Among them, h Dj Multimodal data for historical browsing news; c i is the vector of interest indicators, i.e., learnable parameters; For dense networks; is the attention weight of the jth news; e i is the i-th interest indicator in the user's interest feature data; i and j are both natural numbers from 1 to K; K is the number of interest indicators.

3. The information recommendation method according to claim 1, characterized in that: The recommending information to the user based on the user's interest feature data and the multimodal data of the information to be recommended includes: Determining the degree of matching between the user's interest feature data and the multimodal data of the information to be recommended; Recommending to the user the to-be-recommended information corresponding to the multimodal data whose matching degree is greater than or equal to a preset matching value.

4. The information recommendation method according to claim 3, characterized in that: The determining the matching degree between the user's interest feature data and the multimodal data of the information to be recommended includes: Based on a first matching formula, determining a matching degree between the user's interest feature data and the multimodal data of the information to be recommended; The first matching formula is: Among them, H Newi is the multimodal data of the information to be recommended; j is the jth interest indicator in the interest feature data of the user; i is a natural number from 1 to n; n is the number of multimodal data of the information to be recommended; j is a natural number from 1 to K; K is the number of interest indicators in the interest feature data of the user.

5. The information recommendation method according to claim 1, characterized in that: The first type of data includes an image, and the first feature extraction model includes a multimodal pre-trained neural network; extracting a first feature vector of the first type of data based on the first feature extraction model includes: Inputting the first type of data into the image encoder of the multimodal pre-trained neural network to obtain a multi-layer feature vector; A last layer of feature vectors is selected from the multiple layers of feature vectors as an image feature vector of the first type of data.

6. The information recommendation method according to claim 1, characterized in that: The first type of data includes a plurality of images of the same news type; the first feature extraction model includes a multimodal pre-trained neural network; and extracting a first feature vector of the first type of data based on the first feature extraction model includes: Inputting the plurality of images of the same news type into the multimodal pre-trained neural network to obtain a plurality of first image feature vectors corresponding to the plurality of images of the same news type; An average value of the plurality of first image feature vectors is calculated to obtain image feature vectors corresponding to the plurality of images of the same news type.

7. The information recommendation method according to claim 1, characterized in that: The second type of data includes text; the second feature extraction model includes a first pre-trained language model and a second pre-trained language model; and extracting a second feature vector of the second type of data based on the second feature extraction model includes: Inputting the second type of data into the first pre-trained language model to obtain a sentence vector of the second type of data; The sentence vector is input into the second pre-trained language model to obtain a text feature vector of the second type of data.

8. An information recommendation device, characterized in that: include: A first acquisition module, used to acquire information to be recommended, where the information to be recommended includes first type data and second type data; A first feature extraction module, used for extracting a first feature vector of the first type of data based on a first feature extraction model; A second feature extraction module, used for extracting a second feature vector of the second type of data based on a second feature extraction model; A fusion module, used for concatenating and fusing the first feature vector and the second feature vector and performing dimension reduction processing to obtain multimodal data of the information to be recommended; The second acquisition module is used to acquire the user's interest feature data; The recommendation module is used to recommend news information to the user based on the user's interest feature data and the multimodal data of the information to be recommended.

9. A computer device, characterized in that: The computer device includes a memory and a processor; The memory is used to store computer programs; The processor is used to execute the computer program and implement the information recommendation method as described in any one of claims 1 to 7 when executing the computer program.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the information recommendation method according to any one of claims 1 to 7 are implemented.

Images