Information fusion method, content recommendation method and device, electronic equipment and medium

Abstract

The invention provides an information fusion method, a content recommendation method and apparatus, an electronic device and a medium. The method comprises the steps of obtaining historical operation data of a target object; wherein the historical operation data comprises original content coding information and original auxiliary coding information of the original content coding information; inputting the original content coding information into a matrix mapping model to generate a content correlation matrix, and inputting each piece of original auxiliary coding information into the matrix mapping model to generate an auxiliary correlation matrix; inputting the content correlation matrix and the auxiliary correlation matrix into a matrix fusion model to obtain a correlation fusion matrix; and inputting the correlation fusion matrix and the original content coding information into a content fusion model to obtain target fusion information. According to the embodiment of the invention, the auxiliary coding information of different dimensions can be fused into the content coding information, and the representation of the content information is enriched. The embodiment of the invention can be applied to various scenes such as cloud technology, artificial intelligence, intelligent traffic, auxiliary driving and the like.

Description

Technical Field

[0001] The present disclosure relates to the field of machine learning, and in particular to an information fusion method, a content recommendation method and device, an electronic device, and a medium. Background Art

[0002] In machine learning, content recommendations are often made based on the preferences of the target object, such as video recommendations, audio recommendations, item recommendations, and text recommendations. Among them, how to represent content information in the content recommendation process is the key to the accuracy of the recommendation. In related technologies, auxiliary information is injected into the learning representation of content information, but the auxiliary information of the same dimension needs to be encoded and then integrated into the content information to construct a new learning representation, which limits the integration of auxiliary information of other dimensions into the content information, thereby reducing the richness of the content information representation and making the recommended content inaccurate. Summary of the invention

[0003] The embodiments of the present disclosure provide an information fusion method, a content recommendation method and device, an electronic device, and a medium, which can fuse auxiliary information of different dimensions, improve the richness of content information representation, and thus improve the accuracy of recommended content.

[0004] According to one aspect of the present disclosure, there is provided an information fusion method, comprising:

[0005] Acquire historical operation data of the target object; wherein the historical operation data includes: original content encoding information and original auxiliary encoding information of the original content encoding information;

[0006] Inputting the original content coding information into a matrix mapping model to generate a content correlation matrix, and inputting each of the original auxiliary coding information into the matrix mapping model to generate an auxiliary correlation matrix;

[0007] Inputting the content relevance matrix and the auxiliary relevance matrix into a matrix fusion model to obtain a relevance fusion matrix;

[0008] The correlation fusion matrix and the original content encoding information are input into a content fusion model to obtain target fusion information.

[0009] According to one aspect of the present disclosure, there is provided an information fusion device, comprising:

[0010] A data acquisition module, used to acquire historical operation data of a target object; wherein the historical operation data includes: original content coding information and original auxiliary coding information of the original content coding information;

[0011] A mapping module, used for inputting the original content coding information into a matrix mapping model to generate a content correlation matrix, and inputting each of the original auxiliary coding information into the matrix mapping model to generate an auxiliary correlation matrix;

[0012] A matrix fusion module, used for inputting the content relevance matrix and the auxiliary relevance matrix into a matrix fusion model to obtain a relevance fusion matrix;

[0013] The content fusion module is used to input the correlation fusion matrix and the original content encoding information into a content fusion model to obtain target fusion information.

[0014] Optionally, the mapping module is specifically used for:

[0015] Inputting the original content coding information into the matrix mapping model, and using the matrix mapping model to map the original content coding information to a preset content parameter matrix to generate the content relevance matrix;

[0016] Each of the original auxiliary coding information is input into the matrix mapping model, and the matrix mapping model maps each of the original auxiliary coding information to a preset auxiliary parameter matrix to generate the auxiliary correlation matrix; wherein each of the auxiliary correlation matrices has the same dimension.

[0017] Optionally, the matrix fusion module is specifically used for:

[0018] Inputting the content relevance matrix and the auxiliary relevance matrix into a matrix fusion model, and calculating a first weight of the content relevance matrix and a second weight of each of the auxiliary relevance matrices through the matrix fusion model;

[0019] Based on the first weight and the second weight, the content relevance matrix and the auxiliary relevance matrix are weightedly fused to obtain the relevance fusion matrix.

[0020] Optionally, the matrix fusion model includes: a convolution layer, a pooling layer, a fully connected layer and an activation function layer; the matrix fusion module is specifically used for:

[0021] Inputting the content relevance matrix into the convolution layer to obtain content relevance features, and inputting each of the auxiliary relevance matrices into the convolution layer to obtain auxiliary relevance features;

[0022] Inputting the content correlation feature into the pooling layer to obtain a first vector, and inputting the auxiliary correlation feature into the pooling layer to obtain a second vector;

[0023] Input the first vector to the fully connected layer to obtain a third vector, and input each of the second vectors to the fully connected layer to obtain a fourth vector;

[0024] The third vector is input into the activation function layer to obtain the first weight of the content correlation matrix, and the fourth vector is input into the activation function layer to obtain the second weight of each auxiliary correlation matrix.

[0025] Optionally, the information fusion device further includes:

[0026] A training module is used to jointly train the matrix mapping model, the matrix fusion model and the content fusion model in advance through the following process:

[0027] Acquire training sample data; wherein the training sample data includes: historical behavior sample data, the historical behavior sample data includes sample content coding information and sample auxiliary coding information of the sample content coding information;

[0028] Inputting the sample content coding information into the matrix mapping model to generate a sample content matrix; inputting the sample auxiliary coding information into the matrix mapping model to generate a sample auxiliary matrix;

[0029] Inputting the sample content matrix and the sample auxiliary matrix into the matrix fusion model to obtain a sample fusion matrix;

[0030] Inputting the sample fusion matrix and the sample content encoding information into the content fusion model to obtain sample fusion information;

[0031] Inputting the sample fusion information into a content estimation model to obtain content estimation information;

[0032] Constructing loss data based on preset content verification information and the content estimation information;

[0033] The parameters of the matrix mapping model, the matrix fusion model, and the content fusion model are adjusted based on the loss data.

[0034] Optionally, the training module is specifically used for:

[0035] Extracting content coding element information from the sample fusion information;

[0036] The content coding element information is input into the content prediction model to obtain the content prediction information.

[0037] Optionally, the data acquisition module is specifically used to:

[0038] Acquire candidate auxiliary information of the candidate content information, and acquire candidate content coding information of the candidate content information and candidate auxiliary coding information of each of the candidate auxiliary information;

[0039] Acquire historical operation data of the target object; wherein the historical operation data includes browsing content information;

[0040] selecting the original content encoding information from the candidate content encoding information based on the browsing content information and the candidate content information;

[0041] The original auxiliary encoding information is selected from among the candidate auxiliary encoding information based on the browsing content information, the candidate content information and the candidate auxiliary information.

[0042] Optionally, the data acquisition module is specifically used to:

[0043] Based on the browsing content information, selecting selected content information from the candidate content information; wherein the selected content information is the same as the browsing content information;

[0044] The original auxiliary coding information is selected from among the candidate auxiliary coding information based on the selected auxiliary information of the selected content information and the candidate auxiliary information.

[0045] According to one aspect of the present disclosure, a content recommendation method is provided, comprising:

[0046] Obtaining target fusion information browsed at each time point in the historical operation data of the target object; wherein the target fusion information is obtained by the above-mentioned information fusion method;

[0047] Inputting the target fusion information into a content estimation model to obtain a content estimation probability sequence;

[0048] Recommended content information is constructed based on the content estimation probability sequence.

[0049] According to one aspect of the present disclosure, there is provided a content recommendation device, comprising:

[0050] An information acquisition module, used to acquire target fusion information browsed at each time point in the historical operation data of the target object; wherein the target fusion information is obtained by the above-mentioned information fusion method;

[0051] A content estimation module, used for inputting the target fusion information into a content estimation model to obtain a content estimation probability sequence;

[0052] The recommendation module is used to construct recommended content information based on the content estimation probability sequence.

[0053] Optionally, the content estimated probability sequence includes: an estimated probability of each candidate content information; and the recommendation module is specifically used to:

[0054] According to the estimated probability, the candidate content information is sorted in ascending order to obtain a content sorting sequence number;

[0055] The candidate content information of the preset recommended number before the content sorting sequence number is obtained as the recommended content information.

[0056] According to one aspect of the present disclosure, an electronic device is provided, including a memory and a processor, wherein the memory stores a computer program, and the processor implements the above-mentioned information fusion method or content recommendation method when executing the computer program.

[0057] According to one aspect of the present disclosure, a computer-readable storage medium is provided, wherein the storage medium stores a computer program, and when the computer program is executed by a processor, the information fusion method or the content recommendation method as described above is implemented.

[0058] The disclosed embodiment converts the original content coding information and the original auxiliary coding information in the historical operation data into a matrix of the same latitude through a matrix mapping model, so that the original auxiliary coding information of different dimensions from the original content coding information can also be fused with the original content coding information, and then the original auxiliary coding information of different dimensions can be fully utilized to generate a more accurate original content learning representation, that is, to obtain more accurate target fusion information. This method improves the richness of the content information representation dimension, and thus improves the accuracy of the learned content information representation 。

[0059] Other features and advantages of the present disclosure will be described in the following description, and partly become apparent from the description, or understood by practicing the present disclosure. The purpose and other advantages of the present disclosure can be realized and obtained by the structures particularly pointed out in the description, claims and drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0060] The accompanying drawings are used to provide further understanding of the technical solution of the present disclosure and constitute a part of the specification. Together with the embodiments of the present disclosure, they are used to explain the technical solution of the present disclosure and do not constitute a limitation on the technical solution of the present disclosure.

[0061] Figure 1 is a system architecture diagram of an information fusion method application system according to an embodiment of the present disclosure;

[0062] Figure 2A -C is an interface diagram of the information fusion method according to an embodiment of the present disclosure applied in a content recommendation scenario;

[0063] Figure 3A -C is a schematic diagram of the application of the information fusion method according to an embodiment of the present disclosure in an article recommendation scenario;

[0064] Figure 4A -C is a schematic diagram of applying the information fusion method according to an embodiment of the present disclosure to a commodity recommendation scenario;

[0065] Figure 5A -C is a schematic diagram of applying the information fusion method according to an embodiment of the present disclosure in a video recommendation scenario;

[0066] Figure 6 is a flow chart of an information fusion method according to an embodiment of the present disclosure;

[0067] Figure 7 is a schematic diagram of obtaining historical operation data in the information fusion method according to an embodiment of the present disclosure;

[0068] Figure 8 yes Figure 6 A specific flow chart of step 601 in FIG.

[0069] Fig. 9A -B is a schematic diagram showing the acquisition of associated information of video A in the information fusion method according to an embodiment of the present disclosure;

[0070] Fig.10 A schematic diagram showing a historical viewing record in an information fusion method according to an embodiment of the present disclosure is shown;

[0071] Fig.11 Shows Figure 8 A specific flow chart of step 804;

[0072] Fig.12 Shows Figure 6 A specific flow chart of step 602;

[0073] Fig.13 Shows Fig.12 A specific flow chart of step 1201 and step 1202;

[0074] Fig.14 Shows Fig.12 A specific structural diagram of the matrix mapping model;

[0075] Fig.15 Shows Figure 6 A specific flow chart of step 603 in FIG.

[0076] Fig.16 Shows Fig.15 A specific flow chart of step 1501 and step 1502;

[0077] Fig.17 Shows Fig.15 A specific flow chart of step 1501 in FIG.

[0078] Fig.18 Shows Fig.16 A specific structural diagram of the SE module;

[0079] Fig.19 Shows Figure 6 A specific flow chart of step 604;

[0080] Fig. 20 Shows joint training Figure 6 A flowchart of an implementation of a matrix mapping model, a matrix fusion model and a content fusion model;

[0081] Fig.21 Shows Fig. 20 Detailed flow chart of joint training matrix mapping model, matrix fusion model and content fusion model;

[0082] Fig. 22 Shows Fig. 20 A specific flow chart of step 2005;

[0083] Fig.23 Shows Fig. 22 A specific structural diagram of the content estimation model in FIG.

[0084] Fig.24 is a flowchart of a content recommendation method according to an embodiment of the present disclosure;

[0085] Fig.25 is a specific flow chart of a content recommendation method according to an embodiment of the present disclosure;

[0086] Fig.26 Shows Fig.24 A specific flow chart of step 2403 in FIG.

[0087] Fig. 27 is a block diagram of an information fusion device according to an embodiment of the present disclosure;

[0088] Fig.28 is a block diagram of a content recommendation device according to an embodiment of the present disclosure;

[0089] Fig.29 is a terminal structure diagram for implementing an information fusion method or a content recommendation method according to an embodiment of the present disclosure;

[0090] Fig.30 It is a server structure diagram for implementing the information fusion method or content recommendation method according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0091] In order to make the purpose, technical solution and advantages of the present disclosure more clear, the present disclosure is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present disclosure and are not used to limit the present disclosure.

[0092] Before further describing the embodiments of the present disclosure in detail, the nouns and terms involved in the embodiments of the present disclosure are described. The nouns and terms involved in the embodiments of the present disclosure are subject to the following interpretations:

[0093] Artificial intelligence: It is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain target results. In other words, artificial intelligence is a comprehensive technology in computer science. Artificial intelligence attempts to understand the essence of intelligence and produce a new intelligent machine that can respond in a similar way to human intelligence. Artificial intelligence is to study the design principles and implementation methods of various intelligent machines so that machines have the functions of perception, reasoning and decision-making. Artificial intelligence technology is a comprehensive discipline that covers a wide range of fields, including both hardware-level technology and software-level technology. The basic technologies of artificial intelligence generally include technologies such as sensors, dedicated artificial intelligence chips, cloud computing, distributed storage, big data processing technology, operation / interaction systems, and mechatronics. Artificial intelligence software technology mainly includes several major directions such as computer vision technology, speech processing technology, natural language processing technology, and machine learning / deep learning. With the research and advancement of artificial intelligence technology, artificial intelligence technology has been studied and applied in many fields, such as common smart homes, smart wearable devices, virtual assistants, smart speakers, smart marketing, unmanned driving, automatic driving, drones, robots, smart medical care, smart customer service, etc. I believe that with the development of technology, artificial intelligence technology will be applied in more fields and play an increasingly important role.

[0094] Content recommendation: is a technology that recommends appropriate content and improves object satisfaction by analyzing multiple aspects such as object information, context information, and content information. Content recommendation is one of the currently widely used personalized recommendation technologies, which aims to provide objects with a better experience and improve the interaction between content providers and objects. Content recommendation usually uses a large amount of data and algorithms to predict the content that the object may be interested in, and adjusts the recommendation strategy to continuously optimize the recommendation effect. Common content recommendation methods include collaborative filtering-based methods, content-based methods, hybrid methods, and deep learning methods. These methods can be applied in multiple fields such as e-commerce, social media, and news media to provide objects with personalized experiences.

[0095] Attention layer: is a common attention mechanism in deep learning, which is used to dynamically select information related to the input to improve model performance. For an input sequence, the attention layer will calculate a weight for each element to indicate its importance in the model. These weights can be used to weight the sum of the elements in the input sequence or as part of the output sequence.

[0096] Content prediction model: A model used to predict the interest of an object in content (such as goods, movies, music, news, etc.) that has not yet been interacted with. It is usually used in recommendation systems. The content prediction model uses machine learning methods to generate a model of possible ratings or probability distributions for uninteracted items based on the object's historical operation data and item features.

[0097] Self-Attention Network: It is a neural network structure based on the attention mechanism. The self-attention network is used to process natural language processing (NLP) tasks such as machine translation and language modeling, and was later applied to computer vision tasks. The main idea of ​​the self-attention network is to use attention weights to calculate the correlation between different positions in the input sequence, and then perform weighted summation on the input sequence based on these correlations to obtain a new vector representation of the sequence.

[0098] Content recommendation is one of the currently widely used personalized recommendation technologies, including product recommendation, advertising recommendation, video recommendation, text recommendation, etc. Among them, content recommendation is to model the changing interests of the target object based on the historical operation data of the target object, so as to make recommendations that meet the interests of the target object. In order to increase the accuracy and richness of content recommendation, the associated information of the browsed content in the historical operation data is used as content auxiliary information, and the content auxiliary information is any content that can assist in recommendation and has a certain relationship with the browsed content. Therefore, by adding auxiliary information to assist the browsing content to generate better representation information, rather than directly changing the representation information of the browsed content, the accuracy of content recommendation can be improved.

[0099] In the related art, before generating representation information from the collected browsing content information, the auxiliary information from multiple sources is fused and input into the multi-head self-attention layer, and the weight of the auxiliary information is calculated by the multi-head self-attention layer, and then the representation information of the content is constructed based on the weight of the auxiliary information and the browsing content information. Since the content auxiliary information from multiple sources is fused before the multi-head self-attention layer, different content auxiliary information must have the same representation dimension to be fused, and the content auxiliary information from multiple sources is fused and then input into the multi-head self-attention layer. When the multi-head self-attention layer adjusts the weights of each input, it can only uniformly adjust the fused content auxiliary encoding information, so that the update gradients of the content auxiliary encoding information from different sources are the same. In this way, the accuracy of the multi-head self-attention layer in allocating weights to different content auxiliary information is reduced, and the content auxiliary information of different dimensions cannot be input, which reduces the richness of the content information representation and reduces the accuracy of content recommendation.

[0100] Description of the system architecture and scenarios for application of the embodiments of the present disclosure.

[0101] Figure 1 1 is a system architecture diagram for the information fusion method according to an embodiment of the present disclosure, which includes a terminal 140, the Internet 130, a gateway 120, a server 110, and the like.

[0102] Terminal 140 is a device that collects historical operation data of the target object, extracts the content information browsed at each time point and the content auxiliary information corresponding to the content information from multiple sources based on the historical operation data, and inserts the content auxiliary information from multiple sources into the content information to obtain new content information. It includes various forms such as desktop computers, laptops, PDAs (personal digital assistants), mobile phones, car terminals, home theater terminals, and dedicated terminals. With the application of the embodiments of the present disclosure in scenarios such as personal emotional stewards, smart driving, home theaters, and home anti-theft alarms as described later, it can be specifically embodied in the form of mobile phones, car terminals, home theater terminals, and anti-theft alarm dedicated terminals. In addition, it can be a single device or a collection of multiple devices. Terminal 140 can communicate with the Internet 130 in a wired or wireless manner to exchange data.

[0103] The server 110 refers to a computer system that can provide certain services to the terminal 140. Compared with ordinary terminals 140, the server 110 has higher requirements in terms of stability, security, performance, etc. The server 110 can be a high-performance computer in a network platform, a cluster of multiple high-performance computers, a part of a high-performance computer (such as a virtual machine), a combination of parts of multiple high-performance computers (such as virtual machines), etc. The server 110 is used in certain application scenarios (such as content recommendation, article recommendation, product recommendation, video recommendation, etc. mentioned later).

[0104] The gateway 120 is also called an internetwork connector or a protocol converter. The gateway 120 realizes network interconnection at the transport layer and is a computer system or device that acts as a converter. The gateway 120 is a translator between two systems that use different communication protocols, data formats or languages, or even have completely different architectures. At the same time, the gateway 120 can also provide filtering and security functions. The message sent by the terminal 140 to the server 110 must be sent to the corresponding server 110 through the gateway 120. The message sent by the server 110 to the terminal 140 must also be sent to the corresponding terminal 140 through the gateway 120.

[0105] The embodiments of the present disclosure can be applied in various scenarios, such as Figures 2A-2C The content shown here recommends the following scenarios: Figure 3A -C The recommended scenario of the article shown in Figure 4A -C shows the recommended scene of the product, Figure 5A -C shows the video recommendation scenes, etc.

[0106] 1. Scenarios for content recommendation

[0107] At present, artificial intelligence has rapidly penetrated every corner of people's production and life. Artificial intelligence is not only used to speed up production efficiency and assist various tasks in life, but also provides various value-added services. At present, in order to increase the frequency of use of the target object, the smart device will make customized recommendations based on the target object's usage records to complete the recommendation according to the changes in the target object's preferences. The disclosed embodiment provides a customized content recommendation function. It can be installed as an application (APP) in the terminal 140 (such as a mobile phone, etc.) of the target object. It actually collects the historical browsing content of the target object in real time, and recommends it based on the historical browsing content and the associated information of the historical browsing content, and updates the currently displayed content information to switch to the recommended content information, so as to recommend the content that the target object is interested in, and realize customized content recommendation.

[0108] like Figure 2A As shown in FIG. 1 , content A, content B, content C, and content D are displayed on terminal 140. When the target object browses the content on terminal 140 and sees the content of interest, he will click to enter and view it. In response to the target object clicking on content C, the interface of terminal 140 jumps to the detailed content of content C. Therefore, the interface of terminal 140 is as follows: Figure 2A Jump to Figure 2BWhen the interface of terminal 140 displays content C, terminal 140 records the click on content C as the browsed content at that time point, and collects the associated information of content C at the same time, wherein the associated information of content C includes: browsed content related information and browsed operation parameters, and the browsed content related information includes content publishing information, similar content, etc., and the browsed operation parameters include: browsed times, browsed duration, etc., and the dimensions of browsed related information and browsed operation parameters may be different. Then the associated information of content C is used as auxiliary information of content C to assist content C in generating more representative distribution information. As the interface of terminal 140 is continuously browsed and content is clicked for viewing, each time a content is clicked to enter, the time point of viewing the content will be recorded, and the associated information of the browsed content at that time point will be obtained as auxiliary information, and the browsed content and auxiliary information will be combined with the time point as the axis to form historical operation data represented by a sequence, and then the historical operation data will be stored in the database or log file in terminal 140.

[0109] like Figure 2C As shown, after the content C and the associated information of the content C are recorded in the terminal 140, the terminal 140 re-recommends the content based on the associated information of the content C and the content C to refresh the interface of the terminal 140. After the refresh, the interface of the terminal 140 displays the content C1, the content C2, the content C3 and the content E, and the content C1, the content C2 and the content C3 are all the content that is re-searched and recommended based on the associated information of the content C and the content C. Therefore, by collecting the browsing content and the auxiliary information of the browsing content at each time point for content recommendation, it is possible to make content recommendations that are more in line with the preferences of the target object.

[0110] (II) Scenarios for article recommendations

[0111] Article reading software generally customizes recommendations based on the preferences of the target audience to provide articles that suit the target audience's preferences. However, traditional article recommendations are too simple, and only make recommendations of the same category based on article categories. For example, if the articles browsed tend to be financial forums, subsequent recommendations will all be articles from financial forums, which limits the reading of articles in other fields, making the article recommendations too single and reducing the target audience's stickiness to the article reading software.

[0112] like Figure 3A As shown, when the main interface of the terminal 140 displays four articles, the four articles are current news and named Topic 1, financial forum and named Topic 2, entertainment news and named Topic 3, and professional knowledge and named Topic 4. When the article of Topic 3 is clicked by the target object, in response to the target object clicking on the article of Topic 3, the interface of the terminal 140 changes from Figure 3A Switch to Figure 3B The interface of terminal 140 enters the detailed interface of theme 3, and the detailed interface of theme 3 is as shown in FIG. Figure 3B As shown. When entering the article of Topic 3, the terminal 140 will collect information associated with Topic 3, and store the articles of Topic 3 browsed at that time point and the associated information of the articles at the same time, and use the associated information of the articles of Topic 3 as auxiliary information for generating distribution information of the articles. Among them, the information associated with Topic 3 includes article associated information and article reading operation parameters, and the article associated information includes other article information of the same topic, other article information of the author of the article, etc., and the article reading operation parameters include reading time, number of times the article is opened, etc.

[0113] After finishing reading the article of Topic 3, when exiting the reading interface and returning to the main interface, the terminal 140 re-recommends articles based on the collected articles of Topic 3 and the auxiliary information, and displays the recommended articles. The interface after completing the article recommendation is as follows: Figure 3C As shown. Figure 3C As shown, the interface of terminal 140 displays entertainment news named Theme 4, entertainment news named Theme 5, real-time news named Theme 3-1, and professional knowledge named Theme 3-2. Articles of Theme 4, Theme 5, and Theme 3 are all entertainment news, and articles of Theme 3-1, Theme 3-2, and Theme 3 are related articles. Therefore, by collecting the related information of browsed articles as auxiliary information, the browsed articles and the auxiliary information of multiple sources and different dimensions are integrated to recommend articles that are more in line with the preferences of the target object. On the premise of recommending articles that meet the preferences of the target object, the recommended articles are also more comprehensive and rich, which can improve the stickiness of the target object to the article reading software, thereby increasing the usage rate of the article reading software.

[0114] (III) Product recommendation scenarios

[0115] As new products are launched and preferences of consumers change, product recommendations not only refer to the browsed product information but also need to add related information of the products to make more accurate recommendations based on product information and related information.

[0116] like Figure 4A As shown in FIG. 1 , after logging into the “XXX website”, the product recommendation interface displays thumbnails of multiple products, and the product information corresponding to the displayed thumbnails includes: product A, product B, product C, product D, product E, product F, product G, product H, product I, and product J. When product G is clicked by the target object, in response to the target object's click on product G, the detailed introduction interface of product G is entered, and the detailed interface of product G is as shown in FIG. Figure 4B Therefore, in response to the click of product G, the interface displayed by terminal 140 is as follows Figure 4A Switch to Figure 4B As shown. Figure 4BAs shown, the interface of the terminal 140 displays a detailed introduction interface of the product G, and the "Add to Cart" or "Buy Now" button can be directly clicked on the interface to complete the purchase of the product. While displaying the detailed introduction interface of the product G, the terminal 140 collects information associated with the product G, and stores the detailed introduction information of the product G browsed at that time point and the associated information of the product G, so as to update the product recommendation based on the product G and the associated information of the product G.

[0117] After completing the browsing of product G and exiting the detailed introduction interface of product G, the product recommendation interface is updated based on the related information of product G and product G. The updated product recommendation interface is as follows Figure 4C As shown. The updated product recommendation interface includes: product G1, product B, product C, product G2, product E, product G3, product G, product H, product G4 and product G5. Product G1, product G2, product G3, product G4 and product G5 are re-recommended products based on product G and the associated information of product G. Therefore, by collecting the associated information of browsed products and re-recommending products based on the browsed product information and the associated information, it is possible to recommend products that are more in line with the preferences of the target object.

[0118] (IV) Video recommendation scenarios

[0119] The video software on the terminal 140 usually displays multiple types of videos, and sets a selected type on the main interface to select several representative videos and place them in the selected area under the selected type. However, the videos placed in the selected area by general video software are all the same, and the content in the area is not personalized according to the preferences of different target objects, so the videos in the selected area cannot be liked by most target objects.

[0120] The information fusion method of the embodiment of the present disclosure is applied to the scenario of video recommendation, and makes more accurate video recommendations based on the fused browsing content and auxiliary information. Figure 5A As shown, the main interface of the terminal 140 displays three selected videos, and the three videos are video A, video B, and video C. When the target object clicks on video B, in response to the target object's click on video B, video B is played, and the playback interface of video B is as shown in FIG. Figure 5B Therefore, in response to a click on video B, the interface of terminal 140 is as follows Figure 5A Switch to Figure 5BAs shown, when entering the playback interface of video B, the associated information of video B is collected while playing video B, and the information of video B at that time point and the associated information of video B are stored as the historical operation data at that time point. If the video information played at multiple time points and the associated information of the video information are collected, the historical operation data of the multiple time points are stored, and the historical operation data represents the video information clicked and viewed at multiple time points and the associated information of the video information in the form of a sequence.

[0121] After watching video B, when exiting the playback interface of video B, video recommendations are made based on the information of video B and the associated information of video B, so as to find out the videos associated with video B and display them in the selected area. Figure 5C As shown, the videos in the selected area include video B1, video B2 and video B3, and video B1, video B2 and video B3 are associated with video B. Therefore, video recommendations are made based on the target object's click and watch video information and the associated information of the video information, so that the video software can formulate personalized video recommendations that meet each target object.

[0122] General description of the disclosed embodiments

[0123] According to an embodiment of the present disclosure, an information fusion method is provided. The information fusion method is applied to the terminal 140, and content recommendation is performed based on the fused information to make recommendations that are more in line with the preferences of the target object.

[0124] like Figure 6 As shown, the information fusion method according to one embodiment of the present disclosure includes:

[0125] Step 601, obtaining historical operation data of the target object; wherein the historical operation data includes: original content encoding information and original auxiliary encoding information of the original content encoding information;

[0126] Step 602, inputting the original content coding information into the matrix mapping model to generate a content correlation matrix, and inputting each original auxiliary coding information into the matrix mapping model to generate an auxiliary correlation matrix;

[0127] Step 603, inputting the content relevance matrix and the auxiliary relevance matrix into a matrix fusion model to obtain a relevance fusion matrix;

[0128] Step 604: input the correlation fusion matrix and the original content encoding information into the content fusion model to obtain target fusion information.

[0129] Steps 601-604 are briefly described below.

[0130] In step 601, the target object is the object operated on the terminal 140, and the historical operation data collected from the target object can be the historical data of the data collection target selected by the target object. The data collection target can be the terminal 140, or a functional software (APP), website and webpage on the terminal 140, or multiple functional software combined on the terminal 140. If a recommendation is made for a certain functional software, the data collection target can also be set to multiple functional software to combine multiple functional software to make a more accurate recommendation. The data collection target is pre-set, and the target functional software or terminal 140 is selected as the data collection target from the candidate functional software. If the data collection target is the terminal 140, it is necessary to collect the log data of all functional software running on the entire terminal 140, and all log data are used as historical operation data. If the data collection target is a certain functional software, it is only necessary to obtain the historical browsing data of one functional software as the historical operation data. Therefore, after determining the data collection target, the historical operation data of the data collection target can be collected in a targeted manner to realize the historical operation data collection of the target object. The historical operation data refers to the activity record or event history of the target object within a period of time. The historical operation data includes the browsing content of a period of time, and the browsing content includes: browsing the network, searching for content, clicking on content, adding to a shopping cart, purchasing goods, etc. The historical operation data is usually stored in a database or a log file. In this embodiment, the historical operation data includes the original content encoding information at each time point and the original auxiliary encoding information of the original content encoding information. The original content encoding information refers to the information after the browsing content information at each time point is encoded, and the original auxiliary encoding information is the information after the content auxiliary information of the browsing content information is encoded. The content auxiliary information of the browsing content information can be any content that can provide useful information for recommendation. For example, any information that is useful for recommendation includes browsing content related information and browsing operation behavior. The browsing content related information can be the creator of the browsing content, the publishing time of the browsing content, the publishing platform of the browsing content, the content type of the browsing content, etc., and the browsing operation behavior can be click, copy, forward, etc. If the historical operation data includes multiple time points, then the historical operation data is a sequence with the time points as the time axis and the original content coding information and the original auxiliary coding information as the specific content, and the historical operation data contains multiple original content coding information, and each original content coding information corresponds to multiple original auxiliary coding information.

[0131] In step 602, the matrix mapping model is a model for mapping the coding information to the correlation matrix. First, each original auxiliary coding information is input into the matrix mapping model to generate an auxiliary correlation matrix, and the dimension of each auxiliary correlation matrix is ​​consistent, so no matter whether the dimension of the original auxiliary coding information is the same, it can be transformed into a correlation matrix of the same dimension, so as to expand the dimensional range of the original auxiliary coding information, so that different content auxiliary information can have different coding dimensions. While each original auxiliary coding information is input into the mapping matrix model, the original content coding information is also input into the matrix mapping model to generate a content correlation matrix, so as to characterize the original content coding information through the content correlation matrix.

[0132] In step 603, the matrix fusion model is used to fuse at least two matrices into one matrix. The dimensions of the content correlation matrix and the auxiliary correlation matrix are the same, so the content correlation matrix and the auxiliary correlation matrix are directly fused through the matrix fusion model to obtain a correlation fusion matrix, so as to represent the fusion of the original content coding information and the original auxiliary coding information through the correlation matrix.

[0133] In step 604, the content fusion model is used to fuse the matrix and the content coding information to form new coding information. By inputting the original content coding information and the correlation fusion matrix into the content fusion model, the fused correlation fusion matrix is ​​integrated into the original content coding information to construct the target fusion information, and the target fusion information is more accurate representation information of the browsing content information. Therefore, according to the correlation between the content auxiliary information and the browsing content information, the auxiliary information from multiple sources is integrated into the original content coding information to obtain more abundant representation information of the browsing content information. Then, content recommendation based on the target fusion information can also recommend content that is more in line with the preferences of the target object, improve the accuracy of content recommendation, and further improve the viscosity of the target object to the recommendation system.

[0134] The advantages of the above steps 601-604 are that after extracting the original content coding information, at least one original auxiliary coding information of the original content coding information is extracted respectively, and each original auxiliary coding information is input into the matrix mapping model to generate an auxiliary correlation matrix separately. Then, the auxiliary coding information of multiple sources does not need to be fused in advance, so there is no need to limit the input original auxiliary coding information to have the same dimension, the dimension selection range of the original auxiliary coding information is increased, and the original auxiliary coding information does not need to be updated uniformly, so that the update gradient of the original auxiliary coding information from different sources can be different, and the original auxiliary coding information can be adjusted in a targeted manner. While generating the auxiliary correlation matrix, the original content coding information is input into the matrix mapping model to generate the content correlation matrix, and then the auxiliary correlation matrix and the content correlation matrix are fused to generate the correlation fusion matrix, and finally the correlation fusion matrix is ​​integrated into the original content coding information, so as to fuse according to the correlation between the original content coding information and different original auxiliary coding information, and construct new content coding information as the target fusion information, so that the representation of the browsing content information is richer, and then the content recommendation based on the target fusion information is more accurate, and the browsing content information that the target object is interested in is recommended, and the viscosity of the target object is improved.

[0135] The above is a brief description of steps 601-604, and these steps are described in detail below.

[0136] Steps 601-604 are described in detail below.

[0137] Step 601 Detailed Description

[0138] In step 601, the historical operation data is the operation data collected in a preset time period, and the preset time period includes multiple time points. The operation data of the target object is recorded periodically at the preset time points and stored in a preset database or log file to obtain the historical operation data. When the historical operation data needs to be called, the preset time period is set in advance, and the historical operation data matching the time point is extracted from the database or log file according to the preset time period to extract the historical operation data that meets the preset time period. There is no need to extract the historical operation data of all time periods, which can make accurate recommendations based on the historical operation data and reduce the duration of data analysis.

[0139] Because there are risks in directly extracting historical operation data, when extracting historical operation data, you need to confirm whether you have the permission to extract historical operation data before you can extract it from the database or log file. Figure 7As shown, after the preset time period, collection time period and data collection target are pre-set, if the data collection terminal 140 is a functional software, after entering the functional software corresponding to the terminal 140, a data acquisition permission request interface will pop up directly, and the content of the data acquisition permission request interface is "Do you agree to obtain historical operation data". If the target object clicks the "Agree" button, the acquisition permission of the historical operation data is configured in response to the click of the "Agree" button, and the historical operation data in the database or log file whose time point is in the preset time period is collected according to the preset collection time period, so as to collect the historical operation data in real time to update the fusion information. If the target object clicks the "Reject" button, the acquisition permission of the historical operation data is not configured in response to the click of the "Reject" button, and the historical operation data in the database or log file cannot be collected. Therefore, the acquisition permission of the historical operation data is determined by setting the data acquisition permission request interface to prevent the arbitrary leakage of historical operation data and improve the security of historical operation data.

[0140] In one embodiment, Figure 8 As shown, step 601 may include:

[0141] Step 801, obtaining candidate auxiliary information of the candidate content information, and obtaining candidate content coding information of the candidate content information and candidate auxiliary coding information of each candidate auxiliary information;

[0142] Step 802, obtaining historical operation data of the target object; wherein the historical operation data includes browsing content information;

[0143] Step 803, selecting original content encoding information from candidate content encoding information based on the browsed content information and the candidate content information;

[0144] Step 804: Select original auxiliary coding information from the candidate auxiliary coding information based on the browsed content information, the candidate content information and the candidate auxiliary information.

[0145] Steps 801-804 are described in detail below.

[0146] In step 801, the candidate content information is the content information that the target object is pre-configured to browse and click to view, which can be pre-stored on the terminal 140, or can be obtained from the server 110 in real time and stored in the terminal 140. After the data collection target of the target object is determined, multiple candidate content information will be pre-set on the data collection target to set multiple candidate content information to be selected for viewing. Candidate auxiliary information is any information that can provide useful information for the representation of candidate content information, and the candidate auxiliary information is divided into two types, one is the relevant information of the candidate content information or the relevant information of the operation of the candidate content information. The associated information that has an auxiliary effect on each candidate content information is obtained, and the associated information of the candidate content information is used as the candidate auxiliary information. Among them, the relevant information of the candidate content information includes content information of the same content type, other content information of the same content author, etc., and the relevant information of the operation of the candidate content information includes: the number of times the candidate content information is browsed, the length of time the candidate content information is browsed, etc. If the target object is a video software, multiple candidate videos will be pre-acquired for selection and playback, and each candidate video has multiple associated contents as candidate auxiliary information. When a video is played, the related information of the currently played video is collected as candidate auxiliary information, and the candidate auxiliary information and the currently played video information are recorded at the same time to construct richer representation information of the video information based on the currently played video information and the candidate auxiliary information to enrich the representation of the video information.

[0147] For example, Figure 9A-9B As shown, the application scenario is video recommendation. When the selected candidate content information is video A information, when playing video A, a video related information acquisition request interface will pop up, and the content of the video related information acquisition request interface is "Do you agree to obtain the related information of video A?". If the target object clicks the "Agree" button, in response to the click of the "Agree" button, the acquisition permission of the related information of video A is obtained, and at the same time, the background of terminal 140 collects the related information of video A as candidate auxiliary information. If the related information of video A needs to be displayed, the interface of terminal 140 is changed from as shown in the figure. Fig. 9A Switch to Fig. 9BAs shown. If the collected related information of video A includes the director information of video A, other video works of the director of video A, similar videos of video A, etc., the related information of video A will be displayed on the interface of terminal 140. In addition to collecting the related information of video A as candidate auxiliary information, the playback information of video A is also obtained as candidate auxiliary information, and the playback information includes: playback duration and number of playbacks. Therefore, the related information and playback information of video A are collected as candidate auxiliary information, and then the original content information is integrated into the candidate auxiliary information to make video recommendations, which is not limited to the recommendation of similar videos of video A, but can also recommend other video works of the same director of video A, making the recommended videos more comprehensive and more accurate, thereby improving the target object's experience of video recommendations.

[0148] If the application scenario is product recommendation, the candidate content information is product information, and the product information is product ID. Relevant information of the product information is obtained as candidate auxiliary information, and the relevant information of the product information is divided into two types, namely product-related information or behavior-related information. Product-related information includes: product price, production date, manufacturer, which is information describing the product itself. Behavior-related information can be purchase actions, rates, execution time, feedback product evaluation scores, etc. Therefore, the product-related information and behavior-related information of the product information are obtained as candidate auxiliary information, and the candidate auxiliary information is used to assist the product information in generating richer representation information, so that product recommendations based on the representation information are more accurate.

[0149] After determining the candidate auxiliary information of each candidate content information, the candidate content encoding information is the information after encoding the candidate content information, and the candidate auxiliary encoding information is the information after encoding the candidate auxiliary information. In this embodiment, the embedding table of the candidate content information is obtained as the candidate content encoding information, and the embedding table of the candidate auxiliary information is obtained as the candidate auxiliary encoding information. Among them, the embedding table is also called an embedding table, and the embedding table is represented as a two-dimensional matrix for storing embedded vectors in a neural network. The embedding table is often used to encode discrete input features into continuous low-dimensional vector representations, that is, to encode the content features in the candidate content information into low-dimensional vector representations as candidate content encoding information, and to encode the candidate auxiliary information into low-dimensional vector representations as candidate auxiliary encoding information.

[0150] After the target object is selected, in step 802, the historical operation data of the target object is extracted from the database or log file. The historical operation data includes browsing content information at multiple time points. The browsing content information is the content browsed by the target object at a certain time point. The historical operation data is represented in the form of a sequence, so the historical operation data is also a historical behavior sequence. The historical operation data is represented as {i 1 ,i 2 ,…,i n},i n It represents the browsing content information at the nth time point. Fig.10 As shown, if the target object of data collection is a video software, enter the historical viewing record interface of the video software, and view the historical viewing records of this morning and yesterday through the historical viewing record interface. In addition, you can continue to obtain historical viewing records of other time periods and collect historical viewing records as historical operation data. Then, the browsing content information at each time point in the historical operation data is the viewing video information. For example, if the collection time is one week, then the historical viewing records from the current to the previous week are obtained as historical operation data to achieve targeted historical operation data acquisition.

[0151] After the historical operation data is determined, it is necessary to further determine the content encoding information corresponding to each browsing content information in the historical operation data. In step 803, the browsing content information and the candidate content information are first matched to find the candidate content information that is the same as the browsing content information as the selected content information, and then the candidate content encoding information corresponding to the selected content information is used as the original content encoding information, so that the original content encoding information can be easily obtained. For example, if the historical operation data is {i 1 ,i 2 ,…,i n}, the historical operation data includes browsing content information at multiple time points, and the original content encoding information corresponding to the browsing content information at each time point is determined to be Therefore, the corresponding candidate content encoding information is determined by the browsing content information in the historical operation data as the original content encoding information E i As shown in formula (1):

[0152]

[0153] For example, if the browsing content information at a certain time point is content i 2 , then the candidate content information is also obtained for content i 2 , and obtain the corresponding candidate content encoding information as Also coming soon As the original content encoding information.

[0154] After the original content coding information is selected, in step 904, the candidate auxiliary coding information is screened based on the browsing content information, the candidate content information and the candidate auxiliary information to screen out the candidate auxiliary coding information corresponding to each original content coding information as the original auxiliary coding information. Each browsing content information corresponds to a plurality of content auxiliary information, so each original content coding information corresponds to a plurality of original auxiliary coding information.

[0155] The advantage of the above steps 801-804 is that all the candidate content coding information of the entire target object is first determined, and then the candidate auxiliary coding information of each candidate content coding information is obtained. After the browsing content information at each time point is obtained, the original content coding information is screened out from the candidate content coding information based on the browsing content information, and then the original content auxiliary coding information is screened out from the candidate auxiliary coding information to determine the original content coding information and the original auxiliary coding information corresponding to the browsing content information, so that the original content coding information and the original auxiliary coding information are easy to obtain. At the same time, by using a two-dimensional matrix to represent the browsing content information and the content auxiliary information, the content auxiliary information of different dimensions can be represented in a matrix form, which is convenient for the subsequent fusion of the content auxiliary information of different dimensions.

[0156] The following is a detailed description of step 804.

[0157] After the original content coding information corresponding to the browsing content information is selected, it is necessary to further obtain the original auxiliary coding information corresponding to the browsing content information to serve as data for subsequent calculation of the correlation between the browsing content information and the auxiliary information.

[0158] In one embodiment, please refer to Fig.11 , step 804 includes:

[0159] Step 1101, based on the browsed content information, select selected content information from the candidate content information; wherein the selected content information is the same as the browsed content information;

[0160] Step 1102: Select original auxiliary coding information from candidate auxiliary coding information based on the selected content information and the candidate auxiliary coding information.

[0161] Steps 1101 - 1102 are described in detail below.

[0162] In step 1101, the candidate content information is searched according to the browsing content information to find out the candidate content information that is the same as the browsing content information as the selected content information. Among them, the selection of the selected content information needs to go through three steps. In the first step, the content type of the browsing content information is first obtained as the browsing type, the content type of the candidate content information is obtained as the candidate type, the selected content information is first screened out from the candidate content information according to the browsing type and the candidate type, and then the selected content information is screened out from the selected content information based on the browsing content information. Therefore, by first screening according to the content type, the candidate content information of the same content type can be quickly located, and then the selected content information that is the same as the browsing content information is searched in a one-to-one manner as the selected content information, and the efficiency of searching the selected content information can be improved by using a hierarchical manner to search for the selected content information.

[0163] After the selected content information is selected, in step 1102, candidate auxiliary information corresponding to the selected content information is obtained as the selected auxiliary information, and then candidate auxiliary coding information corresponding to the selected auxiliary information is obtained as the original auxiliary coding information, so that the original auxiliary coding information can be easily obtained.

[0164] For example, if the selected content information is {i 1 ,i 2 ,…,i n}, each selected content information corresponds to p candidate auxiliary information, then the p candidate auxiliary information is the selected auxiliary information, and the selected auxiliary information is recorded as Represents the xth candidate auxiliary information at the yth time point. Then, based on the selected auxiliary information, the original auxiliary coding information is filtered out from the candidate auxiliary coding information, and the original auxiliary coding information is recorded as Original auxiliary coding information As shown in formula (2):

[0165]

[0166] The advantages of the above steps 1101-1102 are that the candidate content information identical to the browsed content information is first searched out as the selected content information, then the candidate auxiliary information of the selected content information is obtained as the selected auxiliary information, and finally the candidate auxiliary coding information corresponding to the selected auxiliary information is used as the original auxiliary coding information. Therefore, by searching for the original auxiliary coding information in a hierarchical manner, the original auxiliary coding information is easily obtained.

[0167] After the original content coding information and the original auxiliary coding information are obtained, the correlation between each auxiliary information and the browsing content information needs to be calculated, so as to use the correlation as a reference for how the auxiliary information from multiple sources is integrated into the original content coding information. Therefore, the correlation matrix of the original content coding information and the original auxiliary coding information needs to be calculated respectively, so as to serve as a matrix for measuring the linear correlation between each auxiliary information and the browsing content information.

[0168] Detailed description of step 602

[0169] like Fig.12 As shown, step 602 includes:

[0170] Step 1201, inputting the original content coding information into the matrix mapping model, and the matrix mapping model maps the original content coding information to a preset content parameter matrix to generate a content correlation matrix;

[0171] Step 1202, input each original auxiliary coding information into a matrix mapping model, and the matrix mapping model maps each original auxiliary coding information to a preset auxiliary parameter matrix to generate an auxiliary correlation matrix; wherein each auxiliary correlation matrix has the same dimension.

[0172] The correlation matrix is ​​a very commonly used technique in data analysis and machine learning, and each element of the correlation matrix is ​​the Pearson correlation coefficient of the corresponding feature, and the Pearson correlation coefficient is a value between -1 and 1, representing the degree of correlation between two features. When the two features are completely independent, the Pearson correlation coefficient is 0. When the two features are positively correlated, the Pearson correlation coefficient is a positive value, and the larger the value, the stronger the correlation; when the two features are negatively correlated, the Pearson correlation coefficient is a negative value, and the smaller the value, the stronger the correlation. Therefore, by constructing the content correlation matrix and the auxiliary correlation matrix, it is convenient to analyze the correlation between content auxiliary information and browsing content information based on the content correlation matrix and the auxiliary correlation matrix.

[0173] In step 1201, the preset content parameter matrix is ​​three randomly initialized global vectors, and then the three global vectors are trained. By inputting the original content encoding information into the matrix mapping model, the original content encoding information is mapped to the content parameter matrix, that is, the original content encoding information is mapped to the three global vectors. The original content encoding information is recorded as E i , and the content parameter matrix corresponding to the original content encoding information is The specific calculation formula for mapping the original content encoding information to the content parameter matrix is ​​shown in formula (3):

[0174]

[0175] Among them, Mi is the content relevance matrix, d i is the proportional factor, Q i , K i is a derived component of the query,key.,The content relevance matrix of the original content encoding information is calculated,through formula (3), so as to calculate the relevance between the browsing content information and the,content auxiliary information at each time point according to the content relevance matrix.

[0176] In step 1202, the preset auxiliary parameter matrix is ​​also three randomly generated global vectors, and the auxiliary parameter matrix is ​​obtained after the three global vectors are trained. By inputting each original auxiliary coding information into the matrix mapping model, the original auxiliary coding information is mapped to the auxiliary parameter matrix, that is, the original auxiliary coding information is mapped to the three global vectors through the matrix mapping model to generate an auxiliary correlation matrix. Each auxiliary correlation matrix is ​​independent, and the dimensions of each auxiliary correlation matrix are the same. If the original auxiliary coding information is recorded as The auxiliary parameter matrix is The original auxiliary coding information is mapped to the auxiliary parameter matrix as shown in formula (4):

[0177]

[0178] in, is the auxiliary correlation matrix, is the derived component of the query,key, is a proportional factor. By inputting each original auxiliary coding information and the auxiliary parameter matrix into formula (4) for calculation, an auxiliary correlation matrix corresponding to each original auxiliary coding information is obtained, so as to characterize the correlation between each original auxiliary coding information and the original content coding information at each time point through the auxiliary correlation matrix. Then, after determining the auxiliary correlation matrix, the auxiliary correlation matrix and the content correlation matrix can be fused to realize the fusion of the original content coding information and the original auxiliary coding information, so as to better integrate the content auxiliary information into the browsing content information according to the correlation matrix between each content auxiliary information and the original content information at each time point.

[0179] like Fig.13 As shown, the original content encoding information is recorded as E i =ε i [i 1 ,i 2 ,…,i n ], each auxiliary coding information is recorded as The original content encoding information is mapped according to the content parameter matrix to form an n*n dimensional content correlation matrix M i At the same time, each original auxiliary coding information is mapped to the auxiliary parameter matrix to obtain an auxiliary correlation matrix of n*n dimensions By respectively inputting the original auxiliary coding information of different dimensions into the matrix mapping model to generate a correlation matrix, the separation and decoupling of multi-source auxiliary information is achieved. There is no need to fuse the original auxiliary coding information of multiple sources in advance, so the input content auxiliary information is no longer restricted to the same dimension. The input original auxiliary coding information can have different dimensions, and different original auxiliary coding information updates also have different gradients. Therefore, the adjustment of the original auxiliary coding information can also be adjusted in a targeted manner, making the update of the original auxiliary coding information more accurate.

[0180] In one embodiment, the multi-head self-attention layer will cause a low-rank bottleneck problem. Among them, the low-rank bottleneck problem is caused by the mapping of the attention head in a low-dimensional space, which leads to a low-rank bottleneck problem in the multi-head self-attention layer, which is not conducive to the information expression of the attention mechanism. At the same time, the multi-head self-attention layer needs to continuously calculate the weight of each content auxiliary information, which requires a large amount of calculation. In this embodiment, in order to prevent the low-rank bottleneck problem, and to reduce the amount of calculation and prevent overfitting problems, the matrix mapping model of the embodiment of the present application adopts a single-head attention network with a fixed mapping dimension. The original content coding information and the original auxiliary coding information are mapped to a matrix of fixed dimensions through the single-head attention network, which simplifies the amount of calculation of mapping the original content coding information and the original auxiliary coding information to the matrix, and solves the low-rank bottleneck problem caused by the multi-head self-attention layer.

[0181] The single-head attention network is a neural network based on the attention mechanism. The structure of the single-head attention network is as follows: Fig.14 As shown, the single-head attention network includes a first fully connected layer, a first attention layer, a first normalization layer, a feedforward neural network and a second normalization layer. Among them, the specific process of mapping the original content coding information to the content parameter matrix through the single-head attention network is: the input original content coding information is mapped into three vectors through the first fully connected layer, and the three vectors are query vector, key vector and value vector. The first attention network calculates the dot product between the query vector and the key vector for the original content coding information at each time point, normalizes the dot product through the first normalization layer to obtain the attention weight of the original content coding information at the corresponding position, and then weighted sums the value vector at the position through the feedforward neural network and the second normalization layer to obtain a new vector representation to obtain a content relevance matrix, so that the content relevance matrix is ​​easy to generate. In addition, the process of generating the auxiliary relevance matrix from the original auxiliary coding information is the same as the specific process of generating the content relevance matrix from the original content coding information, which will not be repeated here.

[0182] The advantage of the above steps 1201-1202 is that the content correlation matrix is ​​obtained by mapping the original content coding information into a content parameter matrix of fixed dimension, and each original auxiliary coding information is mapped into an auxiliary parameter matrix of fixed dimension to obtain an auxiliary correlation matrix, so that the original content coding information and the original auxiliary coding information are represented by a correlation matrix of unified dimension, so as to calculate the correlation between the browsing content information and the auxiliary information at each time point according to the content correlation matrix and the auxiliary correlation matrix.

[0183] After the content relevance matrix and the auxiliary relevance matrix are calculated, the content relevance matrix and all the auxiliary relevance matrices need to be merged. i ∈R n×n , there are p auxiliary correlation matrices, and the dimensions of the content correlation matrix and the p auxiliary correlation matrices are the same, so it is necessary to fuse the p+1 correlation matrices, that is, to stack the p+1 correlation matrices.

[0184] Detailed description of step 603.

[0185] like Fig.15 As shown, step 603 includes:

[0186] Step 1501, inputting the content relevance matrix and the auxiliary relevance matrix into a matrix fusion model, and calculating the first weight of the content relevance matrix and the second weight of each auxiliary relevance matrix through the matrix fusion model;

[0187] Step 1502: Based on the first weight and the second weight, the content relevance matrix and the auxiliary relevance matrix are weightedly fused to obtain a relevance fusion matrix.

[0188] Inputting the content correlation matrix and multiple auxiliary correlation matrices into the matrix fusion model for fusion does not simply superimpose the content correlation matrix and multiple auxiliary correlation matrices, but fuses p+1 correlation matrices in a weighted summation manner to construct a more representative correlation fusion matrix.

[0189] The key point of the fusion of the content relevance matrix and the auxiliary relevance matrix is ​​how to assign weights, so the matrix weights are learned based on the content relevance matrix and the auxiliary relevance matrix based on the matrix fusion model, so as to assign high weights to the relevance matrix with strong relevance and assign small weights to the relevance matrix with weak relevance. In step 1501, the first weight is assigned to the content relevance matrix through the matrix fusion model, and the second weight is assigned to each auxiliary relevance matrix, and the second weights of different auxiliary relevance matrices are different. Because the content relevance matrix is ​​a matrix corresponding to the browsing content information at multiple time points, the auxiliary relevance matrix is ​​a matrix corresponding to the time point, and the auxiliary relevance matrix is ​​also a matrix formed by the combination of the relevance data of the same auxiliary information and the browsing content information, the relevance between the browsing content information at different time points and the auxiliary information of the same content can be determined according to the specific data in the auxiliary relevance matrix, and the second weight value of each auxiliary relevance matrix can be determined according to the relevance between the browsing content information at different time points and the auxiliary information of the same content. If the auxiliary relevance matrix includes the feature relevance between the auxiliary information of the same content and the browsing content information at different time points, then when assigning the second weight of each auxiliary relevance matrix, it is necessary to assign it according to the feature relevance, so as to realize that the auxiliary information of different content has different importance in the relevance fusion matrix. If the auxiliary correlation matrix is ​​represented by a sequence, then when assigning the second weight to the same auxiliary correlation matrix, the average of multiple feature correlations can be obtained, and then the second weight can be assigned according to the average correlation of each auxiliary correlation matrix to highlight the importance of auxiliary information of different contents. That is, in the representation learning process of the object, different weights can be assigned to different auxiliary information according to the importance of different auxiliary information, so that the learned features of the object (i.e., target fusion information) are more accurate.

[0190] In step 1502, the content relevance matrix and the auxiliary relevance matrix are weightedly fused based on the first weight and multiple second weights, so as to weight the relevance matrices of different channels and fuse different content auxiliary information and browsing content information. Therefore, the content relevance matrix and the auxiliary relevance matrix are fused in a weighted fusion manner, so that different content auxiliary information has different importance, so that the content encoding information fused based on the relevance fusion matrix is ​​more accurate. By fusing multiple relevance matrices in a weighted manner, the dimension of the generated relevance fusion matrix is ​​the same as the dimension of each auxiliary relevance matrix, reducing the amount of calculation in the subsequent relevance fusion matrix processing process.

[0191] In the process of fusing multiple auxiliary correlation matrices and the content correlation matrix, a matrix fusion model is used. In this embodiment, the matrix fusion model can be used as follows: Fig.16The SE module shown in the figure is generally called Squeeze-and-Excitation Networks. The SE module mainly includes two processes, one is the entire global feature, and the other is to learn the weights through the branches and apply them to the original input to score each channel. Therefore, after integrating the content relevance matrix and the auxiliary relevance matrix through the SE module, the first weight is assigned to the content relevance matrix, and the second weight is assigned to each auxiliary relevance matrix. Fig.16 As shown, the content correlation matrix of n*n dimensions and each of the auxiliary correlation matrices of n*n dimensions are input into the matrix fusion model, that is, input into the SE module. Then the dimension of the correlation matrix input into the SE module is C*n*n, because the content correlation matrix and p auxiliary correlation matrices are combined to obtain p+1 correlation matrices, so C=p+1. Therefore, the number of channels of the SE module is determined to be C=p+1, so the SE module needs to score the correlation matrices of p+1 channels, that is, assign weights. After the SE module assigns weights to the correlation matrices of each channel and then fuses the correlation matrices, the size of the correlation matrix is ​​reduced, so the SE module finally outputs a correlation fusion matrix of n*n dimension.

[0192] In this embodiment, the matrix fusion model adopts Fig.16 SE module, but no specific limitation is made to the matrix fusion model. In other embodiments, the matrix fusion model may include but is not limited to fusion algorithms such as Gating, Adding, and Concate. Among them, the Gating fusion algorithm is a feature fusion algorithm based on a gating mechanism, which aims to organically integrate information from multiple different feature sources to improve the expressiveness of feature representation. The Adding fusion algorithm is a multimodal feature fusion algorithm based on addition operation, which aims to effectively fuse features of multiple different modes to improve the performance of the model. The Concate fusion algorithm is a multimodal feature fusion algorithm based on matrix connection operation. The Concate fusion algorithm connects feature matrices of different modes by columns to form a new feature matrix. Therefore, in addition to the SE module, other fusion algorithms can be selected according to needs to fuse the content correlation matrix and the auxiliary correlation matrix to construct a more representative correlation fusion matrix.

[0193] The advantage of the above steps 1501-1502 is that different weights are assigned to the content relevance matrix and the auxiliary relevance matrix, and then the content relevance matrix and the auxiliary relevance matrix are weightedly fused, so that different content auxiliary information has different importance, so as to generate a representative relevance fusion matrix, making the representation information of the browsing content information richer.

[0194] Detailed description of step 1501.

[0195] In this embodiment, the matrix fusion model adopts the SE module, and the SE module includes: a convolution layer, a pooling layer, a fully connected layer and an activation function layer; please refer to Fig.17 , step 1501 includes:

[0196] Step 1701, input the content relevance matrix into the convolution layer to obtain content relevance features, and input each auxiliary relevance matrix into the convolution layer to obtain auxiliary relevance features;

[0197] Step 1702, input the content correlation feature into a pooling layer to obtain a first vector, and input the auxiliary correlation feature into a pooling layer to obtain a second vector;

[0198] Step 1703, input the first vector to a fully connected layer to obtain a third vector, and input each second vector to a fully connected layer to obtain a fourth vector;

[0199] Step 1704, input the third vector to the activation function layer to obtain the first weight of the content relevance matrix, and input the fourth vector to the activation function layer to obtain the second weight of each auxiliary relevance matrix.

[0200] In step 1701, the convolution layer is the core component of the convolutional neural network, which is used to extract local features in the data. In this embodiment, the convolution layer is used to extract the features of the content correlation matrix and the features of the auxiliary correlation matrix. The content correlation matrix is ​​convolved by the convolution layer to extract the features in the content correlation matrix to obtain the content correlation features. At the same time, the convolution layer also convolves each auxiliary correlation matrix to extract the features of each auxiliary correlation matrix to obtain the auxiliary correlation features. The content correlation matrix and each auxiliary correlation matrix are respectively extracted by the convolution layer to determine the features of each correlation matrix.

[0201] After completing the feature extraction of the correlation matrix, in step 1702, a pooling layer is used to perform pooling processing on the content correlation features and the auxiliary correlation features respectively. The pooling layer is also an important component of the convolutional neural network, and usually follows the convolution layer to perform dimensionality reduction and feature extraction on the output of the convolution layer. In this embodiment, the pooling layer is used to extract the key information of the area in a certain area, that is, to extract the key features of the content correlation features and the auxiliary correlation features to achieve dimensionality reduction. The pooling layer is used to extract the key features from each correlation matrix, that is, to extract the key features from the content correlation features to obtain the first vector, and to extract the key features from each auxiliary correlation feature to obtain the second vector. Therefore, by using the first vector to represent the key features of the content correlation features and the second vector to represent the key features of the auxiliary correlation features, the feature quantity output by the convolution layer is reduced, the subsequent feature quantity calculation can be reduced, and the accuracy of the feature calculation can be ensured.

[0202] In step S1703, two fully connected layers are set, and the two fully connected layers combine the key features of the content correlation features and the auxiliary correlation features in this embodiment. Therefore, the two fully connected layers first map the first vector to a low-dimensional matrix, and then map it back to obtain the third vector. Among them, the dimension of the first vector is reduced to 1 / r through the first fully connected layer, and then raised back to the original dimension through the second fully connected layer. The third vector has more nonlinearity, which can better fit the complex correlations between different channels and greatly reduce the amount of parameter calculation. At the same time, the second vector is also first reduced in dimension through the first fully connected layer, and then restored in dimension through the second fully connected layer to obtain the fourth vector. The fourth vector can also have more nonlinearity, which can better fit the complex correlations between channels. At the same time, it can also greatly reduce the amount of parameters and calculations.

[0203] After the third vector and the fourth vector are output through the second fully connected layer, in step 1704, the activation function layer is used to output the vector as a value of 0-1. The third vector is normalized by the activation function layer to obtain a first weight between 0 and 1, that is, to determine the first weight of the content relevance matrix channel. The activation function layer also performs normalization calculations on the second vector to obtain a second weight between 0 and 1, that is, to determine the second weight of each auxiliary relevance matrix channel, to determine how much attention is placed on each channel, to achieve weight distribution of each relevance matrix, so that different auxiliary information has different importance. Therefore, the auxiliary relevance matrix and the content relevance matrix are fused according to the weights, that is, the auxiliary information and browsing content information of different importance are fused to obtain a more accurate relevance fusion matrix.

[0204] In addition, the activation function layer uses the Sigmoid function to calculate the third vector to obtain a value of 0-1 as the first weight, and uses the Sigmoid function to calculate the multiple fourth vectors to obtain a value of 0-1 as the second weight. In other embodiments, the activation function layer can use the tanh function and the ReLu function. In this embodiment, the specific function of the activation function layer is not specifically limited, so as to select the corresponding activation function according to the data calculation requirements.

[0205] For example, please refer to the logic flow of the SE module Fig.18 , Fig.18The composition structure of the SE module is shown, and the output dimension is next to each level. The content correlation matrix and p auxiliary correlation matrices are input into the input layer together, and the input layer is also the convolution layer. The input layer outputs a correlation matrix with a dimension of n*n*C to the pooling layer, and then each correlation matrix is ​​compressed into a scalar through the pooling layer to obtain a vector with a dimension of 1*1*C. If the hyperparameter is set to r, the dimension of the vector is reduced to 1 / r of the input through the first fully connected layer, and then activated by the activation layer and input to the second fully connected layer to rise back to the original dimension. Therefore, after reducing the dimension through the first fully connected layer and then rising back, the complex correlation between different channels can be better fitted, and the amount of parameters and calculations can be greatly reduced. Then, it is input to the activation function layer, that is, the output vector is calculated by the Sigmod function to output weights between 0 and 1. Finally, the normalized weights are added to the features of each channel through a size reduction operation. That is, the content correlation matrix and the auxiliary correlation matrix are weightedly fused according to the weights to obtain a correlation fusion matrix. This is to achieve the fusion of the content correlation matrix according to the importance of different auxiliary correlation matrices, and to construct a more accurate correlation fusion matrix.

[0206] Detailed description of step 603

[0207] After the content relevance matrix and the auxiliary relevance matrix are fused, a relevance fusion matrix is ​​obtained, which represents the relevance between each content auxiliary information and the browsing content information. In step 703, the relevance fusion matrix and the original content encoding information are input into the content fusion model, and the relevance fusion matrix is ​​integrated into the original content encoding information through the content fusion model to adjust the original content encoding information and reconstruct the encoding information of the browsing content information, and the fusion information output by the content fusion model is defined as the target fusion information. The target fusion information is also the new representation information of the browsing content information, which makes the representation information of the browsing content information richer.

[0208] The correlation fusion matrix is ​​denoted as M∈R n×n , the original content encoding information is recorded as E i , by multiplying the correlation fusion matrix and the original content coding information, the fused correlation fusion matrix can be integrated into the original content coding information, and the fusion process is shown in formula (5):

[0209]

[0210] Among them, R is the target fusion information. Therefore, according to the correlation between different content auxiliary information and browsing content information, the content auxiliary information of different dimensions is integrated into the browsing content information to construct the target fusion information. Among them, each content auxiliary information is first converted into an auxiliary correlation matrix, and the browsing content correlation matrix is ​​converted into a content correlation matrix. Then, the auxiliary correlation matrix and the content correlation matrix are weighted and fused to construct a correlation fusion matrix. The correlation fusion matrix determines the degree of correlation between each content auxiliary information and the browsing content information at different time points, so the correlation fusion matrix is ​​integrated into the original content encoding information to obtain the target fusion information. By using the target fusion information as a richer representation information of the browsing content information, rather than simply constructing the representation information based on the browsing content information, content recommendation based on the target fusion information can recommend more accurate content.

[0211] like Fig.19 As shown, the original content coding information is input into the matrix mapping model to generate a content correlation matrix, each original auxiliary coding information is input into the matrix mapping model to generate an auxiliary correlation matrix, and then the content correlation matrix and the auxiliary correlation matrix are fused through the SE module to obtain a correlation fusion matrix after reducing the size. Finally, the correlation fusion matrix and the original content coding information are multiplied to integrate the correlation fusion matrix into the original content coding information to obtain the target fusion information R.

[0212] Before actually using the matrix mapping model, the matrix fusion model and the content fusion model, they need to be trained first, so that the matrix mapping model, the matrix fusion model and the content fusion model can be jointly trained to fuse multi-source content auxiliary information into the browsing content information to obtain accurate fusion information, the trained fusion information can be accurately estimated, and accurate content recommendations can be made based on the fusion information.

[0213] The following describes in detail the training process of the matrix mapping model, matrix fusion model and content fusion model.

[0214] like Fig. 20 As shown, the matrix mapping model, matrix fusion model and content fusion model are jointly trained in advance through the following process:

[0215] Step 2001, obtaining training sample data; wherein the training sample data includes: historical behavior sample data, the historical behavior sample data includes sample content coding information and sample auxiliary coding information of the sample content coding information;

[0216] Step 2002, inputting sample content coding information into a matrix mapping model to generate a sample content matrix; inputting sample auxiliary coding information into a matrix mapping model to generate a sample auxiliary matrix;

[0217] Step 2003, inputting the sample content matrix and the sample auxiliary matrix into the matrix fusion model to obtain a sample fusion matrix;

[0218] Step 2004, inputting the sample fusion matrix and the sample content encoding information into the content fusion model to obtain sample fusion information;

[0219] Step 2005, inputting the sample fusion information into the content estimation model to obtain content estimation information;

[0220] Step 2006, constructing loss data based on preset content verification information and content estimation information;

[0221] Step 2007, adjusting the parameters of the matrix mapping model, the matrix fusion model, and the content fusion model based on the loss data.

[0222] Steps 2001-2007 are described in detail below.

[0223] The training sample data of step 2001 includes multiple historical behavior sample data. The more historical behavior sample data there are, the better the training effect. Each historical behavior sample data in the training sample data includes sample content coding information browsed at each time point and sample auxiliary coding information of the sample content coding information. The sample content coding information is the sample content information after encoding, and the sample auxiliary coding information is the information after the sample auxiliary information of the sample content information is encoded. By obtaining the sample content information browsed at each time point and obtaining the sample auxiliary information of the sample content information, the sample auxiliary information is any content that can provide additional useful information for recommendation. After obtaining the sample content information and the sample auxiliary information, the embedding table of the sample content information is obtained to obtain the sample content coding information, and the embedding table of the sample auxiliary information is obtained to obtain the sample auxiliary coding information.

[0224] In addition, each historical behavior sample data contains sample content information at multiple time points, so one historical behavior sample data represents multiple sample content information in sequence. There are f sample auxiliary information for each sample content information, so there are also f sample auxiliary coding information for each sample content coding information. After constructing richer content coding information based on sample content coding information and sample auxiliary coding information, it is necessary to determine whether the content coding information is accurate in content estimation, and to label each historical behavior sample data in advance to indicate that new content information will be estimated based on the historical behavior sample data, and to use the new content information as content verification information, and to determine whether the fused information can accurately make content estimation through the difference between the content information identified by the content verification information and the new content coding information. In this embodiment, labeling can be done manually, or it can be done by unsupervised learning to manually label a part of the historical behavior sample data, and then automatically label other unlabeled historical behavior sample data through an unsupervised learning model to save manpower for manual labeling.

[0225] Then, in step 2002, the sample content coding information in each historical behavior sample data is input into the matrix mapping model to generate a sample content matrix, and the sample auxiliary coding information in the historical behavior sample data is input into the matrix mapping model to generate a sample auxiliary matrix. The sample content matrix is ​​a sequence of sample content information converted into binary numbers in order, which represents the sample content information browsed at each time point. The sample auxiliary matrix contains a sequence of each sample auxiliary information converted into binary numbers, which represents the correlation between the same sample auxiliary information and the sample content information at different time points, so the correlation between the same sample auxiliary information and the sample content information at different time points is characterized in digital form. Therefore, by using digital form to characterize the sample content information and sample auxiliary information at each time point, so as to facilitate the integration of the sample content information according to the importance of the sample auxiliary information based on the sample content matrix and the sample auxiliary matrix. The process of step 2002 is similar to step 602, except that the sample content coding information and the sample auxiliary coding information are input here for training the matrix mapping model to adjust the content parameter matrix and the auxiliary parameter matrix in the matrix mapping model so that the matrix mapping model can generate a more accurate correlation matrix.

[0226] Next, in step 2003, the sample content matrix and the sample auxiliary matrix are input into the matrix fusion model, so as to fuse the sample content matrix and the sample auxiliary matrix through the matrix fusion model to obtain the sample fusion matrix. Here, step 2003 is similar to step 603, except that the matrices input here are the sample content matrix and the sample auxiliary matrix generated in the model training. And what is input into the matrix fusion model in step 603 is the auxiliary correlation matrix and the content correlation matrix when the actual model is used. Therefore, in the training process, for each historical behavior sample data, the matrix fusion model generates a sample fusion matrix.

[0227] Next, in step 2004, the fused sample fusion matrix and sample content coding information are input into the content fusion model, and the sample fusion matrix is ​​integrated into the sample content coding information through the content fusion model to obtain sample fusion information, so as to obtain more abundant representation information of the sample content information. Step 2004 here is similar to step 604, except that what is input into the content fusion model here is the sample fusion matrix and sample content coding information generated in the model training, and what is input into the content fusion model in step 604 is the correlation fusion matrix and original content coding information generated in the use of the model. For each historical behavior sample data, the content fusion model will generate a sample fusion information of a new representation of the sample content information, and the sample fusion information incorporates the sample auxiliary information, rather than the representation information produced solely by the features in the sample content information, thereby enriching the representation information of the sample content information.

[0228] Next, in step 2005, the sample fusion information newly represented by the sample content information is input into the content prediction model, and the sample fusion information is content-predicted by the content prediction model to predict the content information that can be currently recommended to obtain the content prediction information. The content prediction information can be used to analyze whether the sample fusion information that integrates multiple sample auxiliary information can be accurately predicted to predict the current content information, that is, to make a content prediction at the current time point based on the historical operation data. Among them, the content prediction information is represented by a sequence, and the sequence contains the probability value of each sample content information. The predicted content information can be intuitively determined by the probability value of each sample content information in the sequence.

[0229] Next, in step 2006, each historical behavior sample data is pre-labeled, and the label is the content verification information corresponding to the historical behavior sample data, and the content verification information is represented by a sequence, including the probability value of each sample content information. Loss data is obtained by calculating the loss between the content verification information and the content estimation information. According to the loss data, it can be analyzed whether the sample fusion information after fusion of the sample auxiliary information can accurately make content estimation.

[0230] The loss data is constructed based on the content verification information and the content prediction information, and in this embodiment, the cross entropy loss function is used to construct the loss data. In other embodiments, mean square error, square error loss, regularization loss, etc. can be used, and in this embodiment, there is no specific restriction on the function used for loss calculation. The process of calculating the loss between the content verification information and the content prediction information through the cross entropy loss function is shown in formula (6):

[0231]

[0232] Among them, Loss is the loss data, y k is the content verification information, s k The content verification information and the content prediction information are input into formula (6) to perform cross entropy calculation to obtain the loss data, which makes the loss data calculation simple.

[0233] After the loss data is constructed, the parameters of the matrix mapping model, the matrix fusion model, and the content fusion model can be adjusted based on the loss data. Specifically, a first threshold value can be set in advance, and when the loss data is less than the first threshold value, the training process ends. When the loss data is greater than or equal to less than the threshold value, continue to adjust the parameters of the matrix mapping model, the matrix fusion model, and the content fusion model until the loss data is less than the first threshold value. Alternatively, the number of training times is set in advance, and the parameters of the matrix mapping model, the matrix fusion model, and the content fusion model are adjusted once each time the loss data is calculated, until the number of parameter adjustments reaches the number of training times to complete the training. When setting the number of training times, the number of training times shall not be less than 300 times, and the higher the number of training times, the better the training effect of the model, and the more accurate the content estimation. The parameters of the matrix mapping model, the matrix fusion model, and the content fusion model refer to the following: Fig.14 , Fig.18 and Fig.19 The weights of each layer in the matrix fusion model are adjusted. For the matrix mapping model, not only the parameters of the first fully connected layer, the first attention layer, the first normalization layer, the feedforward neural network, and the second normalization layer are adjusted, but also the content parameter matrix and the auxiliary parameter matrix in the matrix mapping model need to be adjusted so that the matrix mapping model can generate a more accurate correlation matrix. The weights of the input layer, pooling layer, fully connected layer, activation layer, fully connected layer, and activation function layer in the matrix fusion model are adjusted, and the weights of each layer in the content fusion model are also adjusted.

[0234] like Fig.21 As shown, the historical behavior sample data is marked as {j 1 , j 2 , …, j n}, sample content information j n and sample auxiliary information The corresponding embedding tables are recorded as Through embeddingtable and historical behavior sample data, we can get sample content encoding information and sample auxiliary encoding information as follows:

[0235]

[0236]

[0237] The sample content coding information is mapped according to the content parameter matrix to generate a sample content matrix, and the sample auxiliary coding information is mapped according to the auxiliary parameter matrix to generate a sample auxiliary matrix. Then the sample content matrix and the sample auxiliary matrix are input into the matrix fusion model, and the matrix fusion model is an SE module, and the n*n*D matrices are fused by the SE module and then the size is reduced to n*n to obtain a sample fusion matrix. Then the sample fusion matrix is ​​fused with the sample content coding information to generate sample fusion information. A coding element is selected from the sample fusion information and input into the content estimation model, and content estimation is performed according to the preset target and the sample fusion information to obtain content estimation information, so as to evaluate whether the information after the fusion of the sample content information and the sample auxiliary information is accurate representation information through the content estimation information.

[0238] The advantage of the above steps 2001-2007 is that by jointly training the matrix mapping model, the matrix fusion model and the content fusion model, the parameters of the matrix mapping model, the matrix fusion model and the content fusion model can influence each other, thereby improving the accuracy of the trained model in fusing browsing content information and content auxiliary information.

[0239] Step 2005 is described in detail below.

[0240] The sample fusion information is output through the content fusion model, and the sample fusion information integrates the sample content information and the auxiliary information from multiple sources. In order to judge whether the sample fusion information can accurately represent the sample content information, it is necessary to estimate the sample content information so as to judge whether the sample fusion information can be used to accurately make content recommendations based on the estimation results.

[0241] like Fig. 22 As shown, step 2005 includes:

[0242] Step 2201, extracting content coding element information from sample fusion information;

[0243] Step 2202: input the content coding element information into the content estimation model to obtain content estimation information.

[0244] Steps 2201-2202 are described in detail below.

[0245] In step 2201, the sample fusion information includes multiple content coding element information, each content coding element information is the coding information obtained by fusing the browsing content information of multiple time points and the auxiliary information of multiple sources, and the last content coding element information fuses the browsing content information of all time points and the auxiliary information of multiple sources, so in this embodiment, the last content coding element information in the sample fusion information is extracted. In other embodiments, a content coding element information can be randomly extracted from the sample fusion information, or the first content coding element information can be selected, or the target coding element information can be obtained by weighted averaging of multiple content coding element information, or the new content coding element information can be obtained by fusing multiple content coding element information by using pooling fusion of different channels.

[0246] After the content coding element information is selected, in step 2202, the content prediction model is a model for predicting the interest level of the content that has not been exchanged, and generates a model for the score or probability distribution of the content that has not been exchanged according to the sample fusion information. The content coding element information is input into the content prediction model to make a content prediction according to the current target project to obtain content prediction information, and the content prediction information is a probability distribution information, through which the probability value of each content information that has not been exchanged can be determined, so as to facilitate content recommendation based on the content prediction information.

[0247] In addition, the content prediction model is also a rating predictor, which is used to predict the rating or probability distribution of content that has not yet been interacted. The content prediction model is usually a user classification model. In this embodiment, the content prediction model is a logistic regression model of the classification model, and the structure of the logistic regression model is as follows: Fig.23 As shown, the classification model includes: a feature extractor and a classifier. The feature extractor is used to convert the content element coding information into an algorithm of low-dimensional representation that can be used by the classifier, and the classifier is used to classify the features. If the last content element coding information R[n] of the sample fusion information is obtained, the content element coding information R[n] contains the fusion information between the sample content information and the sample auxiliary information at multiple time points, and the classifier of this embodiment is a Sigmoid function, the Sigmoid function is used to map the features in the sample fusion information to between 0 and 1 to obtain the probability value of the content information that has not yet been interacted, and the calculation process of the Sigmoid function is shown in formula (7), the content estimation information is obtained as s k :

[0248] s k =sigmoid(target k ×R[n] T ) (7)

[0249] Therefore, the content prediction information of the probability distribution sequence is calculated through the Sigmoid function, and the probability value is used to represent the content information that has not yet been interacted with, so as to make content recommendations based on the content prediction information, so that the content recommendations are more in line with the preferences of the target objects, improve the satisfaction of the target objects, and thus increase the stickiness to the target objects.

[0250] In addition, in this embodiment, the content prediction model adopts a logistic regression model, and the content prediction model of the embodiment can adopt a decision tree, a support vector machine, a random forest, etc., and there is no specific limitation on the specific model adopted by the content prediction model.

[0251] Description of the content recommendation method of the present disclosure

[0252] In addition, this embodiment also discloses a content recommendation method, which makes content recommendations by combining the content recommendation method with target fusion information obtained by the information fusion method. Since the target fusion information is fusion information of the original content encoding information and multi-source original auxiliary encoding information, more accurate and rich content recommendations can be made based on the target content information.

[0253] like Fig.24 As shown, the content recommendation method disclosed in this embodiment includes:

[0254] Step 2401, obtaining target fusion information browsed at each time point in the historical operation data of the target object; wherein the target fusion information is obtained by the above-mentioned information fusion method;

[0255] Step 2402, inputting the target fusion information into the content estimation model to obtain a content estimation probability sequence;

[0256] Step 2403: construct recommended content information based on the content estimation probability sequence.

[0257] Steps 2401-2403 are described in detail below.

[0258] like Figure 2A -C. Figure 3A -C. Figure 4A -C. Figure 5A -C, content recommendations can include article recommendations, video recommendations, and product recommendations. Figure 2A -C, when the target object browses the content and clicks on the content, the terminal 140 will record the browsing content information at that time point, and collect the content auxiliary information of the browsing content information, that is, any information that is helpful for recommendation as content auxiliary information. By storing the browsing content information and content auxiliary information at each time point in a database or log file, when content recommendation is needed, it can be directly extracted from the database or log file as reference information for content recommendation.

[0259] In step 2401, historical operation data of the target object is extracted from a database or a log file, and the historical operation data includes original content information browsed at multiple time points and original auxiliary information of the original content information. The original content information and the original auxiliary information are input into the above-mentioned information fusion method for fusion processing to obtain the browsing content information that can accurately represent each time point. The generation of the target fusion information is specifically referred to the above-mentioned information fusion method, which will not be repeated here.

[0260] After the target fusion information is generated, in step 2402, the target fusion information is input into the content estimation model, and the content estimation model is used to perform content estimation on the target fusion information to estimate the score or probability distribution information of the candidate content information that has not been interacted with to obtain a content estimation probability sequence. Step 2402 is the same as step 2005, except that step 20005 is an estimation operation in the training process, and the input is the sample fusion information used for training and the output is the content estimation information used for training, while step 2402 is after the content estimation model completes the training, and in the actual use process, the target fusion information actually obtained is input into the content estimation model to obtain a real content estimation probability sequence, and the probability value of the candidate content information that has not been interacted with can be known through the content estimation probability sequence, so as to make more accurate content recommendations based on the content estimation probability sequence.

[0261] Next, in step 2403, recommended content information is constructed based on the content estimation probability sequence. The content estimation probability sequence includes the estimation probability of each candidate content information, and the association degree between each candidate content information and the historical operation data is determined according to the estimation probability, so that the candidate content information is selected as the recommended content information according to the estimation probability, so that the content information recommendation is more accurate, and the content information that meets the preferences of the object can be recommended, thereby improving the stickiness of the object to the recommendation system on the terminal 140.

[0262] For example, Fig.25 As shown, after the target fusion information is output by the above information fusion method, the last target coding element information is selected from the target fusion information, and the target coding element information is input into the content estimation model for estimation to obtain the estimated probability of each candidate content information, thereby constructing a content estimation probability sequence. Finally, the content estimation probability sequence is input into the recommendation model for content recommendation to obtain recommended content information, so that content recommendation is simpler and more accurate, in line with the preferences of the target object, and the viscosity of the target object to the functional software on the terminal 140 is increased.

[0263] The advantage of the above steps 2401-2403 is that after the original auxiliary information and original content information of different dimensions are fused to generate the target fusion information based on the information fusion method, the estimated probability of the candidate content information that has not yet been interacted is made according to the target fusion information, so as to generate more accurate and rich content recommendations based on the estimated probability of the candidate content information as content recommendations, thereby improving the target object's satisfaction and experience with the recommended content, thereby increasing the target object's stickiness to the website, platform or functional software on the terminal 140.

[0264] Step 2403 is described in detail below.

[0265] like Fig.26 As shown, step 2403 includes:

[0266] Step 2601, sorting candidate content information in descending order according to estimated probability to obtain a content sorting sequence number;

[0267] Step 2602: Obtain candidate content information with a preset number of recommendations before the content sorting sequence number as recommended content information.

[0268] In step 2601, the content estimated probability sequence includes the estimated probability of each candidate content information, and the estimated probability represents the interest level of the target object in the candidate content information. Therefore, the candidate content information is sorted according to the estimated probability, that is, the candidate content information is sorted from large to small according to the object's interest level in the candidate content information, and then a content sorting sequence number is set according to the sorting order of each candidate content information. When sorting the candidate content information, the content information in the candidate content information that is the same as the browsing content information will be eliminated, and then the remaining candidate content information will be sorted according to the estimated probability to reduce the recommendation of duplicate content.

[0269] For example, if there are candidate content information such as content A, content B, content C, content D, content E and content F, and the estimated probability of each candidate content information is a, b, c, d, e, f, but the size of the estimated probability is c>a>e>f>b>d, then the content ranking number of content A is 2, the content ranking number of content B is 5, the content ranking number of content C is 1, the content ranking number of content D is 6, the content ranking number of content E is 3, and the content ranking number of content F is 4. Therefore, the candidate content information is sorted according to the estimated probability, so that the target object's interest in the candidate content information can be more intuitively seen according to the content ranking number.

[0270] After the content ranking number is assigned to the candidate content information, in step 2702, a preset number of recommendations is pre-set, and the candidate content information is selected according to the preset number of recommendations and the content ranking number, that is, the candidate content information with the preset number of recommendations before the content ranking number is obtained as the recommended content information, so that the output recommended content information better matches the interests of the target object, improves the target object's experience, and further increases the target object's stickiness to the platform, webpage, or functional software on the terminal 140. For example, if the preset number of recommendations is 4, then the 4 candidate content information before the content ranking number is obtained as the recommended content information, that is, content C, content A, content E, and content F are obtained as the recommended content information.

[0271] The advantage of the above steps 2601-2602 is that by sorting the candidate content information according to the estimated probability, the candidate content information with a preset number of recommendations before sorting is selected as the recommended content information, so as to recommend content according to the degree of interest of the target object. Then, the output recommended content information is more in line with the preferences of the target object, thereby improving the target object's experience of browsing the content, and further increasing the target object's stickiness to the website, web page or functional software on the terminal 140.

[0272] System architecture and scenario description of the application of the embodiments of the present disclosure

[0273] If the application scenario is a video recommendation scenario, the historical operation data of the target object is obtained, and the historical operation data includes the video information watched at each time point and the content auxiliary information of the video information watched. The embedding table of the video information watched is first obtained to obtain the original video encoding information, and then the embedding table of the content auxiliary information is obtained to obtain the auxiliary encoding information. The original video encoding information is input into the matrix mapping model to output the video correlation matrix, and each auxiliary encoding information is input into the matrix mapping model to output the auxiliary correlation matrix. The video correlation matrix and the auxiliary correlation matrix are input into the matrix fusion model to output the correlation fusion matrix, and then the correlation fusion matrix and the original video encoding information are input into the content encoding information to output the updated video encoding information, and the updated video encoding information includes the video encoding information watched at each time point. Then the updated video encoding information is input into the content prediction model to output the video prediction information, and the video prediction information includes the estimated probability of the candidate videos that have not yet interacted. Therefore, after the candidate videos that have not yet interacted are sorted from large to small according to the estimated probability, the candidate videos with the preset number of recommendations are selected and recommended in the selected area. Therefore, content auxiliary information of different dimensions and inconsistent update gradients is integrated into the original video encoding information to construct richer representation information for viewing video information. Then, video recommendations based on the richer representation information can make more accurate personalized video recommendations for each target object, thereby improving the target object's satisfaction with the recommended video and increasing the target object's stickiness to the video software.

[0274] Description of the apparatus and device of the present disclosure

[0275] It is to be understood that, although the steps in the above-mentioned flowcharts are sequentially displayed according to the characterization of arrows, these steps are not necessarily executed in sequence according to the order of arrow characterization. Unless there is a clear description in the present embodiment, the execution of these steps does not have a strict order restriction, and these steps can be executed in other orders. Moreover, at least a portion of the steps in the above-mentioned flowcharts can include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a portion of the steps or stages in other steps.

[0276] It should be noted that in each specific implementation of the present application, when it comes to the need to perform relevant processing based on the attribute information or historical operation data of the target object and other data related to the characteristics of the target object, the permission or consent of the target object will be obtained first, and the collection, use and processing of these data will comply with relevant laws, regulations and standards. In addition, when the embodiment of the present application needs to obtain the historical operation data of the target object, the separate permission or separate consent of the target object will be obtained through a pop-up window or jump to a confirmation page. After clearly obtaining the separate permission or separate consent of the target object, the necessary target object-related data used to enable the normal operation of the embodiment of the present application is obtained.

[0277] Fig. 27 This is a schematic diagram of the structure of the information fusion device provided in the embodiment of the present disclosure. The information fusion device includes:

[0278] A data acquisition module is used to acquire historical operation data of a target object; wherein the historical operation data includes: original content coding information and original auxiliary coding information of the original content coding information;

[0279] A mapping module, used for inputting the original content coding information into the matrix mapping model to generate a content correlation matrix, and inputting each original auxiliary coding information into the matrix mapping model to generate an auxiliary correlation matrix;

[0280] A matrix fusion module, used for inputting the content relevance matrix and the auxiliary relevance matrix into the matrix fusion model to obtain a relevance fusion matrix;

[0281] The content fusion module is used to input the correlation fusion matrix and the original content encoding information into the content fusion model to obtain the target fusion information.

[0282] Optionally, the mapping module is specifically used for:

[0283] Inputting the original content coding information into the matrix mapping model, the matrix mapping model maps the original content coding information to a preset content parameter matrix to generate a content correlation matrix;

[0284] Each original auxiliary coding information is input into a matrix mapping model, and the matrix mapping model maps each original auxiliary coding information to a preset auxiliary parameter matrix to generate an auxiliary correlation matrix; wherein each auxiliary correlation matrix has the same dimension.

[0285] Optionally, the matrix fusion module is specifically used for:

[0286] Inputting the content relevance matrix and the auxiliary relevance matrix into a matrix fusion model, and calculating the first weight of the content relevance matrix and the second weight of each of the auxiliary relevance matrices through the matrix fusion model;

[0287] Based on the first weight and the second weight, the content relevance matrix and the auxiliary relevance matrix are weightedly fused to obtain a relevance fusion matrix.

[0288] Optionally, the matrix fusion model includes: a convolution layer, a pooling layer, a fully connected layer and an activation function layer; the matrix fusion module is specifically used for:

[0289] Inputting the content relevance matrix into the convolution layer to obtain content relevance features, and inputting each auxiliary relevance matrix into the convolution layer to obtain auxiliary relevance features;

[0290] Inputting the content relevance feature into the pooling layer to obtain a first vector, and inputting the auxiliary relevance feature into the pooling layer to obtain a second vector;

[0291] Input the first vector to a fully connected layer to obtain a third vector, and input each second vector to a fully connected layer to obtain a fourth vector;

[0292] The third vector is input into the activation function layer to obtain the first weight of the content correlation matrix, and the fourth vector is input into the activation function layer to obtain the second weight of each auxiliary correlation matrix.

[0293] Optionally, the information fusion device further includes:

[0294] The training module is used to jointly train the matrix mapping model, the matrix fusion model and the content fusion model in advance through the following process:

[0295] Acquire training sample data; wherein the training sample data includes: historical behavior sample data, the historical behavior sample data includes sample content encoding information browsed at each time point and sample auxiliary encoding information of the sample content encoding information;

[0296] Inputting sample content coding information into the matrix mapping model to generate a sample content matrix; inputting sample auxiliary coding information into the matrix mapping model to generate a sample auxiliary matrix;

[0297] Input the sample content matrix and the sample auxiliary matrix into the matrix fusion model to obtain the sample fusion matrix;

[0298] Input the sample fusion matrix and the sample content encoding information into the content fusion model to obtain the sample fusion information;

[0299] Inputting the sample fusion information into the content estimation model to obtain content estimation information;

[0300] Construct loss data based on preset content verification information and content estimation information;

[0301] Adjust the parameters of the matrix mapping model, matrix fusion model, and content fusion model based on the loss data.

[0302] Optionally, the training module is specifically used for:

[0303] Extracting content coding element information from sample fusion information;

[0304] The content coding element information is input into the content estimation model to obtain content estimation information.

[0305] Optionally, the data acquisition module is specifically used for:

[0306] Acquire candidate auxiliary information of the candidate content information, and acquire candidate content coding information of the candidate content information and candidate auxiliary coding information of each candidate auxiliary information;

[0307] Obtaining historical operation data of the target object; wherein the historical operation data includes browsing content information at each time point;

[0308] Selecting original content encoding information from candidate content encoding information based on the browsed content information and the candidate content information;

[0309] Based on the browsed content information, the candidate content information and the candidate auxiliary information, original auxiliary coding information is selected from the candidate auxiliary coding information.

[0310] Optionally, the data acquisition module is specifically used for:

[0311] Based on the browsed content information, selecting selected content information from the candidate content information; wherein the selected content information is the same as the browsed content information;

[0312] The original auxiliary coding information is selected from the candidate auxiliary coding information based on the selected auxiliary information of the selected content information and the candidate auxiliary information.

[0313] Fig.28 This is a schematic diagram of the structure of a content recommendation device provided by an embodiment of the present disclosure. The content recommendation device includes:

[0314] An information acquisition module is used to acquire target fusion information browsed at each time point in the historical operation data of the target object; wherein the target fusion information is obtained by the above-mentioned information fusion method;

[0315] A content estimation module is used to input the target fusion information into the content estimation model to obtain a content estimation probability sequence;

[0316] The recommendation module is used to construct recommended content information based on the content estimation probability sequence.

[0317] Optionally, the content estimation probability sequence includes: an estimated probability of each candidate content information; and the recommendation module is specifically used for:

[0318] Sort the candidate content information in ascending order according to the estimated probability to obtain a content sorting sequence number;

[0319] The candidate content information of the preset recommended number before the content sorting sequence number is obtained as the recommended content information.

[0320] Reference Fig.29 , Fig.29 The structural block diagram of the terminal 140 for implementing the information fusion method or content recommendation method of the embodiment of the present disclosure includes: a radio frequency (RF) circuit 2910, a memory 2915, an input unit 2930, a display unit 2940, a sensor 2950, ​​an audio circuit 2960, a wireless fidelity (WiFi) module 2970, a processor 2980, and a power supply 2990. Those skilled in the art can understand that Fig.29 The structure of the terminal 140 shown in the figure does not constitute a limitation on a mobile phone or a computer, and may include more or less components than those shown in the figure, or combine certain components, or arrange the components differently.

[0321] RF circuit 2910 can be used for receiving and sending signals during information transmission or calls. In particular, after receiving the downlink information from the base station, it is sent to processor 2980 for processing; in addition, the designed uplink data is sent to the base station.

[0322] The memory 2915 may be used to store software programs and modules. The processor 2980 executes various functional applications and data processing of the terminal 140 by running the software programs and modules stored in the memory 2915 .

[0323] The input unit 2930 may be used to receive input digital or character information and generate key signal input related to the setting and function control of the terminal 140. Specifically, the input unit 2930 may include a touch panel 2931 and other input devices 2932.

[0324] The display unit 2940 may be used to display input information or provided information and various menus of the terminal 140. The display unit 2940 may include a display panel 2941.

[0325] The audio circuit 2960, the speaker 2961, and the microphone 2962 may provide an audio interface.

[0326] In this embodiment, the processor 2980 included in the terminal 140 can execute the information fusion method or content recommendation method of the previous embodiment.

[0327] The terminal 140 of the embodiment of the present disclosure includes but is not limited to mobile phones, computers, intelligent voice interaction devices, smart home appliances, vehicle terminals, aircraft, etc. The embodiment of the present invention can be applied to various scenarios, including but not limited to cloud technology, artificial intelligence, smart transportation, assisted driving, etc.

[0328] Fig.30 A block diagram of the structure of a portion of the server 110 for implementing the information fusion method or content recommendation method of the embodiment of the present disclosure. The server 110 may have relatively large differences due to different configurations or performances, and may include one or more central processing units (CPUs) 3022 (for example, one or more processors) and a memory 3032, and one or more storage media 3030 (for example, one or more mass storage devices) storing application programs 3042 or data 3044. Among them, the memory 3032 and the storage medium 3030 may be temporary storage or permanent storage. The program stored in the storage medium 3030 may include one or more modules (not shown in the figure), and each module may include a series of instruction operations in the server 110. Furthermore, the central processing unit 3022 may be configured to communicate with the storage medium 3030 and execute a series of instruction operations in the storage medium 3030 on the server 110.

[0329] The server 110 may also include one or more power supplies 3026, one or more wired or wireless network interfaces 3050, one or more input and output interfaces 3058, and / or one or more operating systems 3041, such as Windows ServerTM, Mac OS XTM, UnixTM, LinuxTM, FreeBSDTM, etc.

[0330] The processor in the server 110 may be used to execute the information fusion method or content recommendation method of the embodiment of the present disclosure.

[0331] The embodiments of the present disclosure also provide a computer-readable storage medium, which is used to store program codes, and the program codes are used to execute the information fusion method or content recommendation method of the aforementioned embodiments.

[0332] The embodiment of the present disclosure also provides a computer program product, which includes a computer program. A processor of a computer device reads and executes the computer program, so that the computer device executes and implements the above-mentioned information fusion method or content recommendation method.

[0333] The terms "first", "second", "third", "fourth", etc. (if any) in the specification of the present disclosure and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present disclosure described herein can, for example, be implemented in an order other than those illustrated or described herein. In addition, the terms "comprises" and "comprising" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0334] It should be understood that in the present disclosure, "at least one (item)" means one or more, and "plurality" means two or more. "And / or" is used to describe the association relationship of associated objects, indicating that three relationships may exist. For example, "A and / or B" can mean: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally indicates that the objects associated before and after are in an "or" relationship. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple.

[0335] It should be understood that in the description of the embodiments of the present disclosure, the meaning of multiple (or multiple items) is more than two, greater than, less than, exceed, etc. are understood to not include the number, and above, below, within, etc. are understood to include the number.

[0336] In the several embodiments provided in the present disclosure, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of units is only a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or units, which can be electrical, mechanical or other forms.

[0337] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0338] In addition, each functional unit in each embodiment of the present disclosure may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0339] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present disclosure is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the various embodiments of the present disclosure. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (Read-Only Memory, referred to as ROM), random access memory (Random Access Memory, referred to as RAM), disk or optical disk and other media that can store program codes.

[0340] It should also be understood that the various implementations provided in the embodiments of the present disclosure can be combined arbitrarily to achieve different technical effects.

[0341] The above is a specific description of the implementation methods of the present disclosure, but the present disclosure is not limited to the above implementation methods. Technical personnel familiar with the art can also make various equivalent modifications or substitutions without violating the spirit of the present disclosure. These equivalent modifications or substitutions are all included in the scope defined by the claims of the present disclosure.

Claims

1. An information fusion method, characterized in that: include: Acquire historical operation data of the target object; wherein the historical operation data includes: original content encoding information and original auxiliary encoding information of the original content encoding information; Inputting the original content coding information into a matrix mapping model to generate a content correlation matrix, and inputting each of the original auxiliary coding information into the matrix mapping model to generate an auxiliary correlation matrix; Inputting the content relevance matrix and the auxiliary relevance matrix into a matrix fusion model to obtain a relevance fusion matrix; The correlation fusion matrix and the original content encoding information are input into a content fusion model to obtain target fusion information.

2. The method according to claim 1, characterized in that The step of inputting the original content coding information into a matrix mapping model to generate a content correlation matrix, and inputting each of the original auxiliary coding information into the matrix mapping model to generate an auxiliary correlation matrix comprises: Inputting the original content coding information into the matrix mapping model, and using the matrix mapping model to map the original content coding information to a preset content parameter matrix to generate the content relevance matrix; Each of the original auxiliary coding information is input into the matrix mapping model, and the matrix mapping model maps each of the original auxiliary coding information to a preset auxiliary parameter matrix to generate the auxiliary correlation matrix; wherein each of the auxiliary correlation matrices has the same dimension.

3. The method according to claim 1, characterized in that The step of inputting the content relevance matrix and the auxiliary relevance matrix into a matrix fusion model to obtain a relevance fusion matrix includes: Inputting the content relevance matrix and the auxiliary relevance matrix into a matrix fusion model, and calculating a first weight of the content relevance matrix and a second weight of each of the auxiliary relevance matrices through the matrix fusion model; Based on the first weight and the second weight, the content relevance matrix and the auxiliary relevance matrix are weightedly fused to obtain the relevance fusion matrix.

4. The method according to claim 3, characterized in that The matrix fusion model includes: a convolution layer, a pooling layer, a fully connected layer and an activation function layer; the content relevance matrix and the auxiliary relevance matrix are input into the matrix fusion model, and the first weight of the content relevance matrix and the second weight of each auxiliary relevance matrix are calculated by the matrix fusion model, including: Inputting the content relevance matrix into the convolution layer to obtain content relevance features, and inputting each of the auxiliary relevance matrices into the convolution layer to obtain auxiliary relevance features; Inputting the content correlation feature into the pooling layer to obtain a first vector, and inputting the auxiliary correlation feature into the pooling layer to obtain a second vector; Input the first vector to the fully connected layer to obtain a third vector, and input each of the second vectors to the fully connected layer to obtain a fourth vector; The third vector is input into the activation function layer to obtain the first weight of the content correlation matrix, and the fourth vector is input into the activation function layer to obtain the second weight of each auxiliary correlation matrix.

5. The method according to any one of claims 1 to 4, characterized in that: The matrix mapping model, the matrix fusion model and the content fusion model are jointly trained in advance through the following process: Acquire training sample data; wherein the training sample data includes: historical behavior sample data, the historical behavior sample data includes sample content coding information and sample auxiliary coding information of the sample content coding information; Inputting the sample content coding information into the matrix mapping model to generate a sample content matrix; inputting the sample auxiliary coding information into the matrix mapping model to generate a sample auxiliary matrix; Inputting the sample content matrix and the sample auxiliary matrix into the matrix fusion model to obtain a sample fusion matrix; Inputting the sample fusion matrix and the sample content encoding information into the content fusion model to obtain sample fusion information; Inputting the sample fusion information into a content estimation model to obtain content estimation information; Constructing loss data based on the preset content verification information and the content estimation information; The parameters of the matrix mapping model, the matrix fusion model, and the content fusion model are adjusted based on the loss data.

6. The method according to claim 5, characterized in that The step of inputting the sample fusion information into a content estimation model to obtain content estimation information includes: Extracting content coding element information from the sample fusion information; The content coding element information is input into the content prediction model to obtain the content prediction information.

7. The method according to any one of claims 1 to 4, characterized in that: The acquiring of historical operation data of the target object includes: Acquire candidate auxiliary information of the candidate content information, and acquire candidate content coding information of the candidate content information and candidate auxiliary coding information of each of the candidate auxiliary information; Acquire historical operation data of the target object; wherein the historical operation data includes browsing content information; selecting the original content encoding information from the candidate content encoding information based on the browsing content information and the candidate content information; The original auxiliary encoding information is selected from the candidate auxiliary encoding information based on the browsing content information, the candidate content information and the candidate auxiliary information.

8. The method according to claim 7, characterized in that The selecting the original auxiliary coding information from the candidate auxiliary coding information based on the browsing content information, the candidate content information and the candidate auxiliary information comprises: Based on the browsing content information, selecting selected content information from the candidate content information; wherein the selected content information is the same as the browsing content information; The original auxiliary coding information is selected from among the candidate auxiliary coding information based on the selected content information and the candidate auxiliary coding information.

9. A content recommendation method, characterized in that: include: Obtaining target fusion information browsed at each time point in the historical operation data of the target object; wherein the target fusion information is obtained by the information fusion method according to any one of claims 1 to 8; Inputting the target fusion information into a content estimation model to obtain a content estimation probability sequence; Recommended content information is constructed based on the content estimation probability sequence.

10. The method according to claim 9, characterized in that The content estimation probability sequence includes: the estimation probability of each candidate content information; the construction of the recommended content information based on the target fusion information includes: According to the estimated probability, the candidate content information is sorted in descending order to obtain a content sorting sequence number; The candidate content information of the preset recommended number before the content sorting sequence number is obtained as the recommended content information.

11. An information fusion device, characterized in that: The device comprises: A data acquisition module, used to acquire historical operation data of a target object; wherein the historical operation data includes: original content coding information and original auxiliary coding information of the original content coding information; A mapping module, used for inputting the original content coding information into a matrix mapping model to generate a content correlation matrix, and inputting each of the original auxiliary coding information into the matrix mapping model to generate an auxiliary correlation matrix; A matrix fusion module, used for inputting the content relevance matrix and the auxiliary relevance matrix into a matrix fusion model to obtain a relevance fusion matrix; The content fusion module is used to input the correlation fusion matrix and the original content encoding information into a content fusion model to obtain target fusion information.

12. A content recommendation device, characterized in that: The device comprises: An information acquisition module, used to acquire target fusion information browsed at each time point in the historical operation data of the target object; wherein the target fusion information is obtained by the information fusion method according to any one of claims 1 to 8; A content estimation module, used for inputting the target fusion information into a content estimation model to obtain a content estimation probability sequence; The recommendation module is used to construct recommended content information based on the content estimation probability sequence.

13. An electronic device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the information fusion method described in any one of claims 1 to 8 or the content recommendation method described in any one of claims 9 to 10 is implemented.

14. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the information fusion method described in any one of claims 1 to 8 or the content recommendation method described in any one of claims 9 to 10 is implemented.

Images