Recommendation information determination method, apparatus and device based on multi-modal features, and storage medium

Abstract

The embodiments of the present application provide a recommendation information determination method, apparatus and device based on multi-modal features, and a storage medium. The method comprises: acquiring collected user feature information, wherein the user feature information comprises user independence information and user social information; constructing a user feature matrix on the basis of the user feature information; by means of a preset algorithm or a set trained neural network model, performing calculation to obtain a user matching degree matrix corresponding to the user feature matrix; and determining recommendation information on the basis of matching degree data between users that is recorded in the user matching degree matrix. The present solution solves the problem of being unable to determine highly matched recommendation information for users due to a coarse labeling granularity, and by means of the modeling method of using user social information to construct a user feature matrix, the reliance on deep learning models that are costly to label and difficult to implement is reduced, thereby reducing overall costs.

Description

Method, device, equipment and storage medium for determining recommendation information based on multimodal features

[0001] This application claims priority to the Chinese patent application No. 2023112114040 filed with the China Patent Office on September 19, 2023, the entire contents of which are incorporated by reference into this application. Technical Field

[0002] Embodiments of the present application relate to the field of computer technology, and in particular to a method, apparatus, device, and storage medium for determining recommendation information based on multimodal features. Background Art

[0003] To improve user retention and provide a better user experience, live streaming apps need to recommend content that meets user needs. Essentially, this involves calculating the match between the live content and the user. Related technologies construct a host feature vector and a user feature vector, then calculate the similarity between the host and user feature vectors. Recommended content is then determined based on this similarity.

[0004] In the above scheme, when constructing the anchor feature vector, the main method is to classify and label the anchor's live broadcast images. However, in the relatively simple live broadcast ecosystem, where indoor broadcasts account for a large proportion, the scene that can be distinguished is very limited, making it difficult to carry out fine-grained classification of anchors. Furthermore, the labeling process mostly uses large-scale deep learning models for computer vision tasks, which is costly and difficult, and the efficiency of generating user recommendation information is low.

[0005] Summary of the Invention

[0006] The embodiments of the present application provide a method, apparatus, device, and storage medium for determining recommendation information based on multimodal features. These methods address the problem of being unable to determine highly compatible recommendation information for users due to coarse labeling granularity. By utilizing user social information to construct a user feature matrix model, the method reduces reliance on deep learning models, which are expensive and difficult to implement, and lowers overall costs. Furthermore, the process of generating recommendation information is highly interpretable, making it easy for product personnel to intervene and adjust the recommendation information generation rules based on human experience.

[0007] In a first aspect, an embodiment of the present application provides a method for determining recommendation information based on multimodal features, the method comprising:

[0008] Acquire collected user characteristic information, wherein the user characteristic information includes user independent information and user social information;

[0009] Constructing a user feature matrix based on the user feature information;

[0010] Obtaining a user matching matrix corresponding to the user feature matrix by calculating a preset algorithm or a set trained neural network model;

[0011] Recommendation information is determined based on the matching data between users recorded in the user matching matrix.

[0012] In a second aspect, an embodiment of the present application further provides a device for determining recommendation information based on multimodal features, comprising:

[0013] A data acquisition module is configured to acquire collected user characteristic information, wherein the user characteristic information includes user independent information and user social information;

[0014] a matrix construction module, configured to construct a user feature matrix based on the user feature information;

[0015] A matching degree calculation module is configured to calculate a user matching degree matrix corresponding to the user feature matrix using a preset algorithm or a set trained neural network model;

[0016] The information generation module is configured to determine recommendation information based on the matching data between users recorded in the user matching matrix.

[0017] In a third aspect, an embodiment of the present application further provides a device for determining recommendation information based on multimodal features, the device comprising:

[0018] one or more processors;

[0019] a storage device for storing one or more programs,

[0020] When the one or more programs are executed by the one or more processors, the one or more processors implement the method for determining recommendation information based on multimodal features described in the embodiment of the present application.

[0021] In a fourth aspect, an embodiment of the present application further provides a non-volatile storage medium storing computer-executable instructions, which, when executed by a computer processor, are used to execute the method for determining recommendation information based on multimodal features described in an embodiment of the present application.

[0022] In a fifth aspect, an embodiment of the present application also provides a computer program product, which includes a computer program stored in a computer-readable storage medium. At least one processor of the device reads and executes the computer program from the computer-readable storage medium, so that the device executes the method for determining recommendation information based on multimodal features described in an embodiment of the present application.

[0023] In an embodiment of the present application, by obtaining collected user feature information, wherein the user feature information includes user independent information and user social information, a user feature matrix is ​​constructed based on the user feature information, and a user matching matrix corresponding to the user feature matrix is ​​calculated by a preset algorithm or a set trained neural network model, and recommendation information is determined based on the matching data between users recorded in the user matching matrix. In the process of determining the above-mentioned recommendation information, the modeling method of constructing the user feature matrix by using user independent information and user social information is used to reduce the dependence on deep learning models with high labeling costs and high implementation difficulties. At the same time, it solves the problem of being unable to determine highly matched recommendation information for users due to the coarse labeling granularity. The recommendation information is determined by the matching data between users recorded in the user matching matrix. The process of determining the recommendation information is highly interpretable, and it is easy for manual product intervention and adjustment of the generation rules of recommendation information based on human experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] FIG1 is a flowchart of a method for determining recommendation information based on multimodal features provided in an embodiment of the present application;

[0025] FIG2 is a flowchart of another method for determining recommendation information based on multimodal features provided in an embodiment of the present application;

[0026] FIG3 is a flowchart of another method for determining recommendation information based on multimodal features provided in an embodiment of the present application;

[0027] FIG4 is a schematic diagram of a neural network model structure provided in an embodiment of the present application;

[0028] FIG5 is a schematic diagram of the structure of a convolution block included in a neural network model provided in an embodiment of the present application;

[0029] FIG6 is a structural block diagram of a device for determining recommendation information based on multimodal features according to an embodiment of the present application;

[0030] FIG7 is a schematic structural diagram of a device for determining recommendation information based on multimodal features provided in an embodiment of the present application. DETAILED DESCRIPTION

[0031] The following is a further detailed description of the embodiments of the present application in conjunction with the accompanying drawings and examples. It should be understood that the specific embodiments described herein are merely for the purpose of explaining the embodiments of the present application, and are not intended to limit the embodiments of the present application. It should also be noted that, for ease of description, the accompanying drawings only illustrate portions of the embodiments of the present application, rather than all structures.

[0032] The terms "first," "second," and the like in the specification and claims of this application are used to distinguish similar objects, and are not used to describe a specific order or precedence. It should be understood that the terms used in this manner are interchangeable where appropriate, so that the embodiments of this application can be implemented in an order other than that illustrated or described herein, and that the objects distinguished by "first," "second," and the like are generally of the same type, and do not limit the number of objects; for example, the first object can be one or more. In addition, the term "and / or" in the specification and claims refers to at least one of the connected objects, and the character " / " generally indicates that the objects connected are in an "or" relationship.

[0033] The method for determining recommended information based on multimodal features provided in the embodiments of the present application can be executed by a computing device, such as a server, laptop, desktop computer, or smart terminal. This method can be applied to live broadcast scenarios to determine recommended information for users. For example, for users on the platform, the corresponding recommended information can be determined and the recommended information can be subsequently pushed.

[0034] FIG1 is a flowchart of a method for determining recommendation information based on multimodal features provided in an embodiment of the present application. As shown in FIG1 , the method includes the following steps:

[0035] Step S101: Acquire collected user characteristic information, where the user characteristic information includes user independent information and user social information.

[0036] User characteristic information refers to information related to a user that characterizes the user's characteristics, including independent user information and social user information. Independent user information refers to the collected characteristic information related to the user, while social user information refers to the characteristic information between users. For example, independent user information may include the user's login status, language used, phone type, live streaming style, preferred style, and top-up amount; social user information may include following, viewing time, reward information, likes, forwarding information, sharing information, private messages, and public screen information.

[0037] The information collected above includes, in addition to user-independent information, user-social information. This user-social information can reflect the characteristics between users. It does not simply use a single user-independent feature as the basis for generating recommendation information. It considers the generation of recommendation information from more dimensions, thereby improving the accuracy of the subsequently generated recommendation information.

[0038] In one embodiment, taking the live broadcast scenario as an example, the live broadcast software collects and records the above-mentioned user independent information and user social information through an integrated functional module. For example, when a user uses the live broadcast app, the user's login status, language used, mobile phone type, live broadcast style, favorite style, recharge amount, etc. are recorded, as well as the user's attention status, viewing time, reward information, like information, forwarding information, sharing information, private message information and public screen information, etc., so that the user can subsequently construct a user feature matrix.

[0039] Step S102: construct a user feature matrix based on the user feature information.

[0040] The user feature matrix is ​​a matrix that reflects the individual characteristics of each user and the social characteristics between users. Optionally, when the user feature information includes both individual user information and social information, the user feature matrix is ​​constructed based on the individual user information and the social information. An optional method for constructing the user feature matrix may be to construct an independent information matrix based on the individual user information, a social information matrix based on the social information, and then combine the independent information matrix and the social information matrix to obtain the user feature matrix. Specifically, after constructing separate feature matrices based on the individual user information and the social information, the two matrices are then combined to obtain the user feature matrix.

[0041] Optionally, taking the case where the user is a user in a live broadcast platform, the user includes viewing users who only watch live broadcasts, give rewards, and recharge, as well as live broadcast users who broadcast live and publish short videos. As for independent user information, it includes user feature data of each user in the platform under different feature categories, where the feature categories may include the user's login status, language used, mobile phone type, live broadcast style, favorite style, recharge amount, and other categories described above. When constructing an independent information matrix, a feature vector of the corresponding feature category is constructed based on the user feature data of each user in each feature category, and each feature vector is matrixed to obtain a matrix feature. Taking the category of user login status in the past 30 days as an example, assuming that there are m users on the platform, the login status of each user in the past 30 days is constructed with the number of logins of the user in the past 30 days as the corresponding feature content to construct a feature vector under this category. The exemplary feature vector corresponding to this feature category is recorded as x=[x1,x2,…,x m ], since the example has m users, the dimension of its vector is m, where x1 is the number of logins of user 1 in the past 30 days, x2 is the number of logins of user 2 in the past 30 days, and so on. m is the number of logins by user m in the past 30 days. After obtaining the feature vector, it is matrixed to obtain matrix features. For the first feature category, it is exemplarily recorded as X1. The optional matrix processing method can be achieved by the following calculation:

[0042] Similarly, for the second feature category, the matrix feature is exemplarily recorded as X2, and so on. The numerator of the above formula is the product of the transpose of the feature vector x and the feature vector x, and the denominator is the matrix x T The norm of x, and the values ​​in the matrix features obtained are all normalized between 0 and 1, so as to facilitate subsequent statistical analysis. Among them, for other feature categories, such as the language used by the user, its feature vector can be the ranking of the language used by the user in the live broadcast application, and the ranking is obtained based on the reverse order of the number of languages ​​used by the user; for the feature category of mobile phone type, it can be pre-divided into three levels of high, medium and low according to the processing and computing power of different mobile phone models, and the feature vector is constructed corresponding to the numbers 1, 2 and 3 respectively; for live broadcast style, two feature vectors can be constructed based on the user live broadcast style recorded in the past 7 days and the user live broadcast style recorded in the past 3 days to generate two matrix features, wherein the live broadcast style can be the user live broadcast type obtained based on the labeling as the live broadcast style; for favorite style, the live broadcast style of the anchor with the longest viewing time of the user is counted as his favorite style to construct the matrix vector; for the feature category of recharge amount, the specific size of the recharge amount is used as the matrix value for constructing the matrix vector.

[0043] Optionally, after obtaining matrix features of multiple feature categories, such as constructing matrix features from X1 to X based on user independent information 15 There are 16 matrix features in total, which are combined to obtain an independent information matrix.

[0044] Optionally, when constructing a social information matrix, matrix features are constructed based on the interactive feature data between users in each feature category, and then the matrix features of each feature category are combined to obtain a social information matrix. Taking the feature category of attention as an example, with a total of m users on the platform, an m*m matrix is ​​constructed. The values ​​in the matrix represent the attention between the users in the corresponding row and the users in the corresponding column. For example, if user 1 follows user 2, the value of the first row and second column in the matrix is ​​1. If there is no attention relationship between the two, the value is 0. For example, this temporary matrix is ​​denoted as T 16 , for this matrix T 16 Normalization is performed to obtain the matrix features under this feature category, such as X 16 , the normalization processing method is as follows:

[0045] Similarly, for other feature categories, construct the user viewing time matrix feature X in the past 30 days. 17 , the matrix feature X of the user's viewing time in the past 7 days 18 , matrix feature X of user viewing time in the past three days19 , the matrix features of user rewards in the past 30 days X 20 , the matrix features of user rewards in the past 7 days X 21 , the matrix features of user rewards in the past three days X 22 , the matrix feature X of users’ likes in the past 30 days 23 、User like matrix feature X in the past 7 days 24 , the matrix feature X of users’ likes in the past three days 25 、User sharing matrix feature X in the past 30 days 26 、User sharing matrix feature X in the past 7 days 27 、User sharing matrix feature X in the past three days 28 , the user priority recommendation matrix feature X in the past 30 days 29 、User priority recommendation matrix feature X in the past 7 days 30 , the user priority recommendation matrix feature X in the past three days 31 , the matrix feature X of user private messages in the past 30 days 32 , the matrix feature X of user private messages in the past 7 days 33 , the matrix feature X of user private messages in the past three days 34 、 User public screen matrix features X in the past 30 days 35 、User public screen matrix features X in the past 7 days 36 、User public screen matrix features X in the past three days 37 Etc., the obtained matrix feature X 17 To matrix feature X 37 Merge into a social information matrix.

[0046] Correspondingly, after obtaining the independent information matrix and the social information matrix, they are combined to obtain the user feature matrix. 15 , the social information matrix includes X 16 To X 37 For example, the combined user feature matrix can be recorded as:

[0047] Where ψ is a three-dimensional tensor. Compared to single-user feature statistics, this user feature matrix construction method can retain higher-dimensional feature information, such as user relationships and graph features. This construction method can leverage social features to refine the stratification of live broadcast rooms, enriching the live broadcast ecosystem and addressing the problem that relying solely on visual depth models for labeling is crude and difficult to achieve personalized recommendations based on labels.

[0048] Step S103: Calculate a user matching matrix corresponding to the user feature matrix using a preset algorithm or a trained neural network model.

[0049] After obtaining the user feature matrix, the user matching matrix corresponding to the user feature matrix is ​​calculated. Optionally, the user feature matrix can be calculated using a preset algorithm or a pre-trained neural network model to obtain the corresponding user matching matrix. The user matching matrix reflects the matching between users. Optionally, a specific matching value can be used to represent the matching between the two users. The matching value ranges from 0 to 1. The higher the value, the higher the matching between the two users, which can be used as the basis for generating recommendation information.

[0050] Step S104: Determine recommendation information based on the matching data between users recorded in the user matching matrix.

[0051] In one embodiment, the process of determining recommended information is based on the user matching matrix obtained in step S103. For example, in the case of recommending a live streamer to a user, the recommended information may be the live streamer determined to be recommended for the user. Optionally, the recommended live streamer information may be displayed on a recommendation interface displayed upon user click, allowing the user to click to view the content. Alternatively, while the user is watching a live stream, a recommendation interface may pop up to display the recommended live streamer information.

[0052] In one embodiment, the calculated matching data between users is a specific matching value, where a higher matching value indicates a stronger match. For example, when determining recommendation information for user 1, the matching degrees between user 1 and other users are ranked to obtain a matching ranking, and the top n matching users are recommended to user 1. For example, in the case of recommending a live streamer to user 1, user 1 is the user to be recommended, i.e., the user whose recommendation information is yet to be determined. When determining the recommendation information, users whose matching data exceeds a preset matching value and do not meet a set exclusion criterion are identified as recommended users. This preset matching value can be set as needed. For example, if the matching data ranges from 0 to 1, the preset matching value can be 0.8. Of course, other methods can also be used, such as sorting the matching data from high to low values ​​and selecting the top 10% of users who do not meet the set exclusion criterion as recommended users. The exclusion criteria can be customized based on the specific scenario. For example, when recommending live streamers to a user, the exclusion criteria could include excluding non-live streamers, such as users who haven't live streamed or posted short videos, from being recommended to the user. Another example is when determining recommendations for User 1, users who speak a different language than User 1 would be excluded from the recommendations. By setting exclusion criteria, unreasonable user relationships can be tailored to suit different scenarios to ensure the rationality and usability of recommended information.

[0053] Optionally, when generating recommendation information, the above-described method for determining recommendation information can be used as one of multiple methods in combination with other methods for generating recommendation information. Assuming that the entire system platform has multiple methods for determining recommended hosts for users, and that the hosts to be recommended have their own weights, the hosts determined using the above-described method can increase the weight of the corresponding hosts to be recommended, thereby playing a reasonable role as a control factor in the final determination of the recommended hosts based on weight ranking.

[0054] The above-mentioned method of determining recommendation information based on the matching data between users recorded in the user matching matrix is ​​highly interpretable, and the basis for generating recommendation information can be intuitively obtained. It is easy for manual intervention in the product and for modifying rules based on human experience, and has strong generalization ability.

[0055] As can be seen from the above, by obtaining the collected user feature information, wherein the user feature information includes user independent information and user social information, a user feature matrix is ​​constructed based on the user feature information, and the user matching matrix corresponding to the user feature matrix is ​​calculated by a preset algorithm or a set trained neural network model, and the recommended information is determined based on the matching data between users recorded in the user matching matrix. In the process of determining the above-mentioned recommended information, the modeling method of constructing the user feature matrix using user independent information and user social information reduces the reliance on deep learning models with high labeling costs and high implementation difficulties. At the same time, it solves the problem of being unable to determine highly matched recommended information for users due to the coarse labeling granularity. The recommended information is determined by the matching data between users recorded in the user matching matrix. The process of determining the recommended information is highly explainable, and it is easy for manual product intervention and adjustment of the recommendation information generation rules based on human experience.

[0056] FIG2 is a flowchart of another method for determining recommendation information based on multimodal features provided in an embodiment of the present application, which provides an optional method for calculating a user matching matrix, as shown in FIG2 , including:

[0057] Step S201: Acquire collected user characteristic information, where the user characteristic information includes user independent information and user social information.

[0058] Step S202: construct a user feature matrix based on the user feature information.

[0059] Step S203: Based on the user feature matrix and the set optimization model, a power method is used to solve and obtain a user matching degree matrix corresponding to the user feature matrix.

[0060] In one embodiment, an optimization model is pre-set for calculating the user matching matrix. Optionally, the optimization model can be: min∑||ψ-X×y||2st X∈R m×m ,y∈R n , X≥0,y≥0 ||X||2=1,||y||2=1

[0061] Wherein, ψ is the user feature matrix described in the above embodiment, X is the user matching matrix, y is the feature importance vector, m is the number of users in modeling, and n is the number of feature categories, such as n is 37. In the above optimization problem, the tensor ψ is mapped to the matrix Use the power method to find the matrix matrix The non-negative eigenvector z = (vec(X), y) of the matrix is ​​obtained, thus obtaining the user matching matrix X. The power method is a vector iteration method that calculates the largest modulo eigenvalue and corresponding eigenvector of a matrix. It is suitable for large sparse matrices. This solution yields a more reasonable result for the user matching matrix and significantly reduces the computational effort compared to using complex large-scale neural network models.

[0062] Optionally, in the process of calculating the user matching matrix using the power method, the method further includes: using the power method to solve for the predicted feature importance values ​​corresponding to the user feature matrix. The predicted feature importance values ​​represent the importance of the features corresponding to each feature category. The calculation method is described in the optimization model described above, where the solved y is the predicted feature importance value. Optionally, after obtaining the predicted feature importance values, the method further includes: adjusting the user feature information used to construct the user feature matrix based on the predicted feature importance values, and / or triggering a pre-set recommendation adjustment strategy based on the predicted feature importance values. Regarding the case where the user feature information used to construct the user feature matrix is ​​adjusted based on the predicted feature importance values, if a feature is manually discovered to logically not be considered an important feature, but its predicted feature importance value is very high, such as if the feature category of mobile phone type ranks first in its predicted feature importance value after calculation using the above-mentioned solution method, this indicates that there are errors in the feature construction, such as failure to perform normalization, and the calculation process can be adjusted accordingly. If it is determined that the feature importance prediction value of the public screen information feature category ranks first after calculation using the above solution method, it is relatively reasonable.

[0063] In the case of triggering a pre-set recommendation adjustment strategy based on the predicted feature importance value, corresponding recommendation strategies can be pre-set for different feature categories. For example, after obtaining the feature importance prediction value corresponding to each feature category, the importance of each feature category can be determined based on the ranking of the feature importance prediction values, and the most important or top-ranked recommendation strategy can be used as the currently triggered strategy. For example, if it is determined that the private message feature category has the highest predicted feature importance value, the determined recommended information can be pushed to the user when the user opens the private message interface.

[0064] Step S204: Determine recommendation information based on the matching data between users recorded in the user matching matrix.

[0065] From the above, it can be seen that based on the user feature matrix and the set optimization model, the power method is used to solve the user matching matrix corresponding to the user feature matrix. Compared with using a large deep learning model, the efficiency of generating recommendation information can be significantly improved. The above modeling and solution method has less computational complexity and can save overall cost expenditure.

[0066] FIG3 is a flowchart of another method for determining recommendation information based on multimodal features provided in an embodiment of the present application. As shown in FIG3 , the method includes:

[0067] Step S301: Acquire collected user characteristic information, where the user characteristic information includes user independent information and user social information.

[0068] Step S302: construct a user feature matrix based on the user feature information.

[0069] Step S303: input the user feature matrix into the trained neural network model, and output the user matching matrix through the first convolution group of the last convolution block in the neural network model.

[0070] In one embodiment, the neural network model is shown in FIG4 , which is a schematic diagram of a neural network model structure provided by an embodiment of the present application, which includes a preset number of convolution blocks connected in sequence. The preset number is recorded as n, and the specific value of n is determined according to the number of feature categories of the input user feature matrix and the corresponding self-learning results. Optionally, the input user feature matrix is

[0071] For example, m is the number of users and 37 is the number of feature categories. Through continuous learning and testing, the number of convolution blocks used is finally determined to be 8, that is, it contains 8 blocks connected in sequence. Each block corresponds to a different hyperparameter k, k is the number of feature categories used in the current convolution block operation, and which feature categories in the specific number of feature categories used are obtained during model learning and training. The basis for the specific number of blocks selected during learning and training of this model can be that when the value of k continues to decrease, when the output result of the last block does not converge, the second to last block is used as the final output block. At this time, the number of convolution blocks used is the number of convolution blocks used in the model.

[0072] Optionally, the training process of the neural network model can be as follows: each convolution block is trained in sequence based on the input sample feature matrix, wherein after the training of the previous convolution block is completed, its output is used as the input of the next convolution block, and each convolution block is set with successively decreasing hyperparameter parameter values, and the hyperparameter parameter value of the first convolution block is equal to the number of feature categories in the sample feature matrix. Thus, after the training of each convolution block is completed, the network parameters obtained by the training of the current convolution block are fixed, and the training of the next convolution block is carried out simultaneously.

[0073] After the model training is completed, the user feature matrix constructed by collecting user data is input into the neural network model to output the user matching matrix. Optionally, in the above-mentioned neural network model, each convolution block includes a first convolution group, which is used to sequentially output a temporary user matching matrix, and the first convolution group of the last convolution block outputs the final user matching matrix. Optionally, each convolution block also includes a second convolution group for outputting a feature importance prediction value. In the above-mentioned process of solving the user matching matrix based on the neural network, it also includes: outputting the feature importance prediction value corresponding to the user feature matrix through the second convolution group of the last convolution block in the neural network model. Accordingly, after obtaining the feature importance prediction value, the user feature information for constructing the user feature matrix is ​​adjusted based on the feature importance prediction value, and / or a pre-set recommendation adjustment strategy is triggered based on the feature importance prediction value. Among them, the meaning and application of the feature importance prediction value refer to the description of the above-mentioned example and will not be repeated here. Optionally, for the case where the convolution block contains two convolution groups, the structure of each convolution block is shown in Figure 5. Figure 5 is a structural schematic diagram of a convolution block included in a neural network model provided in an embodiment of the present application, which includes a first convolution group and a second convolution group. Each convolution group optionally includes three convolution layers. For each convolutional layer of the first convolutional group, the input is a 1*1*k*k matrix, and the output is an m*m*k matrix; for each convolutional layer of the second convolutional group, the input is an m*m*1*k matrix, and the output is a 37*k matrix, where m is the number of users, 37 is the number of feature categories, and k is the hyperparameter corresponding to the convolution block. For the 8 convolutional blocks connected in sequence, for example, the value of k corresponding to the first convolution block is 37, the value of k corresponding to the second convolution block is 32, the value of k corresponding to the third convolution block is 28, the value of k corresponding to the fourth convolution block is 24, the value of k corresponding to the fifth convolution block is 20, the value of k corresponding to the sixth convolution block is 16, the value of k corresponding to the seventh convolution block is 12, and the value of k corresponding to the eighth convolution block is 8. Among them, for convolution blocks 2 to 8, the input of each convolution block is the output of the previous convolution block, and convolution block 8 ultimately outputs the user matching matrix X and the feature importance prediction value y. The above-mentioned neural network model structure can ensure that the output results will not overfit and converge during the training process. After training is completed, the user matching matrix and feature importance prediction value can be output more accurately.

[0074] Step S304: Determine recommendation information based on the matching data between users recorded in the user matching matrix.

[0075] From the above, it can be seen that the neural network model includes a preset number of convolution blocks connected in sequence, each convolution block includes a first convolution group and a second convolution group. During the training process, after training each convolution block in turn, the network parameters of the current convolution block are obtained. When training the next convolution block, each convolution block corresponds to a gradually reduced hyperparameter, and the final output result converges while ensuring that the model is not overfitting.

[0076] On the basis of the above scheme, after the determined recommended information is recommended to the user, it further includes statistics on the overall conversion rate of the recommended information after recommendation, such as whether the user's recharge consumption has increased. If the effect is found to be unsatisfactory, the overall data model will be revised. For situations with low conversion rates, that is, inefficient delivery, it will be suppressed and a delivery method with a relatively high conversion rate will be used, such as adding the use of this scheme to determine the delivery method of recommended information to push the recommended information.

[0077] Figure 6 is a block diagram of a device for determining recommendation information based on multimodal features provided by an embodiment of the present application. As shown in Figure 6, the device is used to execute the method for determining recommendation information based on multimodal features provided by the above embodiment, and has corresponding functional modules and beneficial effects of the execution method. As shown in Figure 6, the device specifically includes: a data acquisition module 101, a matrix construction module 102, a matching degree calculation module 103, and an information generation module 104, wherein:

[0078] The data acquisition module 101 is configured to acquire collected user characteristic information, wherein the user characteristic information includes user independent information and user social information;

[0079] A matrix construction module 102 is configured to construct a user feature matrix based on the user feature information;

[0080] A matching degree calculation module 103 is configured to calculate a user matching degree matrix corresponding to the user feature matrix using a preset algorithm or a set trained neural network model;

[0081] The information generation module 104 is configured to determine recommendation information based on the matching data between users recorded in the user matching matrix.

[0082] As can be seen from the above, by obtaining the collected user feature information, wherein the user feature information includes user independent information and user social information, a user feature matrix is ​​constructed based on the user feature information, and the user matching matrix corresponding to the user feature matrix is ​​calculated by a preset algorithm or a set trained neural network model, and the recommended information is determined based on the matching data between users recorded in the user matching matrix. In the process of determining the above-mentioned recommended information, the modeling method of constructing the user feature matrix using user independent information and user social information reduces the reliance on deep learning models with high labeling costs and high implementation difficulties. At the same time, it solves the problem of being unable to determine highly matched recommended information for users due to the coarse labeling granularity. The recommended information is determined by the matching data between users recorded in the user matching matrix. The process of determining the recommended information is highly explainable, and it is easy for manual product intervention and adjustment of the recommendation information generation rules based on human experience.

[0083] In a possible embodiment, the matrix construction module 102 is configured as follows:

[0084] Building an independent information matrix based on the user's independent information, and building a social information matrix based on the user's social information;

[0085] The independent information matrix and the social information matrix are combined to obtain a user feature matrix.

[0086] In a possible embodiment, the user independent information includes user feature data of each user in the platform under different feature categories. The matrix construction module 102 is configured as follows:

[0087] Constructing a feature vector of the corresponding feature category based on the user feature data of each user in each feature category, and performing matrix processing on each feature vector to obtain a matrix feature;

[0088] The matrix features of each feature category are combined to obtain an independent information matrix.

[0089] In a possible embodiment, the user social information includes interaction feature data between users in the platform under different feature categories. The matrix construction module 102 is configured as follows:

[0090] Construct matrix features based on the interactive feature data between users in each feature category;

[0091] The matrix features of each feature category are combined to obtain the social information matrix.

[0092] In a possible embodiment, the matching degree calculation module 103 is configured as follows:

[0093] Based on the user feature matrix and the set optimization model, the power method is used to solve and obtain the user matching matrix corresponding to the user feature matrix.

[0094] In a possible embodiment, the device further includes a feature importance calculation module configured to:

[0095] The power method is used to solve and obtain the feature importance prediction value corresponding to the user feature matrix;

[0096] The device also includes a setting module configured to:

[0097] After the power method is used to solve the feature importance prediction value corresponding to the user feature matrix,

[0098] The user feature information used to construct the user feature matrix is ​​adjusted based on the feature importance prediction value, and / or a preset recommendation adjustment strategy is triggered based on the feature importance prediction value.

[0099] In a possible embodiment, the neural network model includes a preset number of convolution blocks connected in sequence, each of the convolution blocks includes a first convolution group, and the matching degree calculation module 103 is configured as follows:

[0100] The user feature matrix is ​​input into the trained neural network model, and the user matching degree matrix is ​​output through the first convolution group of the last convolution block in the neural network model.

[0101] In a possible embodiment, the convolution block further includes a second convolution group, and the feature importance calculation module is configured as follows:

[0102] Outputting a feature importance prediction value corresponding to the user feature matrix through the second convolution group of the last convolution block in the neural network model;

[0103] The user feature information used to construct the user feature matrix is ​​adjusted based on the feature importance prediction value, and / or a preset recommendation adjustment strategy is triggered based on the feature importance prediction value.

[0104] In a possible embodiment, the device further includes a model training module configured to:

[0105] Based on the input sample feature matrix, each convolution block is trained in sequence, wherein, after the training of the previous convolution block is completed, its output is used as the input of the next convolution block, and each convolution block is set with successively decreasing hyperparameter parameter values, and the hyperparameter parameter value of the first convolution block is equal to the number of feature categories in the sample feature matrix.

[0106] In a possible embodiment, the information generation module 104 is configured to:

[0107] For the users to be recommended in the user matching matrix, users whose recorded matching data are greater than a preset matching value and do not meet the set elimination conditions are determined as recommended users.

[0108] FIG7 is a schematic diagram of the structure of a device for determining recommended information based on multimodal features, according to an embodiment of the present application. As shown in FIG7 , the device includes a processor 201, a memory 202, an input device 203, and an output device 204. The device may include one or more processors 201, with FIG7 emphasizing a single processor 201. The processor 201, memory 202, input device 203, and output device 204 may be connected via a bus or other means, with FIG7 emphasizing a bus connection as an example. The memory 202, as a computer-readable storage medium, can be used to store software programs, computer-executable programs, and modules, such as the program instructions / modules corresponding to the method for determining recommended information based on multimodal features in the embodiment of the present application. The processor 201 executes the software programs, instructions, and modules stored in the memory 202 to execute various functional applications and data processing of the device, thereby implementing the aforementioned method for determining recommended information based on multimodal features. The input device 203 can be used to receive input numeric or character information and generate key signal input related to user settings and function control of the device. The output device 204 can include a display device, such as a display screen.

[0109] An embodiment of the present application further provides a non-volatile storage medium containing computer-executable instructions. When executed by a computer processor, the computer-executable instructions are used to perform a method for determining recommendation information based on multimodal features as described in the above embodiment, which includes:

[0110] Acquire collected user characteristic information, wherein the user characteristic information includes user independent information and user social information;

[0111] Constructing a user feature matrix based on the user feature information;

[0112] Obtaining a user matching matrix corresponding to the user feature matrix by calculating a preset algorithm or a set trained neural network model;

[0113] Recommendation information is determined based on the matching data between users recorded in the user matching matrix.

[0114] It is worth noting that in the above-mentioned embodiment of the device for determining recommendation information based on multimodal features, the various units and modules included are only divided according to functional logic, but are not limited to the above-mentioned division, as long as the corresponding functions can be achieved; in addition, the specific names of the functional units are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of the embodiments of the present application.

[0115] In some possible implementations, various aspects of the methods provided herein may also be implemented in the form of a program product, which includes program code. When the program product is executed on a computer device, the program code is used to cause the computer device to perform the steps of the methods according to the various exemplary embodiments of the present application described above in this specification. For example, the computer device may perform the method for determining recommendation information based on multimodal features described in the embodiments of the present application. The program product may be implemented using any combination of one or more readable media.

Claims

1. A method for determining recommendation information based on multimodal features, wherein: include: Acquire the collected user characteristic information, wherein the user characteristic information includes user independent information and user social information; Constructing a user feature matrix based on the user feature information; The user matching degree matrix corresponding to the user feature matrix is ​​calculated by a preset algorithm or a set trained neural network model; The recommendation information is determined according to the matching data between users recorded in the user matching matrix.

2. The method for determining recommendation information based on multimodal features according to claim 1, wherein: The constructing a user feature matrix based on the user feature information includes: Building an independent information matrix based on the user independent information, and building a social information matrix based on the user social information; The independent information matrix and the social information matrix are combined to obtain a user feature matrix.

3. The method for determining recommendation information based on multimodal features according to claim 2, wherein: The user independent information includes user feature data of each user in the platform under different feature categories, and the independent information matrix is ​​constructed based on the user independent information, including: Constructing a feature vector of a corresponding feature category based on the user feature data of each user in each feature category, and performing matrix processing on each of the feature vectors to obtain a matrix feature; The matrix features of each feature category are combined to obtain an independent information matrix.

4. The method for determining recommendation information based on multimodal features according to claim 2 or 3, wherein: The user social information includes interaction feature data between users in the platform under different feature categories, and the step of constructing a social information matrix based on the user social information includes: Construct matrix features based on the interactive feature data between users in each feature category; The matrix features of each feature category are combined to obtain the social information matrix.

5. The method for determining recommendation information based on multimodal features according to any one of claims 1 to 4, wherein: The user matching degree matrix corresponding to the user feature matrix is ​​calculated by a preset algorithm, including: Based on the user feature matrix and the set optimization model, the power method is used to solve and obtain the user matching degree matrix corresponding to the user feature matrix.

6. The method for determining recommendation information based on multimodal features according to claim 5, wherein: When the power method is used to solve the user matching degree matrix corresponding to the user feature matrix, the method further includes: The power method is used to solve and obtain the feature importance prediction value corresponding to the user feature matrix; Correspondingly, after the power method is used to solve and obtain the feature importance prediction value corresponding to the user feature matrix, the method further includes: The user feature information for constructing the user feature matrix is ​​adjusted based on the feature importance prediction value, and / or a preset recommendation adjustment strategy is triggered based on the feature importance prediction value.

7. The method for determining recommendation information based on multimodal features according to any one of claims 1 to 6, wherein: The neural network model includes a preset number of convolution blocks connected in sequence, each of which includes a first convolution group, wherein the user matching degree matrix corresponding to the user feature matrix is ​​calculated by the set trained neural network model, including: The user feature matrix is ​​input into the trained neural network model, and the user matching degree matrix is ​​output through the first convolution group of the last convolution block in the neural network model.

8. The method for determining recommendation information based on multimodal features according to claim 7, wherein: The convolution block also includes a second convolution group, and the method further includes: Outputting a feature importance prediction value corresponding to the user feature matrix through the second convolution group of the last convolution block in the neural network model; The user feature information for constructing the user feature matrix is ​​adjusted based on the feature importance prediction value, and / or a preset recommendation adjustment strategy is triggered based on the feature importance prediction value.

9. The method for determining recommendation information based on multimodal features according to claim 7, wherein: The training process of the neural network model includes: Based on the input sample feature matrix, each convolution block is trained in sequence, wherein, after the training of the previous convolution block is completed, its output is used as the input of the next convolution block, and each convolution block is set with successively decreasing hyperparameter parameter values, and the hyperparameter parameter value of the first convolution block is equal to the number of feature categories in the sample feature matrix.

10. The method for determining recommendation information based on multimodal features according to any one of claims 1 to 9, wherein: The determining of recommendation information according to the matching data between users recorded in the user matching matrix includes: For the users to be recommended in the user matching matrix, users whose matching data is greater than a preset matching value and does not meet the set elimination condition are determined as recommended users.

11. A device for determining recommendation information based on multimodal features, wherein: include: A data acquisition module is configured to acquire the collected user characteristic information, wherein the user characteristic information includes user independent information and user social information; A matrix construction module, configured to construct a user feature matrix based on the user feature information; A matching degree calculation module, configured to calculate a user matching degree matrix corresponding to the user feature matrix through a preset algorithm or a set trained neural network model; The information generation module is configured to determine the recommended information according to the matching data between users recorded in the user matching matrix.

12. A device for determining recommendation information based on multimodal features, the device comprising: one or more processors; A storage device for storing one or more programs, when the one or more programs are executed by the one or more processors, the one or more processors implement the method for determining recommendation information based on multimodal features as described in any one of claims 1-10.

13. A non-volatile storage medium storing computer executable instructions, wherein the computer executable instructions are used to execute the method for determining recommendation information based on multimodal features according to any one of claims 1 to 10 when executed by a computer processor.

14. A computer program product comprising a computer program, wherein: When the computer program is executed by a processor, the method for determining recommendation information based on multimodal features according to any one of claims 1 to 10 is implemented.

Images