A method, device, equipment and medium for recommending facial expressions

Abstract

The present application discloses a method, device, equipment and medium for recommending facial expressions, the method comprising: generating a target bipartite topology map of a target object; determining a feature vector of the target object based on the target bipartite topology map; determining the vector similarity between the feature vector of the target object and the facial expression vector of each facial expression image to be recommended in a set of facial expressions to be recommended; and recommending facial expressions to the target object based on the vector similarity. By applying the present application, the real needs of users can be fully considered, and the recommendation effect of facial expressions based on cold start can be effectively improved.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a method, device, equipment and medium for recommending expression images. Background Art

[0002] With the development of Internet technology, the number of emoticon images contained in the emoticon resources of some applications on the terminal is increasing. Faced with an increasing number of emoticon images, some emoticon images are usually recommended to the target user, such as emoticon images used by other users with similar interests, or emoticon images that are currently in use.

[0003] However, the existing recommendation method for emoticon images not only has the problem of cold start, but also does not consider the real needs of users, and the recommendation effect needs to be further improved. Summary of the invention

[0004] The present application provides an expression image recommendation method, device, equipment and medium to solve at least one of the above technical problems.

[0005] On the one hand, the present application provides a method for recommending an expression image, comprising:

[0006] Based on the historical interest expression image information of the target object and the similarity between the expression images in the historical interest expression image information, generating a target bipartite topological graph of the target object;

[0007] Based on the target bipartite topology of the target object, determining a feature vector of the target object; the feature vector of the target object represents feature data of the target object's interest expression image;

[0008] Determining the vector similarity between the feature vector of the target object and the expression vector of each expression image to be recommended in the set of expression images to be recommended;

[0009] An expression image is recommended to the target object based on the vector similarity.

[0010] On the other hand, the present application also provides an expression image recommendation device, comprising:

[0011] A topology map generating module, used for generating a target bipartite topology map of the target object based on the historical interest expression image information of the target object and the similarity between the expression images in the historical interest expression image information;

[0012] A vector determination module, used to determine a feature vector of the target object based on a target bipartite topological graph of the target object; the feature vector of the target object represents feature data of an expression of interest image of the target object;

[0013] A similarity determination module, used to determine the vector similarity between the feature vector of the target object and the expression vector of each expression image to be recommended in the set of expression images to be recommended;

[0014] A recommendation module is used to recommend expression images to the target object based on the vector similarity.

[0015] On the other hand, a device is also provided, comprising a processor and a memory, wherein the memory stores at least one instruction, at least one program, a code set or an instruction set, and the at least one instruction, the at least one program, the code set or the instruction set is loaded and executed by the processor to implement any of the above-mentioned expression image recommendation methods.

[0016] On the other hand, a computer storage medium is also provided, wherein at least one instruction, at least one program, code set or instruction set is stored in the storage medium, and the at least one instruction, at least one program, code set or instruction set is loaded by a processor and executed by any of the above-mentioned expression image recommendation methods.

[0017] The present application provides a method, device, equipment and medium for recommending emoticon images, which have the following technical effects:

[0018] The embodiment of the present application generates a target bipartite topology map of the target object based on the historical interest expression image information of the target object and the similarity between expression images in the historical interest expression image information; determines the feature vector of the target object based on the target bipartite topology map of the target object; the feature vector of the target object represents the feature data of the interest expression image of the target object; determines the vector similarity between the feature vector of the target object and the expression vector of each expression image to be recommended in the set of expression images to be recommended; and recommends expression images to the target object based on the vector similarity. This method can effectively utilize the content features of the historical interest expressions of the target object and the topological structure information between the historical interest expressions, and combines the two to determine the target object node information in the target bipartite topology map, thereby comprehensively considering the real needs of users and effectively improving the recommendation effect of expression images based on cold start. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] In order to more clearly illustrate the technical solutions and advantages in the embodiments of the present application or the prior art, the drawings required for use in the embodiments or the prior art descriptions are briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0020] Figure 1It is a schematic diagram of an implementation environment provided by an embodiment of the present application;

[0021] Figure 2 It is a flowchart of an expression image recommendation method provided in an embodiment of the present application;

[0022] Figure 3 It is a schematic diagram of a process for generating a target bipartite topology graph provided in an embodiment of the present application;

[0023] Figure 4a A schematic diagram of the structure of a target bipartite topology graph provided in this embodiment;

[0024] Figure 4b A schematic diagram of the structure of another target bisection topology graph provided in this embodiment;

[0025] Figure 5 is a schematic diagram of training a feature vector model provided in an embodiment of the present application;

[0026] Figure 6 is a block diagram of an expression image recommendation device provided in an embodiment of the present application;

[0027] Figure 7 It is a schematic diagram of the hardware structure of a device provided by the present application for implementing the method provided by an embodiment of the present application. DETAILED DESCRIPTION

[0028] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present application.

[0029] In order to make the objectives, technical solutions and advantages of the present application clearer, the implementation methods of the present application will be further described in detail below with reference to the accompanying drawings.

[0030] Please refer to Figure 1, which shows a schematic diagram of an implementation environment provided by an embodiment of the present application. The implementation environment may include: a terminal 120 and a server 140 connected to the terminal 120 via a network. The terminal 120 may specifically include software running in a physical device, such as an application installed on the device, etc., and may also include at least one of the physical devices of the type of a smart phone, a desktop computer, a tablet computer, a laptop computer, a digital assistant, a smart wearable device, etc., with an application installed. Specifically, an operating system runs on the terminal, and the operating system may be a desktop operating system such as a Windows operating system, a Linux operating system, or a Mac OS (Apple desktop operating system), or a mobile operating system such as an iOS (Apple mobile terminal operating system) or an Android operating system.

[0031] The server 140 may be an independent server, or a server cluster consisting of multiple independent servers, or a cloud computing service center.

[0032] It should be understood that Figure 1 The implementation environment shown in the figure is only an application environment of the present application scheme and does not constitute a limitation on the application environment of the present application scheme. Other application environments may also include more or fewer computer devices than shown in the figure, or network connection relationships between computer devices.

[0033] The execution subject of the expression image recommendation method of the present application may be the server 140, and the server 140 may recommend expression images according to the needs of the user or automatically recommend expression images to the user. Figure 1 As shown, there may be multiple emoticon images, and the corresponding number of users is also multiple. The emoticon images to be recommended can be determined and recommended for multiple users at the same time.

[0034] It should be noted that if the terminal 120 has strong processing and computing capabilities and storage capabilities, the expression image recommendation method of the present application can be executed by the terminal 120 alone or by the terminal 120 and the server 140 together.

[0035] The following describes a specific embodiment of a method for recommending facial expressions in the present application. Figure 2 This is a flowchart of an expression image recommendation method provided in an embodiment of the present application. The present application provides method operation steps as described in the embodiment or flowchart, but based on conventional or non-creative labor, more or fewer operation steps may be included. The order of steps listed in the embodiment is only one of many step execution orders and does not represent the only execution order. Figure 2 As shown, the execution subject of this method is Figure 1 In the server 140, the method may include:

[0036] S201: Generate a target bipartite topology graph of the target object based on historical interest expression image information of the target object and similarities between expression images in the historical interest expression image information.

[0037] In the embodiment of the present application, the target object may be a user who needs to obtain recommended emoticons. Specifically, the target object may be a user who uses an application on a terminal and needs to obtain the emoticon resources of the application or the recommended emoticons in the emoticon library. The application includes but is not limited to an instant messaging program (such as QQ, WeChat, Weibo, etc.), a video program, a short video program, a chat tool, an interactive program, or other applications that need to use emoticon images.

[0038] The historical interest expression image information may include at least one of the following: the user's historical expression image information (expressions used on the application or on the terminal), the user's historical collection of expression image information, and the user's other historical operation expression image information (such as likes, comments, complaints, reports, etc.). The expression image can be a static expression image, a dynamic expression image, or an expression package composed of multiple expression images. The expression image information may include the relationship between the expression image and the user (whether it is an expression image that a certain user is interested in), the content of the expression image, the category to which the expression image belongs, the label corresponding to the expression, and other feature data of the expression image.

[0039] A bipartite topology graph is a network graph that represents the relationship between nodes. The bipartite topology graph can be expressed as G=(V,E), where V represents the set of nodes in the bipartite topology graph (for example, V=v1,...,vn), and each node corresponds to an entity (for example, a user, an expression); E represents the set of edges connecting the associated nodes in the bipartite topology graph (for example, E=e1,...,en), and each connecting edge is used to reflect the association relationship between the connected nodes.

[0040] The historical interest expression image information of each target object has multiple expression images, each of which is different. By calculating the similarity of the expression images, it can be determined whether there is a correlation between the expressions. Based on the similarity of the expression images, it is determined that there is a correlation between the two expression images. If it is determined that there is a correlation between the two expression images, it is determined that the two have a connection relationship in the bipartite topology graph. Afterwards, when constructing the bipartite topology graph, connecting edges can be set between the nodes corresponding to the expression images with a correlation relationship.

[0041] In one embodiment, Figure 3 Schematic diagram of a process for generating a target bipartition topology graph provided in an embodiment of the present application. Figure 3 As shown, the method may include:

[0042] S301: Generate a first bipartite topological graph of the target object based on the target object and the historical interest expression image information of the target object.

[0043] Specifically, the first bipartite topology graph includes a target object node corresponding to the target object and an expression image node corresponding to the expression image in the historical expression image information of interest. The first bipartite topology graph also includes a connection edge between the target object node and the expression image node, and the connection edge indicates that the expression image corresponding to the expression image node is an expression image that the target object corresponding to the target object node is interested in. That is, the target object has used the expression image, collected the expression image, or otherwise operated the expression image.

[0044] S303: Determine a connection relationship between every two expression image nodes based on the similarity between every two expression image nodes and a first similarity threshold.

[0045] Specifically, the similarity between each two expression image nodes is the similarity of the expression images corresponding to the expression image nodes, and the vector distance (such as cosine distance) between the expression vectors corresponding to each two expression image nodes can be calculated as the similarity between the corresponding two expression image nodes. Afterwards, the similarity between each two expression image nodes is compared with the first similarity threshold in turn, and the connection relationship between each two expression image nodes is determined based on the comparison result.

[0046] If the similarity between two expression image nodes is greater than or equal to a first similarity threshold, it is determined that the expression images corresponding to the two expression image nodes are associated, that is, the two corresponding expression image nodes are connected (that is, there is a connecting edge); otherwise, it is determined that the expression images corresponding to the two expression image nodes are not associated, that is, the two corresponding expression image nodes are not connected. The first similarity threshold is a preset similarity threshold, which can be adaptively adjusted according to actual recommendation effects or manual experience.

[0047] S305: Based on the determined connection relationship between every two expression image nodes and the first bipartite topology graph, a target bipartite topology graph of the target object is determined.

[0048] Specifically, after constructing the first bipartite topology graph and determining the connection relationship between every two expression image nodes in the first bipartite topology graph, the network structure of the bipartite topology graph can be reconstructed based on the connection relationship between the expression image nodes, that is, the expression image nodes with connection relationships are connected to obtain a target bipartite topology graph of the target object. The target bipartite topology graph includes target object nodes, expression image nodes, connection edges between target object nodes and expression image nodes, and connection edges between expression image nodes.

[0049] Figure 4a A schematic diagram of the structure of a target bisection topology diagram provided in this embodiment. Figure 4a As shown, the target bipartite topology graph includes user node u1, expression image node t1, expression image node t2, expression image node t3, expression image node t4, and connecting edges u1t1~u1t4, connecting edge t1t2, and connecting edge t3t4.

[0050] Figure 4b A schematic diagram of another target bisection topology diagram provided in this embodiment. Figure 4b As shown, the target bipartite topology graph includes user node u2, user node u3, expression image node t1, expression image node t2, expression image node t3, expression image node t4, expression image node t5, expression image node t6, and connecting edges u2t1~u2t3, u3t4~u3t6, connecting edge t2t3, connecting edge t3t5, and connecting edge t5t6.

[0051] Specifically, the user can trigger an emoticon recommendation function key or an emoticon search function key on the terminal or an application on the terminal to send a request to the server for recommended emoticons. The server responds to the request and generates a target bipartite topology map of the target object based on the user's historical interest emoticon image information and the similarity between emoticon images in the historical interest emoticon image information, so as to facilitate the subsequent extraction of the feature vector of the target object based on the target bipartite topology map, and then realize the recommendation of the corresponding emoticon image.

[0052] S203: Determine a feature vector of the target object based on the target bipartite topology graph of the target object; the feature vector of the target object represents feature data of the target object's interest expression image.

[0053] In the embodiment of the present application, the feature vector of the target object is feature data used to characterize the interest expression image of the target object. The feature data at least includes content feature data of the interest expression image and topological structure feature data between the interest expression images.

[0054] In one embodiment, determining the feature vector of the target object based on the target bipartite topology graph of the target object may include:

[0055] S501: Performing feature vector representation learning on the target object of the target bipartite topology graph based on a feature vector model to obtain a feature vector of the target object.

[0056] For example, the feature vector model is obtained by training the initial neural network model using the training object and the positive and negative expression images corresponding to the training object as the training sample set; the positive and negative expression images include interest expression images (i.e., positive sample images) and non-interest expression images (i.e., negative sample images). The initial neural network model includes but is not limited to common convolutional neural network models, such as graph convolutional neural network models GNN (such as at least one of GCN, GAN, GAE, GGN, and GSTN). The training process of the feature vector model will be described in detail later.

[0057] By constructing a feature vector model, we can effectively utilize the content features of the target object's historical interest expression images and the topological structure information between the historical interest expression images, and combine the two to learn and recommend the target object node information, further effectively improving the effect of cold start recommendation.

[0058] In another embodiment, each node in the target bipartite topology graph can be mapped to a vector in space through network mapping technology, and the vectors corresponding to all nodes related to a target object node are used as the feature vector of the target object corresponding to the target object node. The network mapping technology includes but is not limited to learning algorithms for graph feature expression such as Deepwalk technology and Node2vec technology.

[0059] S205: Determine the vector similarity between the feature vector of the target object and the expression vector of each expression image to be recommended in the expression image set to be recommended.

[0060] In the embodiment of the present application, the set of emoticon images to be recommended can be stored in a local server in the form of a database, or can be stored in at least one other server, cloud or storage node. The set of emoticon images to be recommended includes multiple emoticon images to be recommended, each of which can be a static image or a dynamic image, or multiple static images and / or dynamic images (such as an emoticon package).

[0061] After obtaining the expression image to be recommended, an expression vector for representing the image features of the expression image to be recommended can be extracted. Then, based on a preset similarity algorithm, the vector similarity between the feature vector and the expression vector is calculated. The similarity algorithm includes but is not limited to a cosine similarity algorithm.

[0062] In one embodiment, determining the vector similarity between the feature vector of the target object and the expression vector of each to-be-recommended expression image in the to-be-recommended expression image set includes:

[0063] S601: Obtain a key image corresponding to each to-be-recommended emoticon image in the to-be-recommended emoticon image set.

[0064] Specifically, if the expression image to be recommended is a static image, the static image is used as the key image. If the expression to be recommended is an expression package, an expression image can be extracted from the expression package. If the extracted expression image is a static image, the extracted static image is used as the key image; if the extracted expression image is a dynamic image, it is necessary to extract the key frame in the dynamic image as the key image. The format of the key image can be a common picture format (such as jpg, tif, img, etc.).

[0065] S603: Extracting image features of the key image corresponding to each of the expression images to be recommended, and obtaining an expression vector of each of the expression images to be recommended.

[0066] Specifically, the image features of the key images corresponding to each to-be-recommended expression image can be extracted through a convolutional network model to obtain an expression vector representing the image features of each to-be-recommended expression image. The convolutional network model includes but is not limited to: rennet18, rennet50, VGG and other models. The image features include image content features, image attribute features, image category features, image label features, and the like.

[0067] S605: Determine the vector similarity between the feature vector of the target object and the expression vector of each expression image to be recommended; the feature vector and the expression vector have the same vector dimension.

[0068] Specifically, the vector distance (eg, cosine distance) between the feature vector of the target object and the extracted expression vector of each expression image to be recommended is calculated, and the vector distance is used as a measure of the vector similarity between the feature vector and the expression vector.

[0069] S207: Recommending an expression image to the target object based on the vector similarity.

[0070] In the embodiment of the present application, based on the vector similarity, at least one target recommended emoticon image corresponding to the maximum vector similarity value is determined; then, the at least one determined target recommended emoticon image is recommended to the target object. Specifically, after determining at least one target recommended emoticon image, the target recommended emoticon image can be recommended to a user corresponding to the identifier of the target object or to a terminal where the target object is located.

[0071] In another embodiment, the recommending an expression image to the target object based on the vector similarity may include:

[0072] Based on the vector similarity, a plurality of to-be-recommended expression images in the to-be-recommended expression set are sorted to determine and obtain an expression image recommendation list;

[0073] Based on the expression image recommendation list, recommending a preset number of expression images ranked top to the target object; and / or

[0074] Determine at least one target recommended expression image corresponding to which the vector similarity is greater than a second similarity threshold;

[0075] The determined at least one target recommended expression image is recommended to the target object.

[0076] The embodiment of the present application generates a target bipartite topology map of the target object based on the historical interest expression image information of the target object and the similarity between expression images in the historical interest expression image information; determines the feature vector of the target object based on the target bipartite topology map of the target object; the feature vector of the target object represents the feature data of the interest expression image of the target object; determines the vector similarity between the feature vector of the target object and the expression vector of each expression image to be recommended in the set of expression images to be recommended; and recommends expression images to the target object based on the vector similarity. This method can effectively utilize the content features of the historical interest expressions of the target object and the topological structure information between the historical interest expressions, and combines the two to determine the target object node information in the target bipartite topology map, thereby comprehensively considering the real needs of users and effectively improving the recommendation effect of expression images based on cold start.

[0077] The following introduces an example of training the feature vector model provided in this application. Figure 5 is a schematic diagram of a feature vector model training provided in an embodiment of the present application. The training step can be performed by Figure 1 The server 140 may also execute the function, and the server 140 only obtains the feature vector model constructed by the server 140. Figure 5 As shown, the training process of the feature vector model is illustrated by taking the initial neural network model including the graph convolutional neural network model (GCN model) as an example, including:

[0078] S701: Constructing a training sample set, wherein the training sample set includes a plurality of training subjects and positive and negative expression image information of the training subjects after a preset time, wherein the positive and negative expression image information includes interest expression images and non-interest expression images.

[0079] Specifically, taking the expression image as an expression package as an example, a GIF image is randomly extracted from each expression package, and the key frame of the expression image is randomly extracted and saved as a jpg image as the representative image of the expression package. The image is feature extracted using resnet18 to obtain a 512-dimensional feature vector as the expression vector of each expression package, thereby obtaining a feature vector representation of each expression package. The positive and negative expression image information corresponding to each training object (user) is obtained. If the training object downloads expression package i after time τ, i can be considered as an expression image of interest, and positive expression image information (positive sample) is obtained. Then, expression package j is randomly extracted from the expression package that the training object has not downloaded or used. It can be considered that j is a non-interest expression image, and negative expression image information (negative sample) is obtained, thereby constructing a training sample set.

[0080] S703: Generate a sample bipartite topology graph of the training object based on the interest expression image information of the training object before a preset time and the similarity between the expression images therein.

[0081] After constructing the training sample set, a sample bipartite topology map can be constructed. The interest expression image information includes expression images downloaded, collected or operated by the training object before time τ, and the interest expression image information and the similarity between the expression images therein are used to construct the sample bipartite topology map of the training object.

[0082] Specifically, the generating of the sample bipartite topology graph of the training object based on the interest expression image information of the training object before a preset time and the similarity between the expression images therein includes:

[0083] S7031: Based on the training object and the interest expression image information of the training object before a preset time, generate a first sample bisection topology map of the training object.

[0084] Specifically, the first sample bipartite topology graph includes a training object node corresponding to the training object and an expression image node corresponding to the expression image in the expression image information of interest. The first sample bipartite topology graph also includes a connection edge between the training object node and the expression image node, and the connection edge indicates that the expression image corresponding to the expression image node is an expression image of interest to the training object corresponding to the training object node.

[0085] S7033: Determine a connection relationship between every two expression image nodes based on the similarity between every two expression image nodes and a first similarity threshold.

[0086] Specifically, the similarity between each two expression image nodes is the similarity of the expression images corresponding to the expression image nodes, and the vector distance (such as cosine distance) between the expression vectors corresponding to each two expression image nodes can be calculated as the similarity between the corresponding two expression image nodes. Afterwards, the similarity between each two expression image nodes is compared with the first similarity threshold in turn, and the connection relationship between each two expression image nodes is determined based on the comparison result.

[0087] If the similarity between two expression image nodes is greater than or equal to a first similarity threshold, it is determined that the expression images corresponding to the two expression image nodes have an associated relationship, that is, the two corresponding expression image nodes have a connection relationship (that is, there is a connecting edge); otherwise, it is determined that the expression images corresponding to the two expression image nodes do not have an associated relationship, that is, the two corresponding expression image nodes do not have a connection relationship. The first similarity threshold is a preset similarity threshold λ, which can be adaptively adjusted according to actual recommendation effects or manual experience.

[0088] S7035: Based on the determined connection relationship between every two expression image nodes and the first sample bipartite topology graph, determine and obtain the sample bipartite topology graph of the training object.

[0089] Specifically, after constructing the first sample bipartite topology graph and determining the connection relationship between every two expression image nodes in the first sample bipartite topology graph, the network structure of the bipartite topology graph can be reconstructed based on the connection relationship between the expression image nodes, that is, the expression image nodes with connection relationships are connected to obtain the sample bipartite topology graph of the training object. The sample bipartite topology graph includes training object nodes, expression image nodes, connection edges between training object nodes and expression image nodes, and connection edges between expression image nodes.

[0090] In practical applications, each training object (user) is first regarded as a training object node, and each set of emoticon packages collected by the training object is also regarded as an emoticon image node. There is a connection edge between the training object node and the emoticon image node corresponding to its corresponding interest emoticon package. Whether there is a connection between two emoticon image nodes is determined by whether the similarity between the emoticon packages collected by the training object is greater than λ, thereby constructing a sample bipartite topology graph G = (V, E).

[0091] S705: Input the sample bipartite topology graph of the training object into the initial graph convolutional neural network model for propagation learning to obtain the initial feature vector of the training object.

[0092] In one embodiment, the step of inputting the sample bisection topology graph of the training object into the initial graph convolutional neural network model for propagation learning to obtain the initial feature vector of the training object includes:

[0093] Obtaining node feature data of each node of the sample bipartite topology graph of the training object;

[0094] Acquire connection feature data between nodes having connection edges in each node of the sample bipartite topology graph of the training object;

[0095] Based on the node feature data and the connection feature data, an initial graph convolutional neural network model is used to perform propagation learning on the sample bipartite topology graph to obtain an initial feature vector of the training object.

[0096] The above-mentioned node feature data can be used as a model input matrix, and the above-mentioned connection feature data can be used as an adjacency matrix. The initial convolutional neural network model is used to propagate and learn the sample binary topology graph, thereby obtaining the initial feature vector of the training object covering richer feature data.

[0097] In practical applications, the GCN model is used for training. The constructed sample bipartite topology graph G = (V, E), the input of which is a matrix X of dimension N × F, where N is the number of nodes in the network, F is the input feature dimension of each node (for example, 512 dimensions), and the adjacency matrix of the sample bipartite topology graph is an N × N matrix A. The GCN algorithm can be used to propagate the bipartite graph, and the propagation rule is f(H′, A) = σ(AHW), where σ is the activation function RELU, W is the weight matrix, and the dimension is F i ×F i+1 , where F 0 =512.

[0098] S707: Determine the initial feature vector of the training object and the sample vector similarity of the expression vector of the expression image in the positive and negative expression image information corresponding to the training object.

[0099] S709: Based on the sample vector similarity and the image type of the corresponding expression image, optimize the model parameters of the initial graph convolutional neural network model to obtain the graph convolutional neural network model.

[0100] During actual model training, a sample binary topology graph is constructed based on the expression images of interest of the training object and input into GCN (e.g., a two-layer network structure) for propagation. The learned initial feature vector of the training object is propagated, and the vector similarity is calculated with the expression vectors corresponding to the positive and negative sample expression image information in the training sample set. The score corresponding to the vector similarity is used as the score of whether the end user will download the expression package, and then the model parameters of the initial graph convolutional neural network model are optimized according to the image type of the positive and negative sample expression image information (e.g., expression images of interest or non-interest expression images), thereby achieving the purpose of model training.

[0101] In the embodiment of the present application, by using the GCN model, GCN can be used to learn the topological structure information between the user (target object or training object) and the interest expression image, and the interest expression images and the interest expression images.

[0102] In addition, the bipartite graph in the GCN model can be represented by the adjacency matrix A. Each node in the input matrix X is a training object node and an expression image node. In terms of input features, user features are randomly initialized, and expression features are features extracted by the convolutional neural network model (such as resnet18). The training object and the expression image of interest of the training object are connected through the adjacency matrix, and the expression image nodes with similarity higher than the preset threshold are connected. The connected nodes (i.e., training object node-expression image node, expression image node-expression image node) will be subjected to feature aggregation and nonlinear table conversion by GCN, which can better learn the characteristics of the nodes. This is also the topological structure information that the deep learning model cannot learn. In addition, by connecting the expression node of interest with the expression node of interest, the similarity information of two similar expressions can be added to the adjacency matrix in the form of connecting edges, which increases the information of the data and is conducive to the optimization of the node features by the model.

[0103] The following is an embodiment of the device of the present application, which can be used to execute the embodiment of the method of the present application. For details not disclosed in the embodiment of the device of the present application, please refer to the embodiment of the method of the present application.

[0104] Please refer to Figure 6 , which shows a block diagram of an expression image recommendation device provided by an embodiment of the present application. The device has the function of implementing the server side in the above method example, and the function can be implemented by hardware or by hardware executing corresponding software. The device 60 may include:

[0105] A topology map generating module 61, configured to generate a target bipartite topology map of the target object based on the historical interest expression image information of the target object and the similarity between the expression images in the historical interest expression image information;

[0106] A vector determination module 62 is used to determine a feature vector of the target object based on the target bisection topology of the target object; the feature vector of the target object represents feature data of the target object's interest expression image;

[0107] A similarity determination module 63, used to determine the vector similarity between the feature vector of the target object and the expression vector of each expression image to be recommended in the set of expression images to be recommended;

[0108] The recommendation module 64 is used to recommend expression images to the target object based on the vector similarity.

[0109] In some embodiments, the topology map generation module may include:

[0110] A first topology map generating unit, configured to generate a first bipartite topology map of the target object based on the target object and the historical interest expression image information of the target object; the first bipartite topology map includes a target object node corresponding to the target object and an expression image node corresponding to the expression image in the historical interest expression image information;

[0111] A connection relationship determining unit, configured to determine a connection relationship between every two expression image nodes based on a similarity between the two expression image nodes and a first similarity threshold;

[0112] The target topology map generating unit is used to determine the target bipartite topology map of the target object based on the determined connection relationship between every two expression image nodes and the first bipartite topology map.

[0113] In some embodiments, the vector determination module includes:

[0114] A vector determination unit, configured to perform feature vector representation learning of a target object on the target bipartite topology graph based on a feature vector model to obtain a feature vector of the target object;

[0115] The feature vector model is obtained by training an initial neural network model using a training object and positive and negative expression images corresponding to the training object as a training sample set; the positive and negative expression images include interest expression images and non-interest expression images.

[0116] In some embodiments, the initial neural network model includes a graph convolutional neural network model, and the feature vector model is trained in the following manner:

[0117] Constructing a training sample set, the training sample set including a plurality of training subjects and positive and negative expression image information of the training subjects after a preset time, the positive and negative expression image information including interest expression images and non-interest expression images;

[0118] Generate a sample bipartite topology graph of the training object based on the interest expression image information of the training object before a preset time and the similarity between the expression images therein;

[0119] Inputting the sample bipartite topology graph of the training object into the initial graph convolutional neural network model for propagation learning to obtain the initial feature vector of the training object;

[0120] Determine the initial feature vector of the training object and the sample vector similarity of the expression vector of the expression image in the positive and negative expression image information corresponding to the training object;

[0121] Based on the sample vector similarity and the image type of the corresponding expression image, the model parameters of the initial graph convolutional neural network model are optimized to obtain the graph convolutional neural network model.

[0122] In some embodiments, the step of inputting the sample bisection topology graph of the training object into an initial graph convolutional neural network model for propagation learning to obtain an initial feature vector of the training object includes:

[0123] Obtaining node feature data of each node of the sample bipartite topology graph of the training object;

[0124] Acquire connection feature data between nodes having connection edges in each node of the sample bipartite topology graph of the training object;

[0125] Based on the node feature data and the connection feature data, an initial graph convolutional neural network model is used to perform propagation learning on the sample bipartite topology graph to obtain an initial feature vector of the training object.

[0126] In some embodiments, the similarity determination module includes:

[0127] An image acquisition unit, used to acquire a key image corresponding to each to-be-recommended expression image in the to-be-recommended expression image set;

[0128] An expression vector determination unit, used to extract image features of a key image corresponding to each expression image to be recommended, and obtain an expression vector of each expression image to be recommended;

[0129] The similarity determination unit is used to determine the vector similarity between the feature vector of the target object and the expression vector of each expression image to be recommended; the feature vector and the expression vector have the same vector dimension.

[0130] In some embodiments, the recommendation module includes:

[0131] A first target expression determination unit, configured to determine, based on the vector similarity, at least one target recommended expression image corresponding to a maximum vector similarity value;

[0132] A first expression recommendation unit is configured to recommend at least one determined target recommended expression image to the target object; and / or

[0133] A first recommendation list determination unit is used to sort a plurality of to-be-recommended expression images in the to-be-recommended expression set based on the vector similarity, and determine to obtain an expression image recommendation list;

[0134] A second expression recommendation unit is configured to recommend a preset number of expression images ranked top to the target object based on the expression image recommendation list; and / or

[0135] A second target expression determination unit, used to determine at least one target recommended expression image corresponding to a vector similarity greater than a second similarity threshold;

[0136] The third expression recommendation unit is used to recommend the determined at least one target recommended expression image to the target object.

[0137] An embodiment of the present application provides a server, which may include a processor and a memory, in which at least one instruction, at least one program, a code set or an instruction set is stored, and the at least one instruction, the at least one program, the code set or the instruction set is loaded and executed by the processor to implement the expression image recommendation method provided in the above method embodiment.

[0138] An embodiment of the present application also provides a storage medium, in which at least one instruction, at least one program, code set or instruction set is stored, and the at least one instruction, at least one program, code set or instruction set is loaded by a processor and executes any of the above-mentioned expression image recommendation methods.

[0139] Further, Figure 7 A schematic diagram of the hardware structure of a device for implementing the method provided in the embodiment of the present application is shown. The device may be a computer terminal, a mobile terminal or other device. The device may also participate in constituting or include the apparatus provided in the embodiment of the present application. Figure 7 As shown, the computer terminal 10 may include one or more (102a, 102b, ..., 102n are used to illustrate) processors 102 (the processor 102 may include but is not limited to a processing device such as a microprocessor MCU or a programmable logic device FPGA), a memory 104 for storing data, and a transmission device 106 for communication functions. In addition, it may also include: a display, an input / output interface (I / O interface), a universal serial bus (USB) port (which may be included as one of the ports of the I / O interface), a network interface, a power supply and / or a camera. It can be understood by those skilled in the art that Figure 7 The structure shown is only for illustration and does not limit the structure of the above electronic device. Figure 7 More or fewer components as shown, or with Figure 7 Different configurations are shown.

[0140] It should be noted that the one or more processors 102 and / or other data processing circuits described above may generally be referred to herein as "data processing circuits". The data processing circuits may be embodied in whole or in part as software, hardware, firmware, or any other combination thereof. In addition, the data processing circuit may be a single independent processing module, or may be incorporated in whole or in part into any of the other components in the computer terminal 10 (or mobile device). As described in the embodiments of the present application, the data processing circuit acts as a processor control (e.g., selection of a variable resistor terminal path connected to an interface).

[0141] The memory 104 can be used to store software programs and modules of application software, such as program instructions / data storage devices corresponding to the methods described in the embodiments of the present application. The processor 102 executes various functional applications and data processing by running the software programs and modules stored in the memory 104, that is, to implement the above-mentioned neural network processing method. The memory 104 may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 104 may further include a memory remotely arranged relative to the processor 102, and these remote memories may be connected to the computer terminal 10 via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0142] The transmission device 106 is used to receive or send data via a network. The specific example of the above network may include a wireless network provided by a communication provider of the computer terminal 10. In one example, the transmission device 106 includes a network adapter (Network Interface Controller, NIC), which can be connected to other network devices through a base station so as to communicate with the Internet. In one example, the transmission device 106 can be a radio frequency (RF) module, which is used to communicate with the Internet wirelessly.

[0143] The display may be, for example, a touch screen liquid crystal display (LCD) that enables a user to interact with a user interface of the computer terminal 10 (or mobile device).

[0144] It should be noted that the above-mentioned sequence of the embodiments of the present application is for description only and does not represent the advantages and disadvantages of the embodiments. The above-mentioned specific embodiments of the present application are described. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in an order different from that in the embodiments and still achieve the desired results. In addition, the processes depicted in the drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0145] Each embodiment in this application is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the device and server embodiments, since they are basically similar to the method embodiments, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiments.

[0146] A person skilled in the art will understand that all or part of the steps to implement the above embodiments may be accomplished by hardware or by instructing related hardware through a program, and the program may be stored in a computer-readable storage medium, and the above-mentioned storage medium may be a read-only memory, a disk or an optical disk, etc.

[0147] The above description is only a preferred embodiment of the present application and is not intended to limit the present application. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present application should be included in the protection scope of the present application.

Claims

1. A method for recommending facial expressions, characterized in that: include: Based on the historical interest expression image information of the target object and the similarity between the expression images in the historical interest expression image information, generating a target bipartite topological graph of the target object; Determining a feature vector of the target object based on a target bipartite topological graph of the target object; The feature vector of the target object represents the feature data of the target object's interest expression image; including: based on the feature vector model, the feature vector representation learning of the target object is performed on the target binary topology map to obtain the feature vector of the target object; the feature vector model is obtained by training the initial neural network model using the training object and the positive and negative expression images corresponding to the training object as the training sample set; the positive and negative expression images include interest expression images and non-interest expression images; Determining the vector similarity between the feature vector of the target object and the expression vector of each expression image to be recommended in the set of expression images to be recommended; recommending an expression image to the target object based on the vector similarity; The initial neural network model includes a graph convolutional neural network model, and the feature vector model is trained in the following manner: Constructing a training sample set, the training sample set including a plurality of training subjects and positive and negative expression image information of the training subjects after a preset time, the positive and negative expression image information including interest expression images and non-interest expression images; Generate a sample bipartite topology graph of the training object based on the interest expression image information of the training object before a preset time and the similarity between the expression images therein; Inputting the sample bipartite topology graph of the training object into the initial graph convolutional neural network model for propagation learning to obtain the initial feature vector of the training object; Determine the initial feature vector of the training object and the sample vector similarity of the expression vector of the expression image in the positive and negative expression image information corresponding to the training object; Based on the sample vector similarity and the image type of the corresponding expression image, the model parameters of the initial graph convolutional neural network model are optimized to obtain the graph convolutional neural network model.

2. The method according to claim 1, characterized in that The step of generating a target bipartite topological graph of the target object based on the historical interest expression image information of the target object and the similarity between expression images in the historical interest expression image information includes: Based on the target object and the historical interest expression image information of the target object, a first bipartite topological graph of the target object is generated; the first bipartite topological graph includes a target object node corresponding to the target object and an expression image node corresponding to the expression image in the historical interest expression image information; Determine a connection relationship between each two expression image nodes based on the similarity between each two expression image nodes and a first similarity threshold; Based on the determined connection relationship between every two expression image nodes and the first bipartite topology graph, a target bipartite topology graph of the target object is determined.

3. The method according to claim 1, characterized in that The step of inputting the sample bisection topology graph of the training object into the initial graph convolutional neural network model for propagation learning to obtain the initial feature vector of the training object includes: Obtaining node feature data of each node of the sample bipartite topology graph of the training object; Acquire connection feature data between nodes having connection edges in each node of the sample bipartite topology graph of the training object; Based on the node feature data and the connection feature data, an initial graph convolutional neural network model is used to perform propagation learning on the sample bipartite topology graph to obtain an initial feature vector of the training object.

4. The method according to claim 1, characterized in that: The step of determining the vector similarity between the feature vector of the target object and the expression vector of each expression image to be recommended in the expression image set to be recommended includes: Obtaining a key image corresponding to each to-be-recommended expression image in the to-be-recommended expression image set; Extracting image features of the key image corresponding to each of the expression images to be recommended, and obtaining an expression vector of each of the expression images to be recommended; Determine the vector similarity between the feature vector of the target object and the expression vector of each expression image to be recommended; the feature vector and the expression vector have the same vector dimension.

5. The method according to claim 1, characterized in that The recommending the expression image to the target object based on the vector similarity includes: Based on the vector similarities, determining at least one target recommended expression image corresponding to a maximum vector similarity value; recommending the determined at least one target recommended expression image to the target object; and / or Based on the vector similarity, a plurality of to-be-recommended expression images in the to-be-recommended expression set are sorted to determine and obtain an expression image recommendation list; Based on the expression image recommendation list, recommending a preset number of expression images ranked top to the target object; and / or Determine at least one target recommended expression image corresponding to which the vector similarity is greater than a second similarity threshold; The determined at least one target recommended expression image is recommended to the target object.

6. An expression image recommendation device, characterized in that: include: A topology map generating module, used for generating a target bipartite topology map of the target object based on the historical interest expression image information of the target object and the similarity between the expression images in the historical interest expression image information; A vector determination module, used to determine a feature vector of the target object based on a target bipartite topological graph of the target object; The feature vector of the target object represents the feature data of the target object's interest expression image; including: based on the feature vector model, the feature vector representation learning of the target object is performed on the target binary topology map to obtain the feature vector of the target object; the feature vector model is obtained by training the initial neural network model using the training object and the positive and negative expression images corresponding to the training object as the training sample set; the positive and negative expression images include interest expression images and non-interest expression images; A similarity determination module, used to determine the vector similarity between the feature vector of the target object and the expression vector of each expression image to be recommended in the set of expression images to be recommended; A recommendation module, used for recommending expression images to the target object based on the vector similarity; The initial neural network model includes a graph convolutional neural network model, and the feature vector model is trained in the following manner: Constructing a training sample set, the training sample set including a plurality of training subjects and positive and negative expression image information of the training subjects after a preset time, the positive and negative expression image information including interest expression images and non-interest expression images; Generate a sample bipartite topology graph of the training object based on the interest expression image information of the training object before a preset time and the similarity between the expression images therein; Inputting the sample bipartite topology graph of the training object into the initial graph convolutional neural network model for propagation learning to obtain the initial feature vector of the training object; Determine the initial feature vector of the training object and the sample vector similarity of the expression vector of the expression image in the positive and negative expression image information corresponding to the training object; Based on the sample vector similarity and the image type of the corresponding expression image, the model parameters of the initial graph convolutional neural network model are optimized to obtain the graph convolutional neural network model.

7. The device according to claim 6, characterized in that The topology map generation module includes: A first topology map generating unit, configured to generate a first bipartite topology map of the target object based on the target object and the historical interest expression image information of the target object; the first bipartite topology map includes a target object node corresponding to the target object and an expression image node corresponding to the expression image in the historical interest expression image information; A connection relationship determining unit, configured to determine a connection relationship between every two expression image nodes based on a similarity between the two expression image nodes and a first similarity threshold; The target topology map generating unit is used to determine the target bipartite topology map of the target object based on the determined connection relationship between every two expression image nodes and the first bipartite topology map.

8. The device according to claim 6, characterized in that The step of inputting the sample bisection topology graph of the training object into the initial graph convolutional neural network model for propagation learning to obtain the initial feature vector of the training object includes: Obtaining node feature data of each node of the sample bipartite topology graph of the training object; Acquire connection feature data between nodes having connection edges in each node of the sample bipartite topology graph of the training object; Based on the node feature data and the connection feature data, an initial graph convolutional neural network model is used to perform propagation learning on the sample bipartite topology graph to obtain an initial feature vector of the training object.

9. The device according to claim 6, characterized in that The similarity determination module comprises: An image acquisition unit, used to acquire a key image corresponding to each to-be-recommended expression image in the to-be-recommended expression image set; An expression vector determination unit, used to extract image features of a key image corresponding to each expression image to be recommended, and obtain an expression vector of each expression image to be recommended; The similarity determination unit is used to determine the vector similarity between the feature vector of the target object and the expression vector of each expression image to be recommended; the feature vector and the expression vector have the same vector dimension.

10. The device according to claim 6, characterized in that The recommendation module includes: A first target expression determination unit, configured to determine, based on the vector similarity, at least one target recommended expression image corresponding to a maximum vector similarity value; A first expression recommendation unit is configured to recommend at least one determined target recommended expression image to the target object; and / or A first recommendation list determination unit is used to sort a plurality of to-be-recommended expression images in the to-be-recommended expression set based on the vector similarity, and determine to obtain an expression image recommendation list; A second expression recommendation unit is configured to recommend a preset number of expression images ranked top to the target object based on the expression image recommendation list; and / or A second target expression determination unit, used to determine at least one target recommended expression image corresponding to a vector similarity greater than a second similarity threshold; The third expression recommendation unit is used to recommend the determined at least one target recommended expression image to the target object.

11. A device, characterized in that: The device includes a processor and a memory, wherein the memory stores at least one instruction, at least one program, a code set or an instruction set, and the at least one instruction, the at least one program, the code set or the instruction set is loaded and executed by the processor to implement the expression image recommendation method as described in any one of claims 1 to 5.

12. A computer storage medium, characterized in that: The storage medium stores at least one instruction, at least one program, code set or instruction set, and the at least one instruction, at least one program, code set or instruction set is loaded by the processor and executed by the expression image recommendation method as described in any one of claims 1 to 5.

Images