Unsupervised cross-modal-based comment recommendation method and system, electronic equipment and storage medium

Abstract

The invention provides an unsupervised cross-modal-based comment recommendation method and system, electronic equipment and a storage medium, and the method comprises the steps: calculating a Hamming distance between a Hash code of published information of a user on a social platform and a Hash code of a multi-modal comment through a comment recommendation model, and outputting the multi-modal comment if the Hamming distance is not smaller than a target Hamming distance; the training process of the comment recommendation model comprises the following steps: reconstructing a picture similarity matrix and a text cosine similarity matrix of a constructed target instance group into a similarity matrix; constructing a pairwise loss function according to the similarity matrix and the cosine distance of the target instance group, constructing a hard classification loss function according to the hard and pseudo labels of the target instances, and constructing a soft classification loss function according to the soft and pseudo labels of the target instances to adjust a comment recommendation model to be trained to obtain an initial comment recommendation model; and if the initial comment recommendation model passes verification, determining the initial comment recommendation model as a comment recommendation model, and determining the Hamming distance of the initial comment recommendation model as a target Hamming distance.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and more specifically, to a review recommendation method, system, electronic device and storage medium based on unsupervised cross-modality. Background Art

[0002] With the widespread use of various smart electronic devices, people can browse and express their opinions on social platforms through smart electronic devices, and the forms of expression are also diverse, including data in multiple modes such as text, images and videos. When a user posts a piece of text, he can add a picture or video that fits the theme, or when a user posts a picture or video, he can also add a piece of text that fits the picture or video. After posting his own comments or opinions, users can find friends with similar interests and hobbies based on similar comments recommended by others, and improve their user activity on social platforms.

[0003] In the prior art, the comment system of a social platform can recommend similar comments based on given tags, such as "food", "cute pets", "characters", etc. This method requires users to mark the tags of their own comments in advance so that the system can recommend corresponding comments based on the tags marked by the users. Although this method can recommend comments corresponding to the tags marked by the users, it cannot recommend corresponding comments based on the content or comments posted by the users. It is easy for the recommended comments to have low relevance or even no relevance to the content or comments posted by the current user, resulting in a poor user experience. Summary of the invention

[0004] In view of this, the present application provides a comment recommendation method, system, electronic device and storage medium based on unsupervised cross-modality to solve the problem in the prior art that it is impossible to recommend corresponding comments based on the content or comments posted by the user, and it is easy for the recommended comments to have low relevance or even no relevance to the content or comments posted by the current user, thereby resulting in a poor user experience.

[0005] The first aspect of the present application provides a review recommendation method based on unsupervised cross-modality, the method comprising:

[0006] Obtaining the target user's published information on the social platform, and obtaining a plurality of multimodal comments corresponding to the published information; wherein the published information includes a published image and a published text; and each of the multimodal comments includes a comment image and a comment text;

[0007] For each of the multimodal reviews, input the release information and the multimodal reviews into a review recommendation model, so that the review recommendation model calculates the Hamming distance between the hash code of the release information and the hash code of the multimodal reviews, and if the Hamming distance is not less than a target Hamming distance, determines the multimodal review as a recommended review, and outputs the recommended review;

[0008] The training process of the review recommendation model includes:

[0009] Acquire a training data set, and filter out multiple target instances from multiple instances in the training data set; wherein the instances include images and texts;

[0010] Combining each of the target instances in pairs to obtain multiple target instance groups;

[0011] Constructing an image similarity matrix and a text cosine similarity matrix for each target instance group, and reconstructing the image similarity matrix and the text cosine similarity matrix for each target instance group into a similarity matrix;

[0012] Calculating the cosine distance between two target instances in each of the target instance groups, and constructing a corresponding pairwise loss function according to the similarity matrix and the cosine distance of each of the target instance groups;

[0013] Constructing a corresponding hard classification loss function according to the hard pseudo labels of each of the target instances;

[0014] Constructing a corresponding soft classification loss function according to the soft pseudo labels of each of the target instances;

[0015] The pairwise loss function, the hard classification loss function and the soft classification loss function are used to adjust the parameters of the review recommendation model to be trained to obtain an initial review recommendation model;

[0016] Acquire a validation data set, and use each validation instance in the validation data set to validate the initial review recommendation model, and determine the Hamming distance of the initial review recommendation model;

[0017] If the initial review recommendation model passes the verification, the initial review recommendation model is determined as the review recommendation model, and the Hamming distance of the initial review recommendation model is determined as the target Hamming distance.

[0018] Optionally, the acquiring a training data set and filtering out a plurality of target instances from a plurality of instances in the training data set includes:

[0019] Acquire a training data set, wherein the training data set includes multiple instances, each of which includes a picture and text;

[0020] Extracting deep features of each of the images, and clustering the images according to the deep features of the images to obtain multiple clusters;

[0021] For each of the clusters, multiple target images are screened out from the images of the cluster according to the Euclidean distance from the cluster center, and the instance to which each of the target images belongs is determined as the target instance.

[0022] Optionally, constructing an image similarity matrix and a text cosine similarity matrix for each target instance group, and reconstructing the image similarity matrix and the text cosine similarity matrix of each target instance group into a similarity matrix, comprises:

[0023] constructing an image similarity matrix of the target instance group according to the hard pseudo labels of the target images in the two target instances in the target instance group;

[0024] Constructing a text cosine similarity matrix of the target instance group according to the bag-of-words vector of the target text in each target instance in the target instance group;

[0025] According to the image similarity matrix and the text cosine similarity matrix of each target instance group, a corresponding similarity matrix is ​​constructed.

[0026] Optionally, calculating the cosine distance between two target instances in each target instance group and constructing a corresponding pairwise loss function according to the similarity matrix and the cosine distance of each target instance group includes:

[0027] Calculating hash codes of two target instances in the target instance group, and calculating a Hamming distance between the two target instances in the target instance group and deriving a cosine distance according to an inner product of the hash codes of the two target instances and a hash code length;

[0028] A corresponding pairwise loss function is constructed according to the similarity matrix and the cosine distance between two target instances in each of the target instance groups.

[0029] Optionally, constructing a corresponding hard classification loss function according to the hard pseudo labels of each of the target instances includes:

[0030] Acquire a hard false label of each of the target instances, wherein the hard false label of the target instance includes a hard false label of a target image in the target instance and a hard false label of a target text in the target instance;

[0031] Constructing a corresponding image hard classification loss function according to the target image of each target instance and its hard pseudo label;

[0032] Constructing a corresponding text hard classification loss function according to the target image of each target instance and its hard pseudo label;

[0033] A corresponding hard classification loss function is constructed according to the image hard classification loss function and the text hard classification loss function.

[0034] Optionally, constructing a corresponding soft classification loss function according to the soft pseudo labels of each of the target instances includes:

[0035] Acquire a soft false label of each of the target instances, wherein the soft false label of the target instance includes a soft false label of a target image in the target instance and a soft false label of a target text in the target instance;

[0036] Constructing a corresponding image soft classification loss function according to the target image and its soft pseudo label of each target instance;

[0037] Constructing a corresponding text hard classification loss function according to the target image and soft pseudo label of each target instance;

[0038] A corresponding soft classification loss function is constructed according to the image hard classification loss function and the text soft classification loss function.

[0039] Optionally, the obtaining of a verification data set, and verifying the initial review recommendation model using each verification instance in the verification data set, and determining the Hamming distance of the initial review recommendation model include:

[0040] Acquire a verification data set, wherein the verification data includes a plurality of verification instance groups, the verification instance groups include two verification instances, and the verification instances include a verification picture and a verification text;

[0041] For each of the verification instance groups, inputting the verification instance group into an initial review recommendation model;

[0042] The initial comment recommendation model utilizes the hash code of the image of each verification instance in the verification instance group, and extracts the hash code of the text of each verification instance in the verification instance group; performs corresponding Hamming distance calculations based on the hash codes of the images and the hash codes of the texts of two verification instances in the verification instance group to obtain multiple Hamming distances; if there is a Hamming distance of not less than the verification instance group among the Hamming distances, it is determined that the verification of the initial comment recommendation model has passed, wherein the multiple Hamming distances include image-image Hamming distance, text-text Hamming distance and image-text Hamming distance.

[0043] The second aspect of the present application provides a review recommendation system based on unsupervised cross-modality, the system comprising:

[0044] A first acquisition unit is used to acquire the published information of the target user on the social platform, and acquire multiple multimodal comments corresponding to the published information; wherein the published information includes a published picture and a published text; each of the multimodal comments includes a comment picture and a comment text;

[0045] A comment recommendation unit, for each of the multimodal comments, inputting the release information and the multimodal comment into a comment recommendation model, causing the comment recommendation model to calculate a Hamming distance between a hash code of the release information and a hash code of the multimodal comment, and if the Hamming distance is not less than a target Hamming distance, determining the multimodal comment as a recommended comment, and outputting the recommended comment;

[0046] The training process of the review recommendation model includes:

[0047] A second acquisition unit is used to acquire a training data set and filter out a plurality of target instances from a plurality of instances in the training data set; wherein the instances include images and texts;

[0048] A combining unit, used for combining each of the target instances in pairs to obtain a plurality of target instance groups;

[0049] A similarity matrix construction unit, used to construct an image similarity matrix and a text cosine similarity matrix for each target instance group, and reconstruct the image similarity matrix and the text cosine similarity matrix of each target instance group into a similarity matrix;

[0050] A pairwise loss function construction unit, used for calculating the cosine distance between two target instances in each of the target instance groups, and constructing a corresponding pairwise loss function according to the similarity matrix and the cosine distance of each of the target instance groups;

[0051] A hard classification loss function construction unit, used to construct a corresponding hard classification loss function according to the hard pseudo labels of each target instance;

[0052] A soft classification loss function construction unit, used to construct a corresponding soft classification loss function according to the soft pseudo labels of each target instance;

[0053] A parameter adjustment unit, configured to adjust parameters of the review recommendation model to be trained by using the pairwise loss function, the hard classification loss function, and the soft classification loss function to obtain an initial review recommendation model;

[0054] A verification unit, used to obtain a verification data set, and verify the initial review recommendation model using each verification instance in the verification data set, and determine the Hamming distance of the initial review recommendation model;

[0055] A determination unit is configured to determine the initial review recommendation model as the review recommendation model if the initial review recommendation model passes verification, and to determine the Hamming distance of the initial review recommendation model as a target Hamming distance.

[0056] The third aspect of the present application provides an electronic device, comprising: a processor and a memory, wherein the processor and the memory are connected via a communication bus; wherein the processor is used to call and execute a program stored in the memory; and the memory is used to store a program, wherein the program is used to implement the unsupervised cross-modal comment recommendation method provided in the first aspect of the present application.

[0057] The fourth aspect of the present application provides a computer-readable storage medium, in which computer-executable instructions are stored, and the computer-executable instructions are used to execute the unsupervised cross-modal review recommendation method provided in the first aspect of the present application.

[0058] The present application provides a method, system, electronic device and storage medium for comment recommendation based on unsupervised cross-modality. First, a massive amount of multimodal training data sets that have not been manually annotated are obtained, and various noises generated by clustering of these data are reduced through unsupervised data balancing, and hard-pseudo labels with discriminative capabilities are generated; first, the semantic information of different modalities of images and texts of instances in the training data set and hard-pseudo labels are combined to construct a corresponding similarity matrix; secondly, the inter-modal and modality semantic reconstruction losses are constructed by combining the similarity matrices to ensure that the hash codes generated by instances with similar topics are more similar; finally, the hard classification loss function and the soft classification loss function constructed using the hard-pseudo labels and the soft-pseudo labels are used to construct the inter-modal and modality semantic reconstruction, hard classification loss and soft classification loss functions. The loss function and the soft classification loss function supervise the training of the comment recommendation model to be trained, so as to improve the accuracy of the model retrieval; after the corresponding comment recommendation model is trained, the target user's published information on the social platform and the corresponding multimodal comments can be input into the comment recommendation model, so that the comment recommendation model can calculate the Hamming distance between the hash code of the published information and the hash code of the multimodal comment, and accurately judge whether the multimodal comment is a recommended comment that needs to be recommended to the target user according to the calculated Hamming distance, thereby solving the problem in the prior art that it is impossible to recommend corresponding comments based on the content or comments published by the user, and it is easy for the recommended comments to have low relevance or even no relevance to the content or comments published by the current user, thus resulting in a poor user experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0059] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying any creative work.

[0060] Figure 1 An example diagram of a training network architecture of a review recommendation model provided in an embodiment of the present application;

[0061] Figure 2 A flowchart of a method for training a review recommendation model to be trained to obtain a corresponding review recommendation model provided in an embodiment of the present application;

[0062] Figure 3 A flowchart of a review recommendation method based on unsupervised cross-modality is provided for an embodiment of the present application;

[0063] Figure 4 A schematic diagram of the structure of an unsupervised cross-modal review recommendation system provided in an embodiment of the present application;

[0064] Figure 5 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0065] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0066] In this application, relational terms such as first and second, etc. are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "comprise", "include" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the statement "comprises a ..." does not exclude the presence of other identical elements in the process, method, article or device including the element.

[0067] In order to better understand this application, the technical names involved in this application are explained below:

[0068] Cross-modal retrieval: Modality refers to the form of data, such as images, text, video, audio, etc. Compared with traditional single-modal retrieval of searching for text by text or searching for images by images, cross-modal retrieval is to return samples of another modality (such as images) with similar themes given a query sample of one modality (such as text).

[0069] Unsupervised learning: Train models based on samples of unknown categories to solve various problems in pattern recognition.

[0070] Hash algorithm: Hash is also called hashing. It transforms input of any length into a binary output of fixed length through a hash algorithm. The output is the hash value. This conversion is a compression mapping, and different inputs may hash to the same output.

[0071] Pseudo-labeling: As a traditional semi-supervised machine learning technology, the original purpose of pseudo-labeling learning is to use labeled data to train the model, then use the trained model to predict the labels of unlabeled data as pseudo-labels, and finally combine two different labeled data as training sets to train the final model. Due to the powerful knowledge transfer ability of pre-trained neural networks, pseudo-labeling is not only effective in semi-supervised machine learning, but also has achieved good performance in unsupervised deep learning.

[0072] Bag of words model: puts all words into a bag without considering grammar and word order. At this time, each word is independent of each other. There are three methods for calculating word values ​​in the bag of words model, namely one-hot representation, term frequency (TF) representation, and term frequency-inverse document frequency (TF-IDF) representation. This article uses the term frequency-inverse document frequency representation.

[0073] Convolutional neural network: It has three basic characteristics: weight sharing, local connection and downsampling. These three characteristics have the effect of reducing network parameters and speeding up training. Weight sharing refers to the use of the same filter to scan the feature map of the same depth, reducing the number of parameters and calculations during training, and enhancing the generalization ability of the model. Local connection means that the nodes in the next layer are only partially connected to the nodes in the previous layer, which is used to learn local features. This idea is mainly derived from the cortical structure of animal vision, which reflects the local correlation that the correlation between pixels with a closer distance is stronger, and the correlation between pixels with a farther distance is weaker. Downsampling refers to proportionally reducing the width and height of the feature map, reducing the dimension of the result, and avoiding overfitting.

[0074] Convolutional neural network: It is mainly composed of convolutional layers, ReLU layers, pooling layers, and fully connected layers. After an image passes through the convolutional layer, it is extracted into features of different levels. The deeper the convolutional layer, the more abstract the features extracted. Subsequently, after the feature map passes through the Rectified Linear Unit (ReLU) layer, the nonlinear non-saturated activation function changes the function learned by the neural network from straight to curved, enhancing the nonlinear expression ability of the model. After that, the maximum pooling or average pooling of the pooling layer reduces the dimension of the feature map, effectively reducing the amount of calculation. Finally, after the flattened high-level features pass through the fully connected layer, Softmax activation, and one-hot vectorization, they are finally transformed into a set of labels corresponding to the downstream classification task.

[0075] K-Means clustering algorithm: It is a distance-based clustering algorithm that uses distance as the evaluation index of similarity, that is, it is believed that the closer the distance between two objects, the greater their similarity. The main idea of ​​this algorithm is to assign each point to the cluster represented by the cluster center closest to it given the K value and K initial cluster center points. After all points are assigned, the center point of a cluster is recalculated based on all points in the cluster, and then the steps of assigning points and updating cluster center points are iterated until the change of cluster center point is small or the specified number of iterations is reached.

[0076] From the above background technology, it can be seen that the current comment system requires users to mark their own comments in advance before recommending corresponding comments based on the tags marked by the users; although this method can recommend comments corresponding to the tags marked by the users, it cannot recommend corresponding comments based on the content or comments posted by the users, and it is easy for the recommended comments to have low relevance or even no relevance to the content or comments posted by the current user, resulting in a poor user experience.

[0077] In the prior art, corresponding comments can also be recommended to users through traditional cross-modal retrieval. Specifically, most traditional cross-modal methods map heterogeneous modal data into a joint common subspace through subspace learning, and complete data similarity matching in the common space. However, with the explosive growth of different modal data on the Internet, these traditional algorithms are facing problems such as high computational complexity and low search accuracy, resulting in low similarity between the comments recommended to users and the content or comments posted by the users themselves. In addition, a large number of cross-modal retrieval studies currently use processed manually annotated data to achieve significant improvements in retrieval performance. However, in real life, massive data is in its original unlabeled state, and the labeling of these data requires a lot of manpower and time costs.

[0078] Therefore, the present application provides a review recommendation method, system, electronic device and storage medium based on unsupervised cross-modal review recommendation. First, a large amount of multimodal training data sets that have not been manually annotated are obtained, and various noises generated by these data clusters are reduced through unsupervised data balancing, and hard-pseudo labels with discriminative capabilities are generated; first, the semantic information of different modalities of the images and texts of the instances in the training data set and the hard-pseudo labels are combined to construct a corresponding similarity matrix; secondly, the similarity matrix is ​​combined to construct inter-modal and modality semantic reconstruction losses to ensure that the hash codes generated by instances with similar topics are more similar; finally, the hard classification loss function and the soft classification loss function are constructed using the hard-pseudo labels and the soft-pseudo labels, so as to utilize the constructed inter-modal and modality semantic reconstruction, hard-pseudo labels and soft-pseudo labels to construct a loss function. The classification loss function and the soft classification loss function supervise the training of the comment recommendation model to be trained to improve the accuracy of model retrieval; after the corresponding comment recommendation model is trained, the target user's published information on the social platform and the corresponding multimodal comments can be input into the comment recommendation model, so that the comment recommendation model can calculate the Hamming distance between the hash code of the published information and the hash code of the multimodal comment, and accurately judge whether the multimodal comment is a recommended comment that needs to be recommended to the target user based on the calculated Hamming distance, thereby solving the problem in the prior art that it is impossible to recommend corresponding comments based on the content or comments posted by the user, and it is easy for the recommended comments to have a low correlation or even no correlation with the content or comments posted by the current user, resulting in a poor user experience.

[0079] See also Figure 1 , showing an example diagram of a training network architecture of a comment recommendation model provided in an embodiment of the present application, the training network architecture can be divided into an online stage and an offline stage, wherein a trained AlexNet network is deployed in the offline stage, and a comment recommendation model to be trained is deployed in the online stage.

[0080] In an embodiment of the present application, for a comment recommendation model to be trained deployed in an online stage, the comment recommendation model to be trained includes an image encoder and a text encoder, and the trained AlexNet network can be used as an image encoder; wherein the trained AlexNet includes five convolutional layers and two fully connected layers and a fully connected layer with R hidden nodes, R=2k0+r, and the 2k0 nodes are used to predict the hard pseudo-label and soft pseudo-label of the image respectively; the r nodes are used to learn the hash code, that is, to extract the hash code of the image; Relu is used as the activation function of the five convolutional layers and the first two fully connected layers, and Tanh and Softmax are used as the activation functions of the hash code and the hard / soft pseudo-label module of the last fully connected layer (a fully connected layer with R hidden nodes) respectively;

[0081] A multi-layer perceptron (MLP) is used as a text encoder, wherein the text encoder uses a bag-of-words vector as input; wherein the text encoder includes two fully connected layers; the first fully connected layer in the text encoder has 4096 hidden nodes and uses relu as an activation function; the second fully connected layer in the text encoder includes R=2k0+r hidden nodes, 2k0 nodes are used to predict the hard pseudo-label and soft pseudo-label of the text, and r nodes are used to learn hash codes, and Tanh and Softmax are used as activation functions of the hash code and hard / soft pseudo-label modules, respectively.

[0082] based on Figure 1 The training network architecture shown in FIG. 1 is shown in FIG. 1 . Accordingly, the embodiment of the present application provides a method for training a review recommendation model to be trained to obtain a corresponding review recommendation model, such as Figure 2 As shown, the method specifically comprises the following steps:

[0083] S201: Obtain a training data set, and filter out multiple target instances from multiple instances in the training data set; wherein the instances include images and texts.

[0084] In the specific process of executing step S201, a training data set can be obtained, wherein the training data set includes multiple multimodal instances, each of which includes images and texts; each instance is clustered to obtain multiple clusters, and multiple target images are screened out from each instance of each cluster, and finally the instances to which each screened target image belongs are determined as target instances.

[0085] Optionally, the process of filtering out multiple target instances from multiple instances in the training data set may be: extracting deep features of each image, and clustering each image according to the deep features of each image to obtain multiple clusters; for each cluster, filtering out multiple target images from each image of the cluster according to the Euclidean distance from the cluster center, and determining the instance to which each target image belongs as a low-noise target instance (i.e., the target instance).

[0086] In the actual application process, the applicant has found that due to the diversity of real scenes, a picture or a text in an instance may contain multiple labels. Similarly, different labels may have the same potential theme. Therefore, it is assumed that there are k0 potential class attributes in the training data set, and each instance corresponds to a potential class attribute. However, due to the influence of multiple labels, k0 may be different from the number of real labels. Based on the powerful knowledge transfer ability of neural networks, clustering algorithms have been proven to be an effective method for generating hard pseudo-labels in unsupervised learning. However, there are still various noises in clustering algorithms that mislead the normal training of the model, among which data noise comes from the misclassification of outliers, and label noise comes from the limited ability of clustering algorithms in multi-classification tasks.

[0087] Therefore, in order to reduce data noise and outlier misclassification, the present application can extract the deep features of the image in each instance through the image feature extraction function in the AlexNet network deployed in the offline stage, as shown in formula (1); then, the classic k-means clustering algorithm is used to divide each image into k0 clusters according to the deep features of each image; a hard false label is marked for each image in the instance Finally, since the closer the distance to the cluster center is, the higher the confidence of the hard false label is, and the less the misclassification of the training data is, we can search for an equal number of n / k0 target images closest to the cluster centroid in each image of each cluster according to the Euclidean distance from the cluster center, and at the same time use their index Γ for training; where n is the number of images, that is, the data of each instance in the training data set.

[0088] F i v =f v (x i |θ p ) (1)

[0089] Among them, f v represents the image feature extraction function, θ p Represents the parameters of the AlexNet network, x i represents the i-th instance, F i v Represents the deep features of the image in the i-th instance.

[0090] S202: Combine target instances in pairs to obtain multiple target instance groups.

[0091] In the specific process of executing step S202, after a plurality of target instances are screened out from each instance of the training data set, each instance may be combined in pairs to obtain a plurality of target instance groups.

[0092] S203: constructing an image similarity matrix and a text cosine similarity matrix for each target instance group, and reconstructing the image similarity matrix and the text cosine similarity matrix of each target instance group into a similarity matrix.

[0093] In the specific process of executing step S203, after obtaining multiple target instance groups, for each target instance group, an image similarity matrix of the target instance group is constructed according to the hard pseudo labels of the target images in the two target instances in the target instance group; a text cosine similarity matrix of the target instance group is constructed according to the bag-of-words vectors of the target text in each target instance in the target instance group; and a corresponding similarity matrix is ​​constructed according to the image similarity matrix and the text cosine similarity matrix of each target instance group.

[0094] In actual application, given that the deep features of the images extracted by the AlexNet network contain semantic information about wind and rain, the K nearest neighbor search for similar instances can be performed through the Euclidean distance of the deep features of different images to construct a corresponding image similarity matrix, in which the target instances within the k nearest neighbors of each target instance are similar; specifically, for each target instance group, the hard pseudo labels of the target images in each target instance in the target instance group can be obtained, and whether the two target instances are similar instances can be judged based on the hard pseudo labels of the target images in the two target instances, so as to construct the image similarity matrix of the target instance group according to the corresponding judgment results. The specific construction method is shown in formula (2).

[0095]

[0096] Among them, x i is the i-th target instance, x j is the jth target instance, (S ve ) ij For the target instance x i and the target instance x j The image similarity matrix.

[0097] In some embodiments, if the target instance x i The hard pseudo labels of the target images and the target instances x in j The hard pseudo labels of the target images in are the same, so it can be considered that the target instance x j is the target instance x i K nearest neighbors, otherwise, the target instance x is considered j Not the target instance x i The K nearest neighbors of .

[0098] In practical applications, in order to capture multi-layer semantic information between instances, the cosine similarity matrix of deep features can be used to protect the original neighborhood information. Specifically, the bag-of-words vector of the target text in each target instance in the target instance group can be extracted through the text feature extraction function in the AlexNet network, and the text cosine similarity matrix of the target instance group can be constructed based on the bag-of-words vectors of the target text in the two target instances in the target instance group. The specific construction method is shown in formula (3).

[0099]

[0100] Among them, S tc is the text cosine similarity matrix, is the bag-of-words vector of the text in the i-th target instance, is the bag-of-words vector of the text in the jth target instance.

[0101] In some embodiments, after obtaining the image similarity matrix and the text cosine similarity matrix of each target instance group, the final similarity matrix S can be reconstructed by linearly combining the image similarity matrix and the text cosine similarity matrix of each target instance group.

[0102] S204: Calculate the cosine distance between two target instances in each target instance group, and construct a corresponding pairwise loss function according to the similarity matrix and the cosine distance of each target instance group.

[0103] In the specific process of executing step S204, after constructing the corresponding similarity matrix, for each target instance group, the hash codes of the two target instances in the target instance group can be calculated, and based on the inner product of the hash codes of the two target instances and the hash code length, the Hamming distance of the two target instances in the target instance group can be calculated and the cosine distance can be derived; based on the similarity matrix and the cosine distance of the two target instances in each target instance group, the corresponding pairwise loss function is constructed.

[0104] In the actual application process, after the corresponding similarity matrix is ​​constructed, for each target instance group, the hash code of the target image of each target instance in the target instance group can be calculated by the hash module in the image encoder deployed in the online stage, and the inner product of the hash codes of the target images of the two target instances can be calculated, so as to calculate the Hamming distance of the hash codes of the target images of the two target instances according to the inner product of the hash codes of the target images of the two target instances and the hash code length. The specific calculation method is shown in formula (4). Based on the relationship between the Hamming distance and the cosine distance (as shown in formula (5)), the Hamming distance of the hash codes of the images of the two target instances is brought into formula (5), and the cosine distance of the two target instances can be reversely derived.

[0105]

[0106] Among them, dis H (b i ,b j ) is the Hamming distance between the hash codes of the target images of the i-th and j-th target instances, < b i ,b j > is the inner product of the hash codes of the target images of the i-th and j-th target instances, r is the hash code length, b i is the hash code vector of the hash code of the target image of the i-th target instance, b j is the hash code vector of the hash code of the target image of the j-th target instance.

[0107]

[0108] Among them, cos(b i ,b j )=<b i ,b j > / ||b i || F ||b j || F is the cosine distance between the i-th and j-th target instances.

[0109] In some embodiments, after obtaining the cosine distance between two target instances in each target instance group, the similarity consistency between the Hamming space and the feature space can be ensured by using a given similarity matrix S. The similarity matrix S and the cosine distance cos(b i ,b j ), and construct the corresponding pairwise loss function (inter-modality and modality semantic reconstruction loss function), as shown in formula (6). The smaller or larger the Hamming distance between the target images of the two target instances, the greater or smaller the similarity between the two target instances.

[0110]

[0111] in, Represents the semantic reconstruction loss between modalities, which can effectively reduce the semantic gap between different modalities. denotes the semantic reconstruction loss between image and text modalities, respectively, λ1 denotes the hyperparameter for balancing the semantic reconstruction loss between modalities and within modalities, L1 is the semantic reconstruction loss function between modalities and modality, and B t is the hash code matrix of the target text, B v is the hash code matrix of the target image.

[0112] It should be noted that, as can be seen from formula (6), in addition to determining the cosine distance of the hash codes of the target images of the two target instances, the cosine distance of the hash codes of the target texts of the two target instances can be further determined, so as to reconstruct the corresponding inter-image modality semantic reconstruction loss function according to the similarity matrix and the cosine distance of the hash codes of the target images of the two target instances in each target instance group, and reconstruct the corresponding inter-image modality semantic reconstruction loss function according to the similarity matrix and the cosine distance of the hash codes of the target texts of the two target instances in each target instance group; wherein, the hash code of the target text of the target instance can be calculated by the hash module in the text encoder deployed in the online stage, and the method for determining the cosine distance of the hash codes of the target texts of the two target instances is the same as the method for determining the cosine distance of the hash codes of the target images of the two target instances, which can be referred to the determination process of the cosine distance of the hash codes of the target images of the two target instances, and will not be repeated in this embodiment of the present application.

[0113] S205: Construct a corresponding hard classification loss function according to the hard pseudo labels of each target instance.

[0114] In the specific process of executing step S205, the hard-false labels of each target instance are obtained, wherein the hard-false labels of the target instance include the hard-false labels of the target images in the target instance and the hard-false labels of the target text in the target instance; a corresponding image hard classification loss function is constructed according to the target images of each target instance and their hard-false labels; a corresponding text hard classification loss function is constructed according to the target images of each target instance and their hard-false labels; and a corresponding hard classification loss function is constructed according to the image hard classification loss function and the text hard classification loss function.

[0115] The applicant has found that most unsupervised cross-modal methods only consider the semantic similarity matrix reconstruction loss due to the lack of semantic labels, which limits the discrimination ability of hash codes. With the help of unsupervised data balancing, the applicant extracts the hard pseudo labels of the images in each target instance through the k-means clustering algorithm in the offline stage, so as to effectively embed the label semantic information into the hash code in the subsequent training process of the evaluation recommendation model to be trained, and uses the generalized classification cross entropy loss to define the image hard classification loss.

[0116] Specifically, the hard false labels of the images in each target instance can be extracted by using the k-means clustering algorithm in the offline stage, so as to construct the corresponding image hard classification loss function according to the image feature extraction function in the AlexNet network, the hard false label classifier, the target images of each target instance and their hard false labels. The specific construction method is shown in formula (7).

[0117]

[0118] Among them, L vh (θ v ) is the image hard classification loss function, In formula (7), it represents the hard pseudo label of the image in the target instance, n is the number of each target instance, and x i The target image represented in this formula, f v represents the image feature extraction function, C vh represents the hard pseudo-label classifier with activation function in the image network, θ v Image neural network parameters.

[0119] The hard label module in the text encoder in the online stage extracts the hard pseudo label of the text in each target instance, so as to construct the corresponding text hard classification loss function according to the text feature extraction function in the AlexNet network, the hard pseudo label classifier, the target text of each target instance and its hard pseudo label. The specific construction method is shown in formula (8).

[0120]

[0121] Among them, L th (θ t ) is the text hard classification loss function, In formula (8), it represents the hard pseudo label of the text in the target instance, n is the number of each target instance, and y i In this formula, it represents the target text, f t represents the text feature extraction function, C th represents the hard pseudo-label classifier with activation function in the text network, θ t Text neural network parameters.

[0122] In some embodiments, after obtaining the image hard classification loss function and the text hard classification loss function, the image hard classification loss function and the text hard classification loss function can be combined to obtain the hard classification loss function. The specific combination method is shown in formula (9).

[0123] L2=L vh +L th . (9)

[0124] Among them, L2 is the hard classification loss function.

[0125] S206: Construct a corresponding soft classification loss function according to the soft pseudo labels of each target instance.

[0126] In the process of specifically executing step S206, the soft pseudo labels of each target instance are obtained, wherein the soft pseudo labels of the target instance include the soft pseudo labels of the target images in the target instance and the soft pseudo labels of the target texts in the target instance; a corresponding image soft classification loss function is constructed according to the target images of each target instance and their soft pseudo labels; a corresponding text hard classification loss function is constructed according to the target images of each target instance and their soft pseudo labels; a corresponding soft classification loss function is constructed according to the image hard classification loss function and the text soft classification loss function.

[0127] In this application, the soft pseudo-label classifiers corresponding to the image and text learning networks are represented as C vs and C ts ,In order to simultaneously train the image-text collaborative network, the paired image-text batches are provided to the two networks. A simple method is adopted to train the collaborative network. The confidence vector output by the above classifier is directly used as a soft pseudo label to train the other network. The specific construction method is shown in formula (10).

[0128]

[0129] Among them, L vs (θ v ) is the image soft classification loss function, In formula (10), it represents the soft pseudo label learned by the text network, n is the number of target instances, and x i In this formula, it represents the image of the target instance, f v represents the image feature extraction function, C vs represents the soft pseudo-label classifier with activation function in the image network, θ v Image neural network parameters.

[0130] The soft label module in the text encoder in the online stage extracts the soft pseudo label of the text in each target instance, so as to construct the corresponding text soft classification loss function according to the text feature extraction function in the AlexNet network, the soft pseudo label classifier, the target text of each target instance and its soft pseudo label. The specific construction method is shown in formula (11).

[0131]

[0132] Among them, L ts (θ t ) is the text soft classification loss function, In formula (12), it represents the soft pseudo label learned by the image network, n is the number of target instances, and y i In this formula, it represents the text of the target instance, f t represents the text feature extraction function, C tsrepresents the soft pseudo-label classifier with activation function in the text network, θ t Text neural network parameters.

[0133] In some embodiments, after obtaining the image soft classification loss function and the text soft classification loss function, the image soft classification loss function and the text soft classification loss function can be combined to obtain the soft classification loss function. The specific combination method is shown in formula (12).

[0134] L3=L vs +L ts .(12)

[0135] Among them, L3 is the soft classification loss function.

[0136] S207: Parameters of the review recommendation model to be trained are adjusted using a pairwise loss function, a hard classification loss function, and a soft classification loss function to obtain an initial review recommendation model.

[0137] In the specific process of executing step S207, after constructing the pairwise loss function, the hard classification loss function and the soft classification loss function, the final loss function can be obtained by linearly combining the hard classification loss function, the soft classification loss function and the intra-modal and inter-modal semantic reconstruction loss functions. The parameters of the image encoder and the text encoder in the evaluation model to be trained are adjusted using the final constructed loss function until the image encoder and the text encoder in the evaluation model to be trained converge, and the corresponding initial comment recommendation model is obtained.

[0138] S208: Acquire a verification data set, and use each verification instance in the verification data set to verify the initial review recommendation model, and determine the Hamming distance of the initial review recommendation model. If the initial review recommendation model passes the verification, execute step S209.

[0139] In the specific process of executing step S208, in order to ensure the accuracy of the comment recommendation model, after obtaining the initial comment recommendation model, a verification data set can be further obtained, and the initial comment recommendation model can be verified using each verification instance in the verification data set, and the Hamming distance of the initial comment recommendation model can be determined; only if the initial comment recommendation model is verified, the initial comment recommendation model is determined as the comment recommendation model, and the Hamming distance of the initial comment recommendation model is determined as the target Hamming distance; if the initial comment recommendation model fails to be verified, return to execute step S201 to re-perform the corresponding training process.

[0140] Optionally, a verification data set is obtained, and each verification instance in the verification data set is used to verify the initial comment recommendation model, and the process of determining the Hamming distance of the initial comment recommendation model can be specifically as follows: a verification data set is obtained, the verification data includes multiple verification instance groups, the verification instance group includes two verification instances, and the verification instance includes a verification image and a verification text; for each verification instance group, the verification instance group is input into the initial comment recommendation model; the initial comment recommendation model uses the hash code of the image of each verification instance in its own verification instance group, and extracts the hash code of the text of each verification instance in the verification instance group; the corresponding Hamming distance calculation is performed according to the hash code of the image and the hash code of the text of the two verification instances in the verification instance group to obtain multiple Hamming distances; if there is a Hamming distance among the various Hamming distances that is not less than the Hamming distance of the verification instance group, it is determined that the verification of the initial comment recommendation model has passed, wherein the multiple Hamming distances include the image-image Hamming distance, the text-text Hamming distance, and the image-text Hamming distance.

[0141] In the actual application process, the initial comment recommendation model calculates the hash code of the image of each verification instance in the verification instance group through the hash module in its own image encoder, and calculates the hash code of the text of each verification instance in the verification instance group through the hash module in its own text encoder; the hash codes of the images of the two verification instances in the verification instance group are subjected to the corresponding Hamming distance calculation to obtain the image-image Hamming distance; the hash codes of the texts of the two verification instances in the verification instance group are subjected to the corresponding Hamming distance calculation to obtain the text-text Hamming distance; the hash codes of the images of the two verification instances in the verification instance group are subjected to the corresponding Hamming distance calculation to obtain the text-text Hamming distance. The corresponding Hamming distance calculation is performed on the hash code of the text to obtain the image-text Hamming distance; if among the obtained image-image Hamming distance, text-text Hamming distance, and image-text Hamming distance, there is an image-image Hamming distance that is not less than the Hamming distance of the verification instance group, and / or a text-text Hamming distance, and / or an image-text Hamming distance, it can be determined that the verification of the initial comment recommendation model has passed. At this time, the Hamming distance that is not less than the Hamming distance of the verification instance group can also be determined as the Hamming distance of the initial comment recommendation model; otherwise, it is considered that the verification of the initial comment recommendation model has failed.

[0142] It should be noted that if there are multiple Hamming distances that are not less than the Hamming distance of the verification instance group, the largest Hamming distance among the multiple Hamming distances can be determined as the Hamming distance of the initial comment recommendation model. For example, if the picture-picture Hamming distance and the text-text Hamming distance are not less than the Hamming distance of the verification instance group, and the picture-picture Hamming distance is less than the picture-text Hamming distance, the picture-text Hamming distance can be determined as the Hamming distance of the initial comment recommendation model, that is, the picture-text Hamming distance is determined as the target Hamming distance of the comment recommendation model, and the picture is determined as the first mode, and the text is determined as the second mode. This is only a preferred method provided in the embodiment of the present application, and the embodiment of the present application is not limited here.

[0143] S209: Determine the initial review recommendation model as the review recommendation model, and determine the Hamming distance of the initial review recommendation model as the target Hamming distance.

[0144] In an embodiment of the present application, a massive amount of multimodal training data sets that have not been manually annotated are obtained, and various noises generated by these data clusters are reduced through unsupervised data balancing, and hard-pseudo labels with discriminative capabilities are generated; first, the semantic information of different modalities of the images and texts of the instances in the training data set and the hard-pseudo labels are combined to construct a corresponding similarity matrix; secondly, the similarity matrix is ​​combined to construct inter-modal and modality-part semantic reconstruction losses to ensure that the hash codes generated by instances with similar topics are more similar; finally, the hard classification loss function and the soft classification loss function are constructed using the hard-pseudo labels and the soft-pseudo labels, so as to utilize the constructed inter-modal and modality-part semantic reconstruction, the hard classification loss function and the soft classification loss function to supervise the training of the review recommendation model to be trained, thereby improving the accuracy of model retrieval.

[0145] based on Figure 2 The review recommendation model obtained by training, accordingly, the embodiment of the present application provides a flowchart of a review recommendation method based on unsupervised cross-modality, such as Figure 3 As shown, the method specifically comprises the following steps:

[0146] S301: Obtain the published information of the target user on the social platform, and obtain multiple multimodal comments corresponding to the published information; wherein the published information includes published pictures and published texts; and each multimodal comment includes a comment picture and a comment text.

[0147] During the specific execution of step S301, the social platform of the target user can be detected in real time. When it is detected that the target user has posted or commented on the social platform, the target user's posted information on the social platform can be obtained, as well as multiple multimodal comments corresponding to the posted information; wherein the posted information includes posted pictures and posted text; each multimodal comment includes a comment picture and a comment text.

[0148] It should be noted that the social platform may be an alumni circle, Weibo, etc., which is not limited in the embodiments of the present application.

[0149] It should also be noted that multimodal comments may include comments by other users on information posted by the target user on the social platform, and comments on the same information posted by the target user.

[0150] S302: For each multimodal comment, the publishing information and the multimodal comment are input into the comment recommendation model, so that the comment recommendation model calculates the Hamming distance between the hash code of the publishing information and the hash code of the multimodal comment. If the Hamming distance is not less than the target Hamming distance, the multimodal comment is determined as a recommended comment, and the recommended comment is output.

[0151] In the specific process of executing step S302, the published information and multimodal comments can be input into the trained comment recommendation, and the comment recommendation model determines the first mode and the second mode indicated in the target Hamming distance; if the first mode is a picture and the second mode is text, the hash code of the published picture of the published information can be calculated by the hash module in its own image encoder, and the hash code of the comment text of the multimodal comment can be calculated by the hash module in its own text encoder, or the hash code of the comment picture of the multimodal comment can be calculated by the hash module in its own image encoder, and the hash code of the published text of the published information can be calculated by the hash module in its own text encoder; and according to the calculated Hamming distance of the hash code of the comment / published picture and the hash code of the comment / published text; it is determined whether the calculated Hamming distance is greater than the target Hamming distance, if it is greater, the multimodal comment is determined as a recommended comment, and the recommended comment is output; otherwise, the next multimodal comment and published information are re-input into the comment recommendation model, and the calculation is repeated in this way until the last multimodal comment is calculated.

[0152] It should be noted that the above first mode and second mode are merely a preferred method of the embodiment of the present application. The specific content of the first mode and the second mode can be determined according to the actual application situation and is not limited to this embodiment of the present application.

[0153] Further, in an embodiment of the present application, when the target user's social platform is an alumni circle, the target user may be a student, and the teacher has the authority and responsibility to supervise and manage the alumni circle. If there are inappropriate multimodal comments about the published information, they can be hidden or deleted. In addition, teachers can regularly organize various activities to activate the alumni circle, such as parent-child activities and excellent article selection competitions. Parents, as guardians of students, also have the responsibility to maintain a friendly atmosphere in the alumni group. If there are inappropriate multimodal comments, they can also be hidden or deleted.

[0154] The present application provides a method for recommending comments based on unsupervised cross-modality. First, a massive amount of multimodal training data sets that have not been manually annotated are obtained, and various noises generated by clustering of these data are reduced through unsupervised data balancing, and hard-pseudo labels with discriminative capabilities are generated; first, the semantic information of different modalities of the images and texts of the instances in the training data set and the hard-pseudo labels are combined to construct a corresponding similarity matrix; secondly, the inter-modal and modality semantic reconstruction losses are constructed by combining the similarity matrices to ensure that the hash codes generated by instances with similar topics are more similar; finally, the hard classification loss function and the soft classification loss function are constructed using the hard-pseudo labels and the soft-pseudo labels, so as to utilize the constructed inter-modal and modality semantic reconstruction, the hard classification loss function and the soft classification loss function The classification loss function supervises the training of the comment recommendation model to be trained to improve the accuracy of model retrieval; after the corresponding comment recommendation model is trained, the target user's published information on the social platform and the corresponding multimodal comments can be input into the comment recommendation model, so that the comment recommendation model can calculate the Hamming distance between the hash code of the published information and the hash code of the multimodal comment, and accurately judge whether the multimodal comment is a recommended comment that needs to be recommended to the target user based on the calculated Hamming distance, thereby solving the problem in the prior art that it is impossible to recommend corresponding comments based on the content or comments posted by the user, and it is easy for the recommended comments to have a low correlation or even no correlation with the content or comments posted by the current user, thereby resulting in a poor user experience.

[0155] Based on the unsupervised cross-modal review recommendation method provided in the above embodiment of the present application, correspondingly, the embodiment of the present application also provides an unsupervised cross-modal review recommendation system, such as Figure 4 As shown in FIG, the unsupervised cross-modal review recommendation system includes:

[0156] The first acquisition unit 41 is used to acquire the published information of the target user on the social platform, and acquire multiple multimodal comments corresponding to the published information; wherein the published information includes a published image and a published text; and each multimodal comment includes a comment image and a comment text;

[0157] A comment recommendation unit 42 is used to input the release information and the multimodal comment into the comment recommendation model for each multimodal comment, so that the comment recommendation model calculates the Hamming distance between the hash code of the release information and the hash code of the multimodal comment, and if the Hamming distance is not less than the target Hamming distance, the multimodal comment is determined as a recommended comment, and the recommended comment is output;

[0158] The training process of the review recommendation model includes:

[0159] A second acquisition unit is used to acquire a training data set and filter out multiple target instances from multiple instances in the training data set; wherein the instances include images and texts;

[0160] A combining unit, used for combining each target instance in pairs to obtain multiple target instance groups;

[0161] A similarity matrix construction unit is used to construct an image similarity matrix and a text cosine similarity matrix for each target instance group, and reconstruct the image similarity matrix and the text cosine similarity matrix of each target instance group into a similarity matrix;

[0162] A pairwise loss function construction unit, used for calculating the cosine distance between two target instances in each target instance group, and constructing a corresponding pairwise loss function according to the similarity matrix and the cosine distance of each target instance group;

[0163] A hard classification loss function construction unit, used to construct a corresponding hard classification loss function according to the hard pseudo labels of each target instance;

[0164] A soft classification loss function construction unit, used to construct a corresponding soft classification loss function according to the soft pseudo labels of each target instance;

[0165] A parameter adjustment unit, used for adjusting the parameters of the review recommendation model to be trained by using a pairwise loss function, a hard classification loss function, and a soft classification loss function to obtain an initial review recommendation model;

[0166] A verification unit, used to obtain a verification data set, and verify the initial review recommendation model using each verification instance in the verification data set, and determine the Hamming distance of the initial review recommendation model;

[0167] The determination unit is used to determine the initial review recommendation model as the review recommendation model if the initial review recommendation model passes the verification, and to determine the Hamming distance of the initial review recommendation model as the target Hamming distance.

[0168] The specific principles and execution processes of each unit in the unsupervised cross-modal review recommendation system disclosed in the above-mentioned embodiment of the present application are the same as the unsupervised cross-modal review recommendation method disclosed in the above-mentioned embodiment of the present application. Please refer to the corresponding parts of the unsupervised cross-modal review recommendation method disclosed in the above-mentioned embodiment of the present application, and will not be repeated here.

[0169] The present application provides a review recommendation system based on unsupervised cross-modality. First, a massive amount of multimodal training data sets that have not been manually annotated are obtained, and various noises generated by clustering of these data are reduced through unsupervised data balancing, and hard-pseudo labels with discriminative capabilities are generated; first, the semantic information of different modalities of the images and texts of the instances in the training data set and the hard-pseudo labels are combined to construct a corresponding similarity matrix; secondly, the inter-modal and modality semantic reconstruction losses are constructed by combining the similarity matrices to ensure that the hash codes generated by instances with similar topics are more similar; finally, the hard classification loss function and the soft classification loss function are constructed using the hard-pseudo labels and the soft-pseudo labels, so as to utilize the constructed inter-modal and modality semantic reconstruction, the hard classification loss function and the soft classification loss function The classification loss function supervises the training of the comment recommendation model to be trained to improve the accuracy of model retrieval; after the corresponding comment recommendation model is trained, the target user's published information on the social platform and the corresponding multimodal comments can be input into the comment recommendation model, so that the comment recommendation model can calculate the Hamming distance between the hash code of the published information and the hash code of the multimodal comment, and accurately judge whether the multimodal comment is a recommended comment that needs to be recommended to the target user based on the calculated Hamming distance, thereby solving the problem in the prior art that it is impossible to recommend corresponding comments based on the content or comments posted by the user, and it is easy for the recommended comments to have a low correlation or even no correlation with the content or comments posted by the current user, thereby resulting in a poor user experience.

[0170] Optionally, the second acquisition unit is specifically configured to:

[0171] A training data set is obtained, wherein the training data set includes multiple instances, each instance includes a picture and text; deep features of each picture are extracted, and each picture is clustered according to the deep features of each picture to obtain multiple clusters; for each cluster, multiple target pictures are screened out from each picture of the cluster according to the Euclidean distance from the cluster center, and the instance to which each target picture belongs is determined as the target instance.

[0172] Optionally, a similarity matrix construction unit is used to:

[0173] According to the hard pseudo labels of the target images in the two target instances in the target instance group, the image similarity matrix of the target instance group is constructed; according to the bag-of-words vectors of the target text in each target instance in the target instance group, the text cosine similarity matrix of the target instance group is constructed; according to the image similarity matrix and the text cosine similarity matrix of each target instance group, the corresponding similarity matrix is ​​constructed.

[0174] Optional, pairwise loss function building block, specifically for:

[0175] The hash codes of two target instances in the target instance group are calculated, and based on the inner product of the hash codes of the two target instances and the hash code length, the Hamming distance of the two target instances in the target instance group is calculated and the cosine distance is derived; based on the similarity matrix and the cosine distance of the two target instances in each target instance group, the corresponding pairwise loss function is constructed.

[0176] Optional, hard classification loss function building block, specifically for:

[0177] Obtain a hard false label for each target instance, wherein the hard false label of the target instance includes a hard false label of a target image in the target instance and a hard false label of a target text in the target instance; construct a corresponding image hard classification loss function according to the target image of each target instance and its hard false label; construct a corresponding text hard classification loss function according to the target image of each target instance and its hard false label; construct a corresponding hard classification loss function according to the image hard classification loss function and the text hard classification loss function.

[0178] Optional, soft classification loss function building block, specifically used for:

[0179] Obtain a soft pseudo label for each target instance, wherein the soft pseudo label of the target instance includes a soft pseudo label of a target image in the target instance and a soft pseudo label of a target text in the target instance; construct a corresponding image soft classification loss function according to the target image of each target instance and its soft pseudo label; construct a corresponding text hard classification loss function according to the target image of each target instance and its soft pseudo label; construct a corresponding soft classification loss function according to the image hard classification loss function and the text soft classification loss function.

[0180] Optional, verification unit, specifically used for:

[0181] A verification data set is obtained, where the verification data includes multiple verification instance groups, each of which includes two verification instances, and each of which includes a verification image and a verification text; for each verification instance group, the verification instance group is input into an initial comment recommendation model; the initial comment recommendation model uses the hash code of the image of each verification instance in its own verification instance group, and extracts the hash code of the text of each verification instance in the verification instance group; a corresponding Hamming distance calculation is performed based on the hash code of the image and the hash code of the text of the two verification instances in the verification instance group to obtain multiple Hamming distances; if there is a Hamming distance of the verification instance group that is not less than the Hamming distance of the verification instance group among the various Hamming distances, it is determined that the verification of the initial comment recommendation model has passed, wherein the multiple Hamming distances include the image-image Hamming distance, the text-text Hamming distance, and the image-text Hamming distance.

[0182] The present application also provides a storage medium, in which program instructions are stored. When the program instructions are loaded and executed by a processor, any one of the above-mentioned unsupervised cross-modal review recommendation method embodiments is implemented.

[0183] The present application also provides an electronic device, such as Figure 5 As shown, the device includes a processor 501 and a memory 502; the memory stores program instructions; the processor calls the program instructions in the memory to execute any of the above-mentioned unsupervised cross-modal review recommendation method embodiments.

[0184] The processor in this article can be a CPU of a terminal, or an MCU integrated in the terminal, or a combination of a CPU and an MCU. Moreover, the processor includes a kernel, which calls a corresponding program from a memory, and one or more kernels can be set.

[0185] The memory may include non-permanent memory in a computer-readable medium, random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM, and the memory includes at least one memory chip.

[0186] Each embodiment in this specification is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the system or system embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment. The system and system embodiments described above are merely schematic, in which the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules can be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0187] Professionals may further appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in the above description according to function. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians may use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0188] The above description of the disclosed embodiments enables those skilled in the art to implement or use the present application. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present application. Therefore, the present application will not be limited to the embodiments shown herein, but will conform to the widest scope consistent with the principles and novel features disclosed herein.

[0189] The above are only preferred implementations of the present application. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present application. These improvements and modifications should also be regarded as the scope of protection of the present application.

Claims

1. A review recommendation method based on unsupervised cross-modality, characterized in that: The method comprises: Obtaining the target user's published information on the social platform, and obtaining a plurality of multimodal comments corresponding to the published information; wherein the published information includes a published image and a published text; and each of the multimodal comments includes a comment image and a comment text; For each of the multimodal reviews, input the release information and the multimodal reviews into a review recommendation model, so that the review recommendation model calculates the Hamming distance between the hash code of the release information and the hash code of the multimodal reviews, and if the Hamming distance is not less than a target Hamming distance, determines the multimodal review as a recommended review, and outputs the recommended review; The training process of the review recommendation model includes: Acquire a training data set, and filter out multiple target instances from multiple instances in the training data set; wherein the instances include images and texts; Combining each of the target instances in pairs to obtain multiple target instance groups; Constructing an image similarity matrix and a text cosine similarity matrix for each target instance group, and reconstructing the image similarity matrix and the text cosine similarity matrix for each target instance group into a similarity matrix; Calculating the cosine distance between two target instances in each of the target instance groups, and constructing a corresponding pairwise loss function according to the similarity matrix and the cosine distance of each of the target instance groups; Constructing a corresponding hard classification loss function according to the hard pseudo labels of each of the target instances; Constructing a corresponding soft classification loss function according to the soft pseudo labels of each of the target instances; The pairwise loss function, the hard classification loss function and the soft classification loss function are used to adjust the parameters of the review recommendation model to be trained to obtain an initial review recommendation model; Acquire a validation data set, and use each validation instance in the validation data set to validate the initial review recommendation model, and determine the Hamming distance of the initial review recommendation model; If the initial review recommendation model passes the verification, the initial review recommendation model is determined as the review recommendation model, and the Hamming distance of the initial review recommendation model is determined as the target Hamming distance.

2. The method according to claim 1, characterized in that The step of obtaining a training data set and selecting a plurality of target instances from a plurality of instances in the training data set includes: Acquire a training data set, wherein the training data set includes multiple instances, each of which includes a picture and text; Extracting deep features of each of the images, and clustering the images according to the deep features of the images to obtain multiple clusters; For each of the clusters, multiple target images are screened out from the images of the cluster according to the Euclidean distance from the cluster center, and the instance to which each of the target images belongs is determined as the target instance.

3. The method according to claim 1, characterized in that The constructing of the image similarity matrix and the text cosine similarity matrix of each target instance group, and reconstructing the image similarity matrix and the text cosine similarity matrix of each target instance group into a similarity matrix, comprises: constructing an image similarity matrix of the target instance group according to the hard pseudo labels of the target images in the two target instances in the target instance group; Constructing a text cosine similarity matrix of the target instance group according to the bag-of-words vector of the target text in each target instance in the target instance group; According to the image similarity matrix and the text cosine similarity matrix of each target instance group, a corresponding similarity matrix is ​​constructed.

4. The method according to claim 1, characterized in that The calculating the cosine distance between two target instances in each target instance group and constructing a corresponding pairwise loss function according to the similarity matrix and the cosine distance of each target instance group includes: Calculating hash codes of two target instances in the target instance group, and calculating a Hamming distance between the two target instances in the target instance group and deriving a cosine distance according to an inner product of the hash codes of the two target instances and a hash code length; A corresponding pairwise loss function is constructed according to the similarity matrix and the cosine distance between two target instances in each of the target instance groups.

5. The method according to claim 1, characterized in that The constructing a corresponding hard classification loss function according to the hard pseudo labels of each target instance includes: Acquire a hard false label of each of the target instances, wherein the hard false label of the target instance includes a hard false label of a target image in the target instance and a hard false label of a target text in the target instance; Constructing a corresponding image hard classification loss function according to the target image of each target instance and its hard pseudo label; Constructing a corresponding text hard classification loss function according to the target image of each target instance and its hard pseudo label; A corresponding hard classification loss function is constructed according to the image hard classification loss function and the text hard classification loss function.

6. The method according to claim 1, characterized in that The step of constructing a corresponding soft classification loss function according to the soft pseudo labels of each target instance includes: Acquire a soft false label of each of the target instances, wherein the soft false label of the target instance includes a soft false label of a target image in the target instance and a soft false label of a target text in the target instance; Constructing a corresponding image soft classification loss function according to the target image and its soft pseudo label of each target instance; Constructing a corresponding text hard classification loss function according to the target image and soft pseudo label of each target instance; A corresponding soft classification loss function is constructed according to the image hard classification loss function and the text soft classification loss function.

7. The method according to claim 1, characterized in that The obtaining of the verification data set, and verifying the initial review recommendation model using each verification instance in the verification data set, and determining the Hamming distance of the initial review recommendation model, includes: Acquire a verification data set, wherein the verification data includes a plurality of verification instance groups, the verification instance groups include two verification instances, and the verification instances include a verification picture and a verification text; For each of the verification instance groups, inputting the verification instance group into an initial review recommendation model; The initial comment recommendation model utilizes the hash code of the image of each verification instance in the verification instance group, and extracts the hash code of the text of each verification instance in the verification instance group; performs corresponding Hamming distance calculations based on the hash codes of the images and the hash codes of the texts of two verification instances in the verification instance group to obtain multiple Hamming distances; if there is a Hamming distance of not less than the verification instance group among the Hamming distances, it is determined that the verification of the initial comment recommendation model has passed, wherein the multiple Hamming distances include image-image Hamming distance, text-text Hamming distance and image-text Hamming distance.

8. A review recommendation system based on unsupervised cross-modal, characterized in that: The system comprises: A first acquisition unit is used to acquire the published information of the target user on the social platform, and acquire multiple multimodal comments corresponding to the published information; wherein the published information includes a published picture and a published text; each of the multimodal comments includes a comment picture and a comment text; A comment recommendation unit, for each of the multimodal comments, inputting the release information and the multimodal comment into a comment recommendation model, causing the comment recommendation model to calculate a Hamming distance between a hash code of the release information and a hash code of the multimodal comment, and if the Hamming distance is not less than a target Hamming distance, determining the multimodal comment as a recommended comment, and outputting the recommended comment; The training process of the review recommendation model includes: A second acquisition unit is used to acquire a training data set and filter out a plurality of target instances from a plurality of instances in the training data set; wherein the instances include images and texts; A combining unit, used for combining each of the target instances in pairs to obtain a plurality of target instance groups; A similarity matrix construction unit, used to construct an image similarity matrix and a text cosine similarity matrix for each target instance group, and reconstruct the image similarity matrix and the text cosine similarity matrix of each target instance group into a similarity matrix; A pairwise loss function construction unit, used for calculating the cosine distance between two target instances in each of the target instance groups, and constructing a corresponding pairwise loss function according to the similarity matrix and the cosine distance of each of the target instance groups; A hard classification loss function construction unit, used to construct a corresponding hard classification loss function according to the hard pseudo labels of each target instance; A soft classification loss function construction unit, used to construct a corresponding soft classification loss function according to the soft pseudo labels of each target instance; A parameter adjustment unit, configured to adjust parameters of the review recommendation model to be trained by using the pairwise loss function, the hard classification loss function, and the soft classification loss function to obtain an initial review recommendation model; A verification unit, used to obtain a verification data set, and verify the initial review recommendation model using each verification instance in the verification data set, and determine the Hamming distance of the initial review recommendation model; A determination unit is configured to determine the initial review recommendation model as the review recommendation model if the initial review recommendation model passes verification, and to determine the Hamming distance of the initial review recommendation model as a target Hamming distance.

9. An electronic device, characterized in that: include: A processor and a memory, wherein the processor and the memory are connected via a communication bus; wherein the processor is used to call and execute a program stored in the memory; The memory is used to store a program, and the program is used to implement the unsupervised cross-modal review recommendation method as described in any one of claims 1-7.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-executable instructions, and the computer-executable instructions are used to execute the unsupervised cross-modal review recommendation method according to any one of claims 1 to 7.

Images