Data digital analysis matching method based on artificial intelligence

Abstract

The invention relates to the technical field of data processing, and discloses an artificial intelligence-based data digital analysis matching method, which comprises the following steps of: obtaining a search word of cross-modal user behavior data; performing feature extraction on the user behavior data in the original user behavior database to obtain a first feature vector corresponding to each piece of user behavior data; the types of the user behavior data comprise text data, text-free picture data and text-containing picture data; performing similarity calculation by utilizing an intelligent matching model, and determining cosine similarity between each first feature vector and the search word; and performing analysis and matching according to the cosine similarity between each first feature vector and the search word, and determining target user behavior data. According to the method, the first feature vector is obtained by performing feature extraction on the user behavior data, and then image and text feature matching is performed on the first feature vector and the feature vector converted from the search word, so that cross-modal intelligent data matching between typical modes of images and texts is effectively realized.

Description

Technical Field

[0001] The present invention relates to the field of data processing technology, and in particular to a data digital analysis and matching method based on artificial intelligence. Background Art

[0002] With the continuous development of information technology, the informatization and intelligentization of various industries are constantly advancing. The dissemination of data is no longer limited to text as a carrier. More carrier forms such as images, sounds, and videos continue to appear. Massive multimodal data are stored in different business systems in various industries. Traditional single-modal methods can only handle the matching and annotation of single-modal data, which is difficult to meet the needs of multimodal data matching. Therefore, related industries have successively carried out cross-modal matching and joint analysis technology research on multimodal data. With the development of informatization and digitalization, the data types generated by user behavior systems are becoming increasingly rich and complex, including text, images, videos, sensor data and other modalities, which brings challenges to data processing and analysis. Traditional user behavior data management and analysis models can no longer meet the needs of modern user behavior data services.

[0003] Cross-modal matching provides an effective method to solve the differences in cross-modal data and realize joint matching and query of data in different forms, so as to obtain more complete and accurate information about business objects. However, there are still two major difficulties. One is the difference in multimodal expression. There are large differences between the data expressions of different modalities, and it is difficult to measure the correlation between different modal data; the other is the semantic gap. The difficulty of feature analysis between different modalities lies in the gap between the underlying expression and the high-level semantics, and it is difficult to mine the high-level semantics of data between different modalities. Summary of the invention

[0004] The present invention provides an artificial intelligence-based data digital analysis and matching method for effectively realizing cross-modal intelligent data matching between typical modalities of images and texts.

[0005] In a first aspect, the present invention provides a data digital analysis and matching method based on artificial intelligence, comprising:

[0006] Search terms to obtain cross-modal user behavior data;

[0007] Performing feature extraction on at least one user behavior data in the original user behavior database to obtain a first feature vector corresponding to each of the user behavior data; wherein the type of the user behavior data includes any one of the following: text data, image data without text, and image data containing text;

[0008] Using an intelligent matching model to perform similarity calculation, determining the cosine similarity between each of the first feature vectors and the search term;

[0009] Analyze and match the cosine similarity between each of the first feature vectors and the search term to determine the target user behavior data.

[0010] According to the artificial intelligence-based data digital analysis and matching method provided by an embodiment of the present invention, the similarity calculation using the intelligent matching model to determine the cosine similarity between each of the first feature vectors and the search term includes:

[0011] Extract key information from the search term using an intelligent matching model to obtain a second feature vector;

[0012] A similarity calculation is performed using an intelligent matching model to determine the cosine similarity between each of the first feature vectors and the second feature vectors.

[0013] In a second aspect, the present invention provides a data digital analysis and matching device based on artificial intelligence, comprising:

[0014] An acquisition module, used to obtain search terms for cross-modal user behavior data;

[0015] A feature extraction module, used to extract features from at least one user behavior data in the original user behavior database to obtain a first feature vector corresponding to each of the user behavior data; wherein the type of the user behavior data includes any one of the following: text data, image data without text, and image data containing text;

[0016] A similarity calculation module, used to perform similarity calculation using an intelligent matching model to determine the cosine similarity between each of the first feature vectors and the search term;

[0017] The analysis and matching module is used to perform analysis and matching based on the cosine similarity between each of the first feature vectors and the search term to determine the target user behavior data.

[0018] In a third aspect, the present invention further provides an electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the data digitalization analysis and matching method based on artificial intelligence as described in the first aspect is implemented.

[0019] In a fourth aspect, the present invention further provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the data digitization analysis and matching method based on artificial intelligence as described in the first aspect.

[0020] In a fifth aspect, the present invention further provides a computer program product, comprising a computer program, which, when executed by a processor, implements the data digitization analysis and matching method based on artificial intelligence as described in the first aspect.

[0021] The artificial intelligence-based data digital analysis and matching method provided by the present invention, since the search term in the present invention is text and the original user behavior database contains pictures, therefore, by performing feature extraction on each user behavior data, a first feature vector corresponding to each user behavior data is obtained, and then the first feature vector is matched with the feature vector converted from the search term for image and text features, thereby effectively realizing cross-modal intelligent data matching between typical modes of pictures and texts. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] In order to more clearly illustrate the technical solutions in the present invention or related technologies, the following briefly introduces the drawings required for use in the embodiments or related technical descriptions. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0023] Figure 1 This is one of the flow charts of the data digital analysis and matching method based on artificial intelligence provided by the present invention;

[0024] Figure 2 It is a schematic diagram of the principle of the bidirectional long short-term memory network model provided by the present invention;

[0025] Figure 3 It is a schematic diagram of the principle of face recognition network image feature extraction provided by the present invention;

[0026] Figure 4 It is a schematic diagram of the principle of the distinguishable binary text detection algorithm provided by the present invention;

[0027] Figure 5 This is the second flow chart of the data digital analysis and matching method based on artificial intelligence provided by the present invention;

[0028] Figure 6 It is a schematic diagram of an information cascade graph embedding layer based on spatiotemporal features provided by the present invention;

[0029] Figure 7 It is a structural diagram of the data digital analysis and matching device based on artificial intelligence provided by the present invention;

[0030] Figure 8 It is a structural diagram of the electronic device provided by the present invention. DETAILED DESCRIPTION

[0031] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present invention.

[0032] In the description of the present invention, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Thus, the features defined as "first" and "second" may explicitly or implicitly include one or more of the features. In the description of the present invention, the meaning of "plurality" is two or more, unless otherwise clearly and specifically defined.

[0033] In the description of the present invention, the term "for example" is used to mean "used as an example, illustration or explanation". Any embodiment described as "for example" in the present invention is not necessarily to be construed as being more preferred or advantageous than other embodiments. The following description is given to enable any technician in the field to implement and use the present invention. In the following description, details are listed for the purpose of explanation. It should be understood that a person of ordinary skill in the art can recognize that the present invention can be implemented without using these specific details. In other examples, well-known structures and processes will not be elaborated in detail to avoid obscuring the description of the present invention with unnecessary details. Therefore, the present invention is not intended to be limited to the embodiments shown, but is consistent with the widest scope consistent with the principles and features disclosed in the present invention.

[0034] See also Figure 1 , Figure 1 This is one of the flow charts of the data digital analysis and matching method based on artificial intelligence provided by the present invention. The data digital analysis and matching method based on artificial intelligence includes:

[0035] Step 10: Obtain search terms for cross-modal user behavior data.

[0036] Specifically, the search term is used to match the corresponding target user behavior data in the original user behavior database, which is usually text data, and in a few cases, there may also be image data. In this embodiment, multi-modal data matching can be achieved, and the following takes the search term as text data as an example.

[0037] Step 20, extracting features from at least one user behavior data in the original user behavior database to obtain a first feature vector corresponding to each user behavior data. The type of user behavior data includes any one of the following: text data, image data without text, and image data containing text.

[0038] Specifically, the original user behavior database contains multiple user behavior data, and the data type is multiple modalities. For example, the type of user behavior data is any one of the following: text data (plain text), picture data without text (plain picture), picture data containing text (picture containing text information).

[0039] Before using the search terms for intelligent retrieval and matching, at least one user behavior data in the original user behavior database may be preprocessed, such as extracting features from the user behavior data to obtain a first feature vector corresponding to each user behavior data.

[0040] Step 30: perform similarity calculation using the intelligent matching model to determine the cosine similarity between each first feature vector and the search term.

[0041] Specifically, after obtaining the first feature vector corresponding to each user behavior data, the cosine similarity calculation may be performed between the second feature vector corresponding to the search term and the first feature vector corresponding to each user behavior data, thereby obtaining the most matching target user behavior data.

[0042] Step 40: Analyze and match the cosine similarity between each first feature vector and the search term to determine the target user behavior data.

[0043] Specifically, a larger value of the cosine similarity between the first feature vector and the second feature vector indicates a higher semantic association between the two.

[0044] After obtaining the cosine similarity between each first feature vector and the search term, the target user behavior data can be determined according to the cosine similarity between each first feature vector and the search term. For example, the user behavior data corresponding to the maximum cosine similarity is determined as the target user behavior data.

[0045] In the embodiment of the present invention, since the search terms in the present invention are texts and the original user behavior database contains pictures, the first feature vector corresponding to each user behavior data is obtained by performing feature extraction on each user behavior data, and then the first feature vector is matched with the feature vector converted from the search terms for image and text features, thereby effectively realizing cross-modal intelligent data matching between typical modalities of pictures and texts.

[0046] In one embodiment, similarity calculation is performed using an intelligent matching model to determine the cosine similarity between each first feature vector and the search term, including:

[0047] The intelligent matching model is used to extract key information from the search terms to obtain the second feature vector;

[0048] The intelligent matching model is used to perform similarity calculation to determine the cosine similarity between each first eigenvector and the second eigenvector.

[0049] Specifically, the intelligent matching model is used to extract key information from the search terms to obtain the second feature vector, wherein the key information extraction from the search terms can be performed using the text key information extraction technology based on BiLSTM-CRF, which can effectively and accurately identify and extract key information from these unstructured texts. BiLSTM (bidirectional long short-term memory network) can capture the front-to-back dependency of time series in text data, and CRF (conditional random field) can consider the context constraint relationship in sequence labeling problems. The combination of the two greatly improves the accuracy and reliability of key information. BiLSTM-CRF consists of a word embedding layer, a BiLSTM layer, and a CRF layer.

[0050] Furthermore, the intelligent matching model is used to perform similarity calculation to determine the cosine similarity between each first feature vector and the second feature vector. After extracting the second feature vector of the search term, the second feature vector and the feature vector corresponding to the user behavior data are semantically associated by calculating the cosine similarity.

[0051] The embodiment of the present invention uses an intelligent matching model and an image-text semantic association algorithm based on cosine similarity measurement to achieve intelligent matching of cross-modal user behavior data.

[0052] In one embodiment, determining the target user behavior data according to the cosine similarity between each first feature vector and the search term includes:

[0053] Calculate the cosine similarity between each first eigenvector and the search term;

[0054] The user behavior data corresponding to the maximum value among the cosine similarities is determined as the target user behavior data.

[0055] Specifically, the cosine similarity between each first feature vector and the search term is calculated.

[0056] Regardless of the type of user behavior data, images and texts are represented as their feature vectors, and then the similarity between them is measured by calculating the cosine similarity between them. Specifically, for images and texts, feature representations are first extracted based on the key information feature technology for images and texts, and then the extracted feature vectors are normalized to unit vectors (i.e., they are divided by their norms) in order to calculate the cosine similarity between them and the search terms.

[0057] The image-text semantic association based on cosine similarity measurement is trained using the idea of ​​generative adversarial networks. By calculating the cosine similarity between image features and text features, the semantic similarity between them is measured, thereby performing association analysis. The cosine similarity value range is between -1 and 1. When the cosine similarity is close to 1, it means that the directions pointed by the two vectors are very similar; when the cosine similarity is close to -1, it means that the directions pointed by the two vectors are exactly opposite; when the cosine similarity is close to 0, it means that the two vectors are orthogonal, that is, there is no association. Therefore, the calculation formula for cosine similarity is as follows:

[0058]

[0059] Among them, A and B represent two vectors to be compared, such as the image feature vector corresponding to the user behavior data and the feature vector corresponding to the search term, · represents the dot product of the vectors, and ||A|| and ||B|| represent the norms (lengths) of A and B, respectively. The image-text semantic association method based on cosine similarity measurement calculates the cosine similarity between the image feature vector and the text feature vector to determine the semantic relevance between them, which can help understand the semantic connection between the image and the text.

[0060] Furthermore, the user behavior data corresponding to the maximum value of each cosine similarity is determined as the target user behavior data, thereby achieving matching of cross-modal user behavior data.

[0061] The embodiment of the present invention uses an image-text semantic association algorithm based on cosine similarity measurement to achieve intelligent matching of cross-modal user behavior data.

[0062] In one embodiment, feature extraction is performed on at least one user behavior data in the original user behavior database to obtain a first feature vector corresponding to each user behavior data, including:

[0063] When the user behavior data is text data, a bidirectional long short-term memory network model is used to extract key information from the text data to obtain a first text feature vector;

[0064] The first text feature vector is determined as the first feature vector corresponding to the user behavior data.

[0065] Specifically, when the user behavior data is text data, a bidirectional long short-term memory network model is used to extract key information from the text data to obtain a first text feature vector.

[0066] Among them, the bidirectional long short-term memory network model is implemented based on the BiLSTM-CRF text key information extraction technology. BiLSTM (bidirectional long short-term memory network) can capture the front-to-back dependency of time series in text data, and CRF (conditional random field) can consider the context constraint relationship in sequence labeling problems. The combination of the two greatly improves the accuracy and reliability of key information. BiLSTM-CRF consists of a word embedding layer, a BiLSTM layer, and a CRF layer.

[0067] The word embedding layer converts discrete characters into distributed representations. LSTM is the core of the BiLSTM layer, which can complete the task of learning long-term dependency information, and a gating is designed to avoid long-term dependency problems, which remembers the information for transmission. The CRF layer confirms the final required prediction label sequence.

[0068] Furthermore, the first text feature vector is determined as the first feature vector corresponding to the user behavior data, thereby achieving accurate feature extraction of the text data.

[0069] In the embodiment of the present invention, when the user behavior data is text data, a bidirectional long short-term memory network model is used to extract key information from the text data to obtain a first text feature vector, and then the first text feature vector is determined as the first feature vector corresponding to the user behavior data. The extraction of key information from the text of the user behavior data based on BiLSTM-CRF is realized, which facilitates the subsequent semantic association of cross-modal data.

[0070] In one embodiment, a bidirectional long short-term memory network model includes a word embedding layer, a bidirectional long short-term memory network layer, and a conditional random field model layer; the bidirectional long short-term memory network model is used to extract key information from text data to obtain a text feature vector, including: inputting the text data into the word embedding layer to obtain a word vector matrix corresponding to the text data, wherein the word vector matrix is ​​a continuous vector representation; inputting the word vector matrix corresponding to the text data into the bidirectional long short-term memory network layer to obtain a context feature sequence of at least two time steps; using the conditional random field model layer, annotating the context feature sequence of each time step, and calculating the output score of each annotated context feature sequence relative to the label prediction sequence, the output score being used to characterize the credibility corresponding to the context feature sequence of the time step; using the conditional random field model layer, determining the text feature vector according to the output score of each annotated context feature sequence relative to the label prediction sequence.

[0071] Specifically, the bidirectional long short-term memory network model includes a word embedding layer, a bidirectional long short-term memory network BiLSTM layer, and a conditional random field CRF model layer. Figure 2 It is a schematic diagram of the principle of the bidirectional long short-term memory network model provided by the present invention.

[0072] Correspondingly, the process of extracting key information from text data using a bidirectional long short-term memory network model to obtain a text feature vector can be achieved through the following steps, including:

[0073] First, the text data is input into the word embedding layer to obtain the word vector matrix corresponding to the text data. The word vector matrix is ​​a continuous vector representation.

[0074] Here, the word embedding layer converts discrete characters into distributed representations, which means using continuous values ​​in the vector space to represent features (such as characters, words, or other discrete objects); in the word embedding layer, the input is the index of discrete characters or words (which may contain contextual information), and the output is the continuous vector representation of these indexes mapped to the dense vector space. First, preprocess the annotated corpus files, usually text files such as CSV, JSON, etc. The annotated information includes part of speech, syntactic structure, entity recognition, etc. The main purpose of preprocessing is to convert the raw data into a format suitable for model training to ensure the quality and consistency of the data. Next, initialize a random matrix for input into the model. Random matrix initialization helps prevent model overfitting. The dimension of the random matrix is ​​n*d, where n represents the dictionary size (that is, the number of different words in the vocabulary) and d represents the attribute vector dimension of each character (that is, the dimension of the word vector). Finally, the word embedding layer outputs the vector representation of each word in the text and forms a matrix.

[0075] Furthermore, the word vector matrix corresponding to the text data is input into the bidirectional long short-term memory network layer to obtain a context feature sequence of at least two time steps.

[0076] Among them, LSTM is the core of the BiLSTM layer, which can complete the task of learning long-term dependency information, and a gate is designed to avoid long-term dependency problems, which will remember the information for transmission. The word vector matrix generated from the word embedding layer is input into the BiLSTM bidirectional long short-term memory network, where BiLSTM consists of two LSTMs, one from the beginning to the end of the sequence (forward LSTM) and the other from the end to the beginning (reverse LSTM). This structure can consider the contextual information of the previous and following words at the same time, making the model's understanding of the sequence more comprehensive. The core calculation formula of the long short-term memory network (LSTM) is as follows:

[0077] f t =σ(W f [h t-1 ,x t ]+b f )

[0078] i t =σ(W i [h t-1 ,x t ]+bi )

[0079]

[0080] o t =σ(W o [h t-1 ,x t ]+b o )

[0081] h t =o t ×tanh(c t )

[0082] The parameters are defined as follows:

[0083] h t-1 is the hidden state (or output) of the previous moment, representing the memory and information of the model at time t-1. t is the input vector at the current time t, which can be the word embedding representation, features, etc. of the text. f , W i , W c , W o is the weight matrix used for linear transformation. f is the weight matrix of the forget gate. i is the weight matrix of the input gate. c is the weight matrix of the candidate memory unit. o is the weight matrix of the output gate. f 、b i 、b c 、b o is a bias vector, added to the calculation of each gate to help the model learn. f is the bias of the forget gate. i is the bias of the input gate. c is the bias of the candidate memory cell. b o is the bias of the output gate. σ is the sigmoid activation function, and the output value is between 0 and 1. tanh is the hyperbolic tangent function, which is used to normalize and scale the content of the candidate memory unit, and the output range is [-1,1]. t is the output of the forget gate. t is the output of the input gate. is the output of the candidate memory unit. t-1 is the state of the memory unit at the previous moment, storing the long-term memory passed from the previous time step. t is the current state of the memory unit, which is updated by combining the output of the input gate and the forget gate. t is the activation value of the output gate, which determines the output content at the current moment. tIt is the hidden state at the current moment and is also the output of LSTM, combining the memory cell state at the current moment and the decision of the output gate.

[0084] After the above calculations, the hidden state sequence {h0,h1...h n-1 BiLSTM passes information forward and backward through two LSTMs respectively, and concatenates the generated hidden state sequences, thereby obtaining contextual information at the same time, effectively improving the performance of the model, and finally outputting the contextual feature representation of each time step, that is, the contextual feature sequence of at least two time steps.

[0085] Furthermore, the conditional random field model layer is used to annotate the context feature sequence of each time step, and the output score of each annotated context feature sequence relative to the label prediction sequence is calculated. The output score is used to characterize the credibility corresponding to the context feature sequence of the time step; the text feature vector is determined according to the output score of each annotated context feature sequence relative to the label prediction sequence.

[0086] The output of the BiLSTM model can be connected to the CRF model. The CRF model decodes the label sequence of the BiLSTM model at the sentence level (rather than decoding it individually) and annotates the sequence. The annotated information includes named entities, parts of speech, semantics, keywords, etc., covering a variety of natural language processing tasks. In the loss function of the CRF module, the label prediction sequence is the sequence with the highest output score, that is, the sequence with the highest credibility or probability of each label sequence calculated by the CRF model. Given a sequence X, let the sequence annotation result be Y, and its definition score is:

[0087]

[0088] Among them, P is the initial score matrix and A is the transformed score matrix. Indicates label Y i-1 Continuation label Y i The probability of Representation word w i Mapped to label Y i Calculate the score of the output label sequence Y corresponding to the input sequence X, and regard the sequence with the highest score as the predicted label sequence required by the model.

[0089] After being processed by the BiLSTM-CRF model, the key information in the text is finally obtained, including the extracted named entities, keywords, etc., that is, the text feature vector.

[0090] The embodiment of the present invention inputs text data into a word embedding layer to obtain a word vector matrix corresponding to the text data, where the word vector matrix is ​​a continuous vector representation. Then, the word vector matrix corresponding to the text data is input into a bidirectional long short-term memory network layer to obtain a context feature sequence of at least two time steps. Then, the conditional random field model layer is used to label the context feature sequence of each time step, and the output score of each labeled context feature sequence relative to the label prediction sequence is calculated, and the output score is used to characterize the credibility corresponding to the context feature sequence of the time step. Afterwards, the conditional random field model layer is used to determine the text feature vector according to the output score of each labeled context feature sequence relative to the label prediction sequence. Therefore, when the user behavior data is text data, the bidirectional long short-term memory network model is used to extract key information from the text data to obtain a first text feature vector, thereby realizing the extraction of text key information of the user behavior data based on BiLSTM-CRF, which is convenient for subsequent semantic association of cross-modal data.

[0091] In one embodiment, feature extraction is performed on at least one user behavior data in the original user behavior database to obtain a first feature vector corresponding to each user behavior data, including:

[0092] In the case where the user behavior data is picture data without text, the picture data with text is input into the face recognition network to extract key information of the picture data with text;

[0093] According to the key information of the text image data, the yolov5 model is used to frame the face in the text image data to obtain the face image;

[0094] Use deep convolutional neural network to extract features from face images and obtain feature vectors corresponding to face images;

[0095] The feature vector corresponding to the face image is determined as the first feature vector corresponding to the user behavior data.

[0096] Specifically, when the user behavior data is picture data without text, the picture data with text is input into the face recognition network to extract key information of the picture data with text. Figure 3This is a schematic diagram of the principle of face recognition network image feature extraction provided by the present invention. Further, according to the key information of the text image data, the yolov5 model is used to frame the face in the text image data to obtain the face image; further, the deep convolutional neural network is used to extract the features of the face image, such as being able to extract the key features in the face image, including the contour, eyes, nose and other features of the face, to obtain the feature vector corresponding to the face image; further, the feature vector corresponding to the face image is determined as the first feature vector corresponding to the user behavior data.

[0097] Here, the face recognition network FaceNet maps face images to Euclidean space, directly reflecting the similarity of face images in spatial distance. The principle is that the distance between face images sampled from the same individual in Euclidean space is smaller, while the distance between face images from different individuals is larger. Through Facenet, the key information in the image is extracted, and the celeba data set is trained using the yolov5 network model to frame the face in the image. In this experiment, the framed face is taken out and put into Facenet to generate facial features. The facial features are compared with those in the existing database through SVM to calculate the similarity.

[0098] The structure of FaceNet includes a deep convolutional neural network, which is used to convert face images into fixed-length embeddings. During the training process, batch processing and a deep convolutional neural network structure Deep Architecture after removing softmax are used, and L2 regularization is used to ensure the stability of the embedding vector in Euclidean space. Finally, the triplet loss function is used to optimize the embedding, so that the embedding distance of the same face is closer and the embedding distance of different faces is farther.

[0099] (1) Use a deep convolutional neural network (DNN) to extract facial features and map the facial image to an embedding vector of fixed length. The deep architecture adopted can capture the complex features in the image and improve recognition accuracy.

[0100] (2) FaceNet introduced a triplet loss function to enhance the model’s discriminative ability. This loss function uses a triplet (Anchor, Positive, Negative) to optimize the network, making the face embedding vectors (Anchor and Positive) of the same person closer in Euclidean space, while the face embedding vectors (Anchor and Negative) of different people farther apart.

[0101] The learning process of the network is to continuously adjust the Euclidean distance between Anchor, Positive, and Negative. During learning, the distance between Anchor and Negative is increased while the distance between Anchor and Positive is reduced, and finally the distance between Anchor and Negative is greater than the distance between Anchor and Positive. Let x be the input image, f(x)∈R d is a eigenvector, then its ternary function is expressed as:

[0102] Then the expression of its ternary function is:

[0103]

[0104] in, Represents the anchor image. express The characteristic vector of Indicates another image of the same person as the anchor image. express The characteristic vector of Indicates another image of a different person than the anchor image. express The eigenvector of represents all possible triple pairs in the training set, α is a constant, so the final loss function is:

[0105]

[0106] in, represents the intra-class distance, Represents the distance between classes.

[0107] The working principle of FaceNet is to recognize the face information in the image by training the deep learning model. Its workflow is as follows:

[0108] (1) Input a face picture: Input a picture containing a face into the FaceNet network and use the yolov5 model to frame the face.

[0109] (2) Feature extraction through deep convolutional networks: FaceNet uses deep convolutional neural networks (CNNs) to extract features from input face images. After training, deep convolutional networks can extract key features from face images, including facial contours, eyes, noses, and other features.

[0110] (3) L2 normalization: In the process of feature extraction, in order to enhance the stability and comparability of features, FaceNet uses the L2 normalization method. This method can normalize the length of feature vectors, reduce the differences between different features, and improve the comparability of feature vectors.

[0111] (4) Obtain a feature vector with a length of 128: After processing and normalization, the features extracted by FaceNet through the deep convolutional network are processed and normalized to obtain a feature vector with a length of 128. This feature vector can uniquely represent the face in the input image, has good feature distinction ability, and can be used for tasks such as face recognition and face verification.

[0112] The embodiment of the present invention realizes the recognition of facial information in an image by training a deep learning model, and finally converts it into a feature vector to facilitate the subsequent cross-modal intelligent matching of images and texts.

[0113] In one embodiment, feature extraction is performed on at least one user behavior data in the original user behavior database to obtain a first feature vector corresponding to each user behavior data, including:

[0114] In the case where the user behavior data is picture data containing text, the picture data containing text is input into a text detection model to identify a text region in the picture data containing text;

[0115] According to the text region in the image data containing the text, the text direction of the text instance in the text region is detected by using a text direction classifier;

[0116] Based on the text direction, the text instance is rotated to a parallel direction to obtain a rotated text instance;

[0117] Inputting the rotated text instance into a text recognition model to obtain a second text feature vector;

[0118] The second text feature vector is determined as the first feature vector corresponding to the user behavior data.

[0119] Specifically, the user behavior data is image data, and it is further determined whether there is text information in the image data. In the case where the user behavior data is image data containing text, this embodiment recognizes the text information to obtain a text feature vector, including steps such as image preprocessing, character segmentation, feature extraction and classification. This technology includes three modules: text detection model, text direction classifier and text recognition model. The implementation process example is as follows:

[0120] Figure 4 Schematic diagram of the principle of the distinguishable binary text detection algorithm provided by the present invention, such as Figure 4As shown, the image data containing text is input into the text detection model to identify the text area in the image data containing text. The main task of the text detection model is to detect the text area from the image, identify the text position in the image, and locate it as independent text areas. This function is implemented using a differentiable binarization (DB) model. The DB model uses a segmentation-based text detection method to transform the text detection problem into a semantic segmentation task. A differentiable binarization function is used to make the entire detection network end-to-end trainable. During the training process, the appropriate binarization threshold and sensitivity can be automatically learned to better match the real data groundtruth.

[0121] The core of the DB model is a differentiable binarization function that receives an input feature map and outputs a probability map between [0,1]. The probability map is generated by the segmentation network and then converted into a binary map. Then, through this probability map, the DB model can obtain the binarized text area mask. The binarization process is:

[0122]

[0123] Among them, t is the predefined threshold, (i, j) is the coordinate point on the graph, and P i，j is the value of the i-th row and j-th column of the input feature map, which is the feature value or score calculated by the model before. i,j Represents the binarization result of the i-th row and j-th column.

[0124] The differentiable binarization that can be optimized synchronously with the segmentation network during the training phase is defined as:

[0125]

[0126] Among them, B i,j is the probability value of the pixel in the i-th row and j-th column, indicating the possibility that the pixel is a text area. k is a constant that adjusts the "slope" or "sensitivity" of the sigmoid function. The larger its value, the steeper the function changes, and the more effective it is in distinguishing a specific range of probability values.

[0127] This formula converts the eigenvalue P i,j and threshold T i.j Combined with the smoothness of the sigmoid function, a probability value in the interval [0,1] is generated, which enables the model to output continuous judgments. i.j and k, the sensitivity and feature selection ability of the model can be fine-tuned, thus affecting the final text detection effect.

[0128] Therefore, it is concluded that when the value in the probability map is less than the threshold of the threshold map, it means that this is not a text area, so it is set to 0, otherwise it is a text area. In actual processing applications, the judgment of a single pixel alone is not enough to effectively recognize text. Therefore, a method of combining regional validity after processing is proposed, requiring a group of adjacent pixels to meet the condition of being higher than the threshold to form a coherent text area, which can be used to eliminate isolated noise points and ensure the reliability of detection results.

[0129] Furthermore, after detecting the text area, the text direction can be further classified. For example, based on the text area in the image data containing the text, a text direction classifier is used to detect the text direction of the text instance in the text area. Furthermore, based on the text direction, the text instance is rotated to a parallel direction to obtain a rotated text instance.

[0130] It should be noted that the task of the text direction classifier is to classify the direction of the text instance detected by the text, rotate the text to 0 degrees, and then send it to the subsequent text recognizer. The MobileNetV3 model is used to implement this function. MobileNetV3 introduces the Squeeze-and-Excitation module of channel attention and the NAS search method to improve network performance. The overall architecture adopts a lightweight depth-separable convolution and residual block structure.

[0131] MobileNetv3 converts the input image into a compact feature vector to represent the direction information of the text image. The direction information is output as a probability distribution vector, and each element represents the possibility that the input image is judged to be in the corresponding direction. For example: the output of [0.1, 0.7, 0.1, 0.1] may indicate that there is a 70% probability that the image is in a 90-degree direction. The features extracted by MobileNetv3 are input into a fully connected layer to achieve fast and efficient direction classification. While maintaining high classification accuracy, MobileNetv3 greatly reduces the number of model parameters and inference time, providing strong support for high-performance direction classification applications.

[0132] Further, the rotated text instance is input into the text recognition model to obtain a second text feature vector. It should be noted that the text recognition model is based on a sequence-to-sequence architecture of deep learning, and can accurately identify text content from the detected text area. The present invention adopts a convolutional recurrent neural network (CRNN) model to achieve this function. The CRNN model efficiently captures the spatial and semantic information of the text. The CNN part is responsible for extracting the visual features of the text area, and the RNN part models the contextual relationship of the text, and finally outputs the text recognition result, which has excellent performance in terms of accuracy, generalization and real-time performance, and can be widely used in various text recognition scenarios. It mainly includes three parts: a convolutional layer sequence for extracting image features, a recursive layer sequence for sequence modeling, and a transcription layer, which decodes the output of the recursive layer into a text sequence.

[0133] Among them, the convolutional layer is responsible for extracting features of the image, which are then flattened and input into the recurrent layer. In the recurrent layer, the RNN processes the input sequence and is able to capture contextual information at each time step of the sequence. This is especially important for understanding the meaning of text, because specific characters of the text often require knowledge of the previous and following context. Finally, the transcription layer (usually using CTC, or Connectionist Temporal Classification) decodes the output of the RNN to generate the final text sequence.

[0134] The first part of CRNN is a series of convolutional layers, which are used to extract visual features from the input image. Let the input image be I, and the feature map obtained after L layers of convolution operations is:

[0135] F L =conv(I;W L ; b L )

[0136] Among them, W L and b L Represent the convolution weight and bias of the Lth layer respectively. The feature F extracted by the convolution operation L It will be sent to the subsequent recursive layer for further processing.

[0137] The second part is the recursive layer, which models the feature sequence and captures the time dependency in the sequence. It outputs a hidden state vector at each time step, which represents the feature representation of the current time step and contains the current input information and the state information of the previous time step.

[0138] The third part is the transcription layer, which is responsible for mapping the output of the recurrent layer to the final sequence label. Transcription is usually done through (Connectionist Temporal Classification, CTC), which uses the principles of probability theory to solve the sequence learning problem of unaligned data, and its goal is to maximize the conditional probability P(π|x). Where π represents a path, which is mapped to the final label sequence l by removing duplicate labels and blank labels. The objective function of CTC is defined as follows, which sums the probabilities of all paths π that may be mapped to the label sequence l.

[0139]

[0140] The input sequence x is the feature sequence generated by the recursive layer, and the label sequence l is the final output label sequence that the model is expected to predict.

[0141] The objective function of CTC flexibly handles the alignment problem between the input sequence and the output label sequence by summing the probabilities of all possible paths mapped to a given label sequence. Its mechanism allows for efficient "transcription" of clear outputs from sparse and unaligned inputs in sequence learning.

[0142] Then, the second text feature vector is determined as the first feature vector corresponding to the user behavior data, so as to realize feature extraction of the image data containing text information, and facilitate subsequent intelligent retrieval and matching.

[0143] In the embodiment of the present invention, when the user behavior data is picture data containing text, the picture data containing text is input into the text detection model to identify the text area in the picture data containing text; based on the text area in the picture data containing text, the text direction of the text instance in the text area is detected by the text direction classifier; based on the text direction, the text instance is rotated to a parallel direction to obtain the rotated text instance; the rotated text instance is input into the text recognition model to obtain the second text feature vector; the second text feature vector is determined as the first feature vector corresponding to the user behavior data. Thus, the image is converted into editable text, and finally the image-text semantic association algorithm based on the cosine similarity measurement is integrated, and the cross-modal intelligent matching of user behavior data is realized.

[0144] Figure 5 This is the second flow chart of the data digital analysis and matching method based on artificial intelligence provided by the present invention, such as Figure 5 As shown, the data digital analysis and matching method based on artificial intelligence includes the following steps:

[0145] First, input user behavior data and preprocess the data. The preprocessing includes the following steps: For text data: use the word embedding layer, BiLSTM layer and CRF layer to represent and extract key information features of the text data, and obtain the text feature vector corresponding to the text data; For image data: determine whether the image data has text information; if the image data has text information, perform text area detection, text direction classification and text recognition to obtain the text feature vector corresponding to the image data with text information; if the image data has no text information, perform intelligent data labeling and feature extraction to obtain the image feature vector corresponding to the image data; input search terms, use the intelligent matching model to calculate the cosine similarity between the search terms and the feature vectors corresponding to the user behavior data, and output a result with high similarity.

[0146] In one embodiment, after analyzing and matching the cosine similarity between each first feature vector and the search term to determine the target user behavior data, it also includes: calling the GCN model to generate the target user behavior data to obtain the user's social relationship embedding features; based on the social relationship embedding features, calling the GRU model to construct the user's spatiotemporal features, and performing feature construction based on the information cascade on the user's spatiotemporal features to obtain the user's spatiotemporal cascade features; obtaining the information dissemination content that the user will participate in, constructing a multimodal joint feature based on the information dissemination content, and determining the interest influence factor based on the multimodal joint feature, wherein the interest influence factor is used to characterize the user's content preference for the information dissemination content; determining the dot product similarity between the user's spatiotemporal cascade feature and the interest influence factor, and predicting the user's participation probability in the information dissemination content based on the dot product similarity.

[0147] Because each user has different social relationships, and each user has different preferences for different information dissemination content, all users are taken as nodes, and the social relationships between users are taken as undirected edges according to the target user behavior data, so as to construct a user social network relationship graph based on the nodes and undirected edges, and the information dissemination content can be freely propagated between different user nodes in the graph according to the undirected edges. Since graph convolutional neural networks (GCN) are suitable for graph structure type data, the present invention calls the GCN model to generate the target user behavior data and obtain the user's social relationship embedding features. That is, the GCN model is used to extract the features of each node (user) in the user social network relationship graph to obtain the user's social relationship embedding features, which are expressed as The extraction process is expressed as the following formula (1) and formula (2):

[0148]

[0149] In the above formula (1) and formula (2), Represents the vector representation of the node (user) in the lth layer of the GCN model (i.e., the social relationship embedding feature), is the input of the first layer of the GCN model, which is a feature vector that can be randomly initialized, and N represents the number of nodes (users), and d represents the dimension of the feature vector. f represents the adjacency matrix constructed by the GCN model based on the user social network relationship graph, E represents the identity matrix, Represents the adjacency matrix A f The matrix obtained by fusing the identity matrix E, D f Indicates A f The degree matrix of Denoting the degree matrix D f The matrix obtained by fusing the identity matrix E, W f Represents a trainable weight matrix. Relu represents the activation function of the network layer in the GCN model.

[0150] The GCN model is used to generate the user's social relationship embedding features, that is, to determine the user's feature vector representation. Next, based on the social relationship embedding features, the GRU model is called to build the user's spatiotemporal features. Since the user's spatiotemporal features need to be integrated with the user's cascade information, the user's cascade information needs to be extracted first.

[0151] The embodiment of the present invention first obtains the user's historical cascade data, and divides the user's historical cascade data into multiple cascade subgraphs according to timestamps. Since there is a spatial and temporal correlation between each cascade subgraph, the modeling process of the dynamically changing information cascade graph is converted into an aggregation problem of spatial and temporal features.

[0152] See also Figure 6 , Figure 6 : This is a schematic diagram of the information cascade graph embedding layer based on spatiotemporal features provided by the present invention. The user information and cascade information are aggregated through a multi-layer GCN network. The social relationship embedding features and the cascade subgraph are aggregated at the input layer to obtain the cascade subgraph embedding features corresponding to each timestamp, which is recorded as A. t , where t represents the timestamp. Since each cascade subgraph is segmented according to the timestamp, each timestamp also corresponds to a cascade subgraph embedding feature. Figure 6 In the example, the n cascaded subgraph embedding features are represented as “A t , A t-1 , …, A t-n The corresponding n timestamps are expressed as “t, t-1, …, tn”.

[0153] Next, determine the time embedding feature corresponding to each timestamp, that is, construct the corresponding time embedding feature according to the timestamp. Figure 6As shown, for each of the n timestamps, a corresponding time embedding feature is constructed. In the spatial feature extraction stage, the cascaded subgraph embedding features and the time embedding features are input into the GCN model for spatial feature extraction to obtain a spatial feature sequence. Considering that the gated recurrent unit (Gated Recurrent Unit, GRU) can better handle long sequences and has a simpler structure, in the spatiotemporal feature extraction stage, the GRU model is called to extract features from the spatial feature sequence to obtain user spatiotemporal features, so that the GRU model can understand the dynamic changes of the data in the spatial feature sequence, so that the user spatiotemporal features can better simulate the feature representation of the user at the corresponding timestamp.

[0154] like Figure 6 As shown, at timestamp t-1 in the spatiotemporal feature extraction phase, the user spatiotemporal features are obtained in the t-1th gated recurrent unit (GRU) in the GRU model. After that, it will be input into the GCN model. In the GCN model, the time embedding feature p of the next timestamp t is combined t , extract the spatial feature sequence at timestamp t, and then input the spatial feature sequence at timestamp t into the tth gated recurrent unit to obtain the user spatiotemporal feature at timestamp t Therefore, at the subsequent timestamp t+1, the corresponding user

[0155]

[0156] In the above formula (3), t represents the timestamp, p t represents the time embedding feature corresponding to timestamp t, represents the dimension of temporal embedding features, Represents the user's spatiotemporal features at timestamp t At the first timestamp (i.e., t=0), the user's spatiotemporal features Using social relationship embedding features (L represents the number of layers of the GCN network) for initialization. represents the adjacency matrix in the user social network relationship graph at timestamp t+1, N represents the number of nodes (users) in the user social network relationship graph. GCN represents the processing function of the network layer in the GCN model. Relu represents the activation function of the gate unit in the GRU model, and GRU represents the processing function of the gate unit in the GRU model.

[0157] After determining the user's spatiotemporal features, the user's spatiotemporal features are then subjected to feature construction based on information cascade to obtain the user's spatiotemporal cascade features. In order to obtain the user's content preference features at different timestamps, the embodiment of the present invention uses a gated fusion mechanism to fuse the user's social information with the user's spatiotemporal features, then splices the randomly initialized position embedding features, and finally uses a masked self-attention mechanism to learn sequence representation.

[0158] Specifically, first, for the user time features corresponding to each timestamp, the social relationship embedding features are gated and fused with the user spatiotemporal features to obtain a fused spatiotemporal feature sequence.

[0159] For the user time feature corresponding to each timestamp, the gated fusion process is shown in the following formula (5):

[0160] GatingFusion(X f ,X c )=rX f +(1-r)X c (5)

[0161] In the above formula (5), GatingFusion represents the processing function of gated fusion, X f represents the user's social relationship embedding feature, X c represents the user's spatiotemporal features, r represents the weight parameter for measuring the social relationship embedding features and spatiotemporal features, and the calculation formula is as follows:

[0162]

[0163] In the above formula (6), exp represents the natural exponential function, W a Represents the weight matrix corresponding to the weight parameter r. f Represents the user social relationship embedding feature X f The corresponding coding feature is calculated as follows: c Represents user spatiotemporal features X c The corresponding coding feature is calculated as follows:

[0164] h f =tanh(w f X f ) (6)

[0165] h c =tanh(w c X c ) (7)

[0166] In the above formulas (6) and (7), w f Represents the user social relationship embedding feature X fThe corresponding weight matrix, w c Represents user spatiotemporal features X c The corresponding weight matrix, tanh represents the hyperbolic tangent function.

[0167] Through the above formulas (5) to (8), the social relationship embedding features at each timestamp can be gated and fused with the user spatiotemporal features, and then the results of the gated fusion at each timestamp can be sequenced to obtain a fused spatiotemporal feature sequence.

[0168] Next, determine the position embedding code corresponding to each sequence element in the fused spatiotemporal feature sequence, denoted as L , and then concatenate the position embedding code with the sequence element to obtain the cascade feature sequence. , that is, the corresponding position feature is incorporated into each sequence element. Therefore, each sequence element in the cascade feature sequence is denoted as h, which is expressed as the following formula (9):

[0169] h=[GatingFusion(X f ,X c ),O L ]

[0170] In the above formula (9), X f represents the user's social relationship embedding feature, X c Represents user spatiotemporal features, GatingFusion(X f ,X c ) represents the result of gated fusion under the timestamp, O L Indicates the corresponding position embedding code.

[0171] Finally, the cascade feature sequence is input into the Transformer encoder model for masked attention mechanism processing to obtain the user spatiotemporal cascade features.

[0172] Considering that the Transformer model is a model based on the attention mechanism, the sequence feature representation of the spatiotemporal cascade feature sequence can be learned through the attention mechanism. Therefore, the Transformer model is called to extract the features of the cascade feature sequence to obtain the user's spatiotemporal cascade feature H. In addition, since some nodes (users) in the user social network relationship graph may be activated users, for example, a user has disseminated (forwarded or shared) information to some other users before the current timestamp, the possibility of the user continuing to disseminate information to other users after the current timestamp will be very small. Therefore, in the cascade feature sequence, the sequence elements corresponding to certain timestamps cannot represent the specific features of the user and belong to irrelevant sequence elements. Therefore, the present invention adopts a masked self-attention mechanism, firstly performs sequence masking on the cascade feature sequence, that is, masks some sequence elements in the cascade feature sequence, and then inputs the masked cascade feature sequence into the Transformer model, and uses the multi-head attention network in the Transformer model to process the cascade feature sequence with the self-attention mechanism to obtain the cascade feature h after each head in the multi-head attention network is processed. i , the processing process of the masked self-attention mechanism is expressed as follows:

[0173]

[0174] In the above formula (10), X represents the sequence element in the cascade feature sequence obtained by sequence masking, They represent the weight matrices corresponding to the query vector, key vector, and value vector of the sequence element X in the self-attention mechanism, respectively. MaskAtt represents the processing function of the self-attention mechanism in the Transformer model, which is expressed as the following formula (11):

[0175]

[0176] In the above formula (11), Q, K, and V represent the query vector, key vector, and value vector in the attention mechanism, respectively. d represents the vector dimension. Mask represents the sequence element for sequence masking in the cascade feature sequence. Softmax represents the normalized exponential function commonly used in the attention mechanism processing process.

[0177] Through the above formulas (10) and (11), the cascade feature h of each head in the multi-head attention network can be calculated i Next, we can extract the cascade sequence based on the user's historical cascade data, and then find the cascade feature corresponding to the user at the last timestamp based on the cascade sequence, thereby determining the cascade feature h at each timestamp. iThe cascade order is calculated and concatenated according to the cascade order to obtain the overall cascade feature combination. Then, the overall cascade feature combination is subjected to feature mapping by the last multilayer perceptron (MLP) in the Transformer model to obtain the final user spatiotemporal cascade feature H, which is expressed as the following formula (12):

[0178]

[0179] In the above formula (12), MLP represents the processing function of the multi-layer perceptron, h head represents the number of heads of the multi-head attention network in the Transformer model, [h1; h2; ...; h Head ] represents the overall cascade feature combination. The final user spatiotemporal cascade feature H can accurately characterize the dynamic cascade features of users in complex social network relationships under time and space conditions.

[0180] In this way, the dynamic cascade characteristics of users in complex social network relationships can be accurately characterized.

[0181] Furthermore, after constructing the user's spatiotemporal cascade features (i.e., the dynamic cascade features of the user under complex social network relationships), the user's content preference features are determined to characterize the user's content preference for information dissemination content, thereby predicting the user's participation probability in information dissemination content.

[0182] First, it is necessary to obtain the information dissemination content. The information dissemination content in the embodiment of the present invention belongs to the collected information graphic data, specifically including visual pictures and text fragments corresponding to the visual pictures. For example, the information dissemination content can be a post, an advertisement or a news, which contains a text introduction and corresponding pictures. The text introduction can be divided into multiple text fragments according to the pictures, and each text fragment introduces a picture. The picture is a visual picture, and the text fragment introducing the picture is the text fragment corresponding to the visual picture. Since the information graphic data involves two forms, text and image, the information dissemination content is also called multimodal joint information. Therefore, the present invention constructs multimodal joint features based on the information dissemination content, which are specifically explained as follows:

[0183] For multimodal joint information, the present invention adopts the Contrastive Language-Image Pre-training (CLIP) model to extract multimodal joint features. For visual images, in the information dissemination content, there are problems such as different sizes and color mismatch in visual images. Therefore, the visual images are transformed to adapt the visual images to the CLIP model. First, the shape of the visual image is changed to a uniform width of 224 and height of 224. Cubic spline interpolation is used when scaling the image. The visual image is then normalized and the data range is transformed into a standard normal distribution. Finally, the processed visual image is input into the image encoder of the CLIP model, and the corresponding image coding features are obtained by encoding through the image encoder. For the j-th visual image corresponding to the information dissemination content (for example, a post, denoted as i), it can be denoted as img ij , the initial image features can be obtained through the image encoder of the CLIP model, denoted as If there is no corresponding visual image in the information dissemination content, the initial image features are all initialized to 0 vectors.

[0184] Considering that the information dissemination content in social networks may contain multiple visual images, the importance of each visual image varies, and the position of the visual image affects the expression of the overall information dissemination content to a certain extent. In order to express the differences in importance and position between visual images, the present invention constructs a visual information processing module based on the Transformer model, and further extracts image features from the initial image features obtained by the CLIP model through the visual information processing module to obtain the image coding features of the visual images in the information dissemination content. That is, the initial image features The input is input into the Transformer model for embedded coding to obtain the image coding features of the visual picture.

[0185] In one embodiment, according to the distribution of visual images in the information dissemination content, the initial image features of each visual image are As sequence elements, the sequence elements are then concatenated to obtain an embedded sequence. For example, according to the distribution of illustrations in a post, the initial image features of each illustration are used as sequence elements, and the embedded sequence of the illustrations is concatenated.

[0186] In order to aggregate the information features of multiple visual images in the information dissemination content, that is, to aggregate the information of multiple pictures in a post and obtain global features, the present invention inserts the CLS tag as the first sequence element into the head of the embedding sequence to obtain the image embedding sequence, that is, to add the "[CLS]" tag at the first position of the embedding sequence of the picture. The CLS tag is inserted at position 0 of the sequence. In addition, in order to avoid bias towards a certain visual picture in the embedding sequence, the embedding of the "[CLS]" tag is set to random initialization, which will be continuously updated during training, and can encode the statistical characteristics of the entire embedding sequence.

[0187] After inserting the CLS tag to obtain the image embedding sequence, the position encoding pos of each sequence element in the image embedding sequence is then determined j , and concatenate the position code with the sequence element to obtain the concatenated feature It is expressed as follows:

[0188]

[0189] Finally, each concatenated feature in the image embedding sequence is input into the Transformer model for encoding processing to obtain the embedded coding features of the visual image. Similar to the processing process of the cascade feature sequence above, some visual images in the information dissemination content may have been forwarded or shared by users and cannot represent the specific characteristics of the information dissemination content. These visual images are irrelevant sequence elements, so the masking mechanism is still used to first perform sequence masking on the image embedding sequence, that is, randomly mask some sequence elements in the image embedding sequence (do not mask the inserted CLS tags), and then input the masked image embedding sequence into the Transformer model. The multi-head attention network in the Transformer Encoder is used to perform self-attention mechanism processing on the image embedding sequence, and then the result of the self-attention mechanism processing is input into the final MLP layer, and the final output embedded coding features of the visual image are recorded as It is expressed as the following formula (14):

[0190]

[0191] In the above formula (14), TransformerEncoder represents the processing function encoded by the Transformer model, and mask represents the sequence element of the mask processing.

[0192] For the text fragments corresponding to the visual images, since each text fragment corresponds to a visual image, for the information dissemination content, the text fragments of each visual image can be intercepted in the form of a sliding window, and then the text encoder of the CLIP model is called to encode the text fragments corresponding to each visual image to obtain the text encoder features, which are recorded as Then the text encoding features of each text segment are merged to obtain the overall text encoding features, recorded as The calculation process is expressed as the following formula (15):

[0193]

[0194] In the above formula (15), n represents the total number of text segments obtained by the sliding window, i represents the i-th information dissemination content, and j represents the j-th text segment in the information dissemination content.

[0195] Finally, the image encoding features The overall encoding features of the text The present invention uses a gated fusion mechanism to learn the importance of different modal information and the interaction between models, fuses image features with text features, and obtains multimodal joint features, denoted as w i , the multimodal joint features can accurately express the information characteristics of the information dissemination content, which is convenient for subsequent matching with user preferences to complete information dissemination. Gated fusion can be achieved by using a neural network model or a feature fusion algorithm. The calculation process is expressed as the following formula (16):

[0196]

[0197] In the above formula (16), GatingFusion represents the processing function of gated fusion.

[0198] Furthermore, after extracting the multimodal joint features of the information dissemination content, the interest influencing factors are determined based on the multimodal joint features to express the user's interest characteristics.

[0199] Considering that user interests change over time and that the information dissemination content that users have interacted with in the early stage will have an impact on their later interests, the information dissemination content that users participate in is arranged in chronological order. In order to determine the user's interest characteristics, the user's historical interaction data is first obtained, that is, the interaction record data of the information dissemination content that the user has participated in. Then, the embedded feature extraction algorithm is used to extract the interaction features of the user's historical interaction data as the user's participation information embedding feature, denoted as W, and W = [w A ,w B,……]. According to the above formula (16), the multimodal joint feature of each information dissemination content is denoted as w i , so the multimodal joint features of the information dissemination content to be predicted can be recorded as w, and w=[w1,w2,…w i ], indicating that each piece of information dissemination must be predicted.

[0200] Next, the Transformer model is called to encode the participation information embedding feature W to obtain the user interest embedding feature W. Because user interests change over time, and the information dissemination content of previous interactions will affect the user's later interests, the Transformer encoding layer is called again for encoding in order to learn the temporal characteristics of user interests and extract the user interest embedding features containing temporal characteristics. The encoding process is expressed as the following formula (17):

[0201]

[0202] TransformerEncoder represents the processing function of the Transformer encoding layer.

[0203] Considering the large amount of historical user interaction data, the feature dimension of the user interest embedding feature W is very large. Therefore, before calculating the user interest feature, the user interest embedding feature W is averaged and pooled to obtain the average pooled feature. Then, the feature similarity I between the average pooled feature and the multimodal joint feature w is determined, and the feature similarity I is used as the interest influencing factor. The feature similarity can be calculated by Euclidean distance or cosine similarity. The higher the feature similarity, the more compatible the corresponding information dissemination content is with the user's interest, and the more likely the user is to participate in the information dissemination content and share or forward it. The lower the feature similarity, the less compatible the corresponding information dissemination content is with the user's interest, and the lower the probability of the user participating in the information dissemination content, the lower the probability of sharing or forwarding the information dissemination content. The formula for calculating the interest influencing factor is expressed as follows: Formula (18):

[0204]

[0205] In the above formula (18), AvgPool represents the processing function of the average pooling operation. represents the user interest embedding feature, w represents the multimodal joint feature to be predicted. Simi lar ity represents the function for calculating feature similarity, which is expressed as the following formula (19):

[0206]

[0207] In the above formula (19), A i and Bi Represents two feature vectors whose feature similarity needs to be calculated, and d represents two feature vectors A i , B i The feature dimension of , i represents the ordinal number of the feature dimension.

[0208] In this way, information content and user preferences are integrated, and a preliminary estimate is made on the possibility of users participating in information dissemination content, which can further improve the accuracy of cascade predictions.

[0209] Furthermore, after determining the user's interest influencing factors, the next step is to combine the user's dynamic cascade features under complex social network relationships (i.e., the final user spatiotemporal cascade features H obtained in step 102) to make a final prediction on whether the user is likely to participate in the information dissemination content.

[0210] The embodiment of the present invention directly calculates the dot product similarity between the user's spatiotemporal cascade feature H and the interest influencing factor I, and then predicts the user's participation probability for the information dissemination content based on the dot product similarity, which is recorded as y'. The prediction process can be implemented by MLP, and the formula is expressed as the following formula (20):

[0211] y'=MLP(H·I) (20)

[0212] In the above formula (20), H·I represents the calculation of the dot product similarity between the user's spatiotemporal cascade feature H and the interest influence factor I, and MLP represents the processing function of the multi-layer perceptron to predict the participation probability.

[0213] In some embodiments, the predicted probability of a user participating in information dissemination content may also be predicted by calling an information cascade prediction model. The information cascade prediction model specifically includes modules such as a GCN model, a GRU model, a Transformer model, an MLP model, and a CLIP model. The training process of the information cascade prediction model is as follows:

[0214] First, you need to obtain training samples for the information cascade prediction model. You can collect target user behavior data samples, information dissemination content samples that users will participate in, user historical cascade data samples, and user historical interaction data samples, and use these samples as model training samples.

[0215] Then, these model training samples are input into the information cascade prediction model for forward propagation to obtain the predicted participation probability of users for the information dissemination content samples. The forward propagation process is described below.

[0216] In the forward propagation process, at the beginning, based on the target user behavior data sample of the user, the GCN model is first called to generate the user's social relationship embedding features, and then based on the user's historical cascade data samples, the cascade subgraph and cascade sequence are generated respectively. By calling the GCN model and the GRU model, the user's time features are generated based on the cascade subgraph and social relationship embedding features. By calling the Transformer model, the user's spatiotemporal cascade features are generated based on the cascade sequence and the user's time features. Next, based on the information dissemination content samples that the user will participate in, the CLIP model is called to extract the multimodal joint features of the information dissemination content samples, and then based on the user's historical interaction data samples, the Transformer model is called to generate the user's interest embedding features. The next step is to calculate the similarity between the user's interest embedding features and the multimodal joint features as the interest influence factor. Finally, the dot product similarity between the interest influence factor and the user's spatiotemporal cascade features is calculated and input into the MLP to obtain the predicted participation probability, that is, the user's participation probability y' for the information dissemination content sample. In this way, the forward propagation process of the information cascade prediction model ends.

[0217] Next, determine the user's actual participation probability for the information dissemination content sample, that is, determine the user's actual participation in the information dissemination content sample based on the actual information dissemination environment, denoted as y. Then, based on the predicted participation probability and the actual participation probability, construct the cross entropy loss function Loss, and the construction process is expressed as the following formula (21):

[0218]

[0219] In the above formula (21), n ​​represents the number of information dissemination content samples, i represents the i-th information dissemination content sample, y' represents the predicted participation probability obtained in the forward propagation process of the information cascade prediction model, and y represents the actual participation probability of the user for the information dissemination content sample.

[0220] After constructing the cross entropy loss function, the cross entropy loss function is used to back propagate in the information cascade prediction model. The process of back propagation is also the process of calculating the gradient. The gradient is optimized by the gradient descent algorithm to update the parameters of the information cascade prediction model, so that the cross entropy loss function gradually converges.

[0221] The training process is also an iterative process. In each iteration, forward propagation and backward propagation are performed. In each iteration, the forward propagation process needs to determine whether the cross entropy loss function Loss has decreased significantly. If it does not decrease significantly, it means that the cross entropy loss function has converged. Then the predicted participation probability is output, and this iteration ends, that is, the training process ends. If it still decreases significantly, it means that the cross entropy loss function has not converged, and backward propagation continues to update the parameters of the information cascade prediction model.

[0222] The trained information cascade prediction model can be used directly for prediction. According to the input user information and information dissemination samples, the user's participation probability in the information dissemination content can be directly predicted. According to the predicted participation probability, the dissemination of information on the social networking platform can be understood very intuitively, so as to make targeted information diffusion and complete the cascade process.

[0223] The following is a description of the data digital analysis and matching device based on artificial intelligence provided by the present invention. The data digital analysis and matching device based on artificial intelligence described below and the data digital analysis and matching method based on artificial intelligence described above can be referred to each other. Figure 7 , Figure 7 This is a diagram showing an embodiment of a data digital analysis and matching device based on artificial intelligence provided by an embodiment of the present invention. Figure 7 As shown, the data digital analysis and matching device based on artificial intelligence includes:

[0224] An acquisition module 710 is used to acquire search terms of cross-modal user behavior data;

[0225] The feature extraction module 720 is used to extract features from at least one user behavior data in the original user behavior database to obtain a first feature vector corresponding to each user behavior data; wherein the type of the user behavior data includes any one of the following: text data, image data without text, and image data containing text;

[0226] A similarity calculation module 730 is used to perform similarity calculation using an intelligent matching model to determine the cosine similarity between each first feature vector and the search term;

[0227] The analysis and matching module 740 is used to perform analysis and matching based on the cosine similarity between each first feature vector and the search term to determine the target user behavior data.

[0228] The artificial intelligence-based data digital analysis and matching device provided by the present invention effectively realizes cross-modal intelligent data matching between typical modalities of images and texts.

[0229] See also Figure 8 , Figure 8 8 is a structural diagram of an electronic device provided by the present invention. The electronic device 800 includes a memory 810, a processor 820, and a computer program 811 stored in the memory 810 and executable on the processor 820. When the processor 820 executes the computer program 811, the following steps are implemented:

[0230] Search terms to obtain cross-modal user behavior data;

[0231] Performing feature extraction on at least one user behavior data in the original user behavior database to obtain a first feature vector corresponding to each user behavior data; wherein the type of the user behavior data includes any one of the following: text data, image data without text, and image data containing text;

[0232] Using the intelligent matching model to perform similarity calculation, determine the cosine similarity between each first feature vector and the search term;

[0233] The target user behavior data is determined by analyzing and matching the cosine similarity between each first feature vector and the search term.

[0234] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods of each embodiment or some parts of the embodiment.

[0235] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A data digital analysis and matching method based on artificial intelligence, characterized in that: include: Search terms to obtain cross-modal user behavior data; Performing feature extraction on at least one user behavior data in the original user behavior database to obtain a first feature vector corresponding to each of the user behavior data; wherein the type of the user behavior data includes any one of the following: text data, image data without text, and image data containing text; Using an intelligent matching model to perform similarity calculation, determining the cosine similarity between each of the first feature vectors and the search term; Analyze and match the cosine similarity between each of the first feature vectors and the search term to determine the target user behavior data.

2. The data digital analysis and matching method based on artificial intelligence according to claim 1 is characterized in that: The step of extracting features from at least one user behavior data in the original user behavior database to obtain a first feature vector corresponding to each user behavior data includes: In the case where the user behavior data is text data, using a bidirectional long short-term memory network model to extract key information from the text data to obtain a first text feature vector; Determining the first text feature vector as a first feature vector corresponding to the user behavior data; Correspondingly, the bidirectional long short-term memory network model includes a word embedding layer, a bidirectional long short-term memory network layer and a conditional random field model layer; the step of extracting key information from the text data using the bidirectional long short-term memory network model to obtain a first text feature vector includes: Input the text data into the word embedding layer to obtain a word vector matrix corresponding to the text data, wherein the word vector matrix is ​​a continuous vector representation; Inputting the word vector matrix corresponding to the text data into the bidirectional long short-term memory network layer to obtain a context feature sequence of at least two time steps; Using the conditional random field model layer, the context feature sequence of each time step is labeled, and the output score of each labeled context feature sequence relative to the label prediction sequence is calculated, and the output score is used to characterize the credibility corresponding to the context feature sequence of the time step; The conditional random field model layer is used to determine the text feature vector according to the output scores of each annotated context feature sequence relative to the label prediction sequence.

3. The data digital analysis and matching method based on artificial intelligence according to claim 1 is characterized in that: The step of extracting features from at least one user behavior data in the original user behavior database to obtain a first feature vector corresponding to each user behavior data includes: In the case where the user behavior data is picture data without text, the picture data with text is input into a face recognition network to extract key information of the picture data with text; According to the key information of the picture data of the text, the face in the picture data of the text is framed by using the yolov5 model to obtain a face picture; Using a deep convolutional neural network to extract features from the face image, and obtain a feature vector corresponding to the face image; The feature vector corresponding to the face image is determined as the first feature vector corresponding to the user behavior data.

4. The data digital analysis and matching method based on artificial intelligence according to claim 1 is characterized in that: The step of extracting features from at least one user behavior data in the original user behavior database to obtain a first feature vector corresponding to each user behavior data includes: In the case where the user behavior data is picture data containing text, the picture data containing text is input into a text detection model to identify a text region in the picture data containing text; According to the text region in the picture data containing text, using a text direction classifier to detect and obtain the text direction of the text instance in the text region; Based on the text direction, the text instance is rotated to a parallel direction to obtain a rotated text instance; Inputting the rotated text instance into a text recognition model to obtain a second text feature vector; The second text feature vector is determined as the first feature vector corresponding to the user behavior data.

5. The data digital analysis and matching method based on artificial intelligence according to any one of claims 1 to 4, characterized in that: After analyzing and matching the cosine similarity between each of the first feature vectors and the search term to determine the target user behavior data, the method further includes: Calling the GCN model to generate the target user behavior data to obtain the user's social relationship embedding features; Based on the social relationship embedding features, the GRU model is called to construct a user spatiotemporal feature, and the user spatiotemporal feature is subjected to feature construction based on information cascade to obtain a user spatiotemporal cascade feature; Acquire information dissemination content that the user will participate in, construct a multimodal joint feature based on the information dissemination content, and determine an interest influence factor according to the multimodal joint feature, wherein the interest influence factor is used to characterize the user's content preference for the information dissemination content; The dot product similarity between the user's spatiotemporal cascade feature and the interest influencing factor is determined, and the user's participation probability in the information dissemination content is predicted based on the dot product similarity.

6. The data digital analysis and matching method based on artificial intelligence according to claim 5 is characterized in that: The embedding feature based on the social relationship, Call the GRU model to build the user's spatiotemporal features, including: Acquire user historical cascade data, and divide the user historical cascade data into multiple cascade subgraphs according to timestamps; Aggregating the social relationship embedding feature with the cascade subgraph to obtain a cascade subgraph embedding feature corresponding to each timestamp; Determine the time embedding feature corresponding to each timestamp; Inputting the cascade subgraph embedding features and the time embedding features into the GCN model to extract spatial features to obtain a spatial feature sequence; The GRU model is called to extract features from the spatial feature sequence to obtain user spatiotemporal features.

7. The data digital analysis and matching method based on artificial intelligence according to claim 5 is characterized in that: The step of constructing the user spatiotemporal features based on information cascade to obtain user spatiotemporal cascade features includes: Performing gated fusion on the social relationship embedding feature and the user spatiotemporal feature to obtain a fused spatiotemporal feature sequence; Determine the position embedding code corresponding to each sequence element in the fused spatiotemporal feature sequence, and concatenate the position embedding code with the sequence element to obtain a spatiotemporal cascade feature sequence; The spatiotemporal cascade feature sequence is input into the Transformer model for masked attention mechanism processing to obtain user spatiotemporal cascade features.

8. The data digital analysis and matching method based on artificial intelligence according to claim 5 is characterized in that: The information dissemination content includes a visual image and a text segment corresponding to the visual image, and the multimodal joint feature is constructed based on the information dissemination content, including: Calling the image encoder of the CLIP model to perform image encoding on each of the visual pictures to obtain initial image features; Inputting the initial image features into the Transformer model for embedding coding to obtain image coding features of the visual picture; Calling the text encoder of the CLIP model to encode the text fragments corresponding to each of the visual images to obtain text encoder features, and merging the text encoding features to obtain the overall text encoding features; The image coding features are gated and fused with the overall text coding features to obtain multimodal joint features.

9. The data digital analysis and matching method based on artificial intelligence according to claim 8 is characterized in that: The step of inputting the initial image features into the Transformer model for embedding coding to obtain the embedded coding features of the visual image includes: Each of the initial image features is concatenated as a sequence element to obtain an embedded sequence, and a CLS tag is inserted into the head of the embedded sequence as the first sequence element to obtain an image embedded sequence; Determine the position code of each sequence element in the image embedding sequence, and concatenate the position code with the sequence element to obtain a concatenation feature; Each concatenated feature in the image embedding sequence is input into the Transformer model for encoding processing to obtain the embedded encoding feature of the visual image.

10. A data digital analysis and matching device based on artificial intelligence, characterized in that: include: An acquisition module, used to obtain search terms for cross-modal user behavior data; A feature extraction module, used to extract features from at least one user behavior data in the original user behavior database to obtain a first feature vector corresponding to each of the user behavior data; wherein the type of the user behavior data includes any one of the following: text data, image data without text, and image data containing text; A similarity calculation module, used to perform similarity calculation using an intelligent matching model to determine the cosine similarity between each of the first feature vectors and the search term; The analysis and matching module is used to perform analysis and matching based on the cosine similarity between each of the first feature vectors and the search term to determine the target user behavior data.

Images