Image-text emotion type determination method and device, equipment and medium

Abstract

The embodiment of the invention discloses an image-text emotion type determination method and device, equipment and a medium. The method comprises the steps of obtaining to-be-analyzed graphic and text information corresponding to a target user, and performing graphic and text preprocessing on the to-be-analyzed graphic and text information to determine target graphic and text information corresponding to the target user; the target graphic and text information comprises target image information and target text information; inputting the target image-text information into a target image-text emotion prediction model for feature extraction, feature intersection, feature fusion and emotion type prediction; and determining a target image-text emotion type corresponding to the target user based on the output of the target image-text emotion prediction model. Through the technical scheme of the embodiment of the invention, the target image-text emotion type can be accurately and conveniently determined, so that the user emotion type is comprehensively predicted based on multi-modal information, and the efficiency and accuracy of determining the image-text emotion type are improved.

Description

Technical Field

[0001] Embodiments of the present invention relate to the field of computer technology, and in particular to a method, device, equipment and medium for determining the emotion type of a picture or text. Background Art

[0002] With the development of technology, it is crucial to deeply understand the emotional needs of users in the design and improvement of financial products. User comments on financial products contain rich emotions and opinions, and are an important source of information reflecting user satisfaction and expectations.

[0003] At present, traditional financial sentiment analysis mainly relies on text data, such as news reports, market analysis, and investor comments, using early machine learning and deep learning technologies to extract the sentiment tendencies and market insights of these texts. However, with the rapid development of the Internet and digital media, methods that rely solely on a single data source have gradually revealed their limitations. Summary of the invention

[0004] The embodiments of the present invention provide a method, device, equipment and medium for determining the emotion type of a picture and text, so as to accurately and conveniently determine the emotion type of a target picture and text, thereby realizing comprehensive prediction of the user emotion type based on multimodal information, and improving the efficiency and accuracy of determining the emotion type of the picture and text.

[0005] In a first aspect, an embodiment of the present invention provides a method for determining a graphic and text emotion type, comprising:

[0006] Acquire the to-be-analyzed graphic information corresponding to the target user, and perform graphic preprocessing on the to-be-analyzed graphic information to determine the target graphic information corresponding to the target user; the target graphic information includes: target image information and target text information;

[0007] Inputting the target image-text information into the target image-text emotion prediction model for feature extraction, feature cross-talk, feature fusion and emotion type prediction;

[0008] Based on the output of the target image-text emotion prediction model, the target image-text emotion type corresponding to the target user is determined.

[0009] In a second aspect, an embodiment of the present invention further provides a device for determining a graphic and text emotion type, the device comprising:

[0010] The target graphic information determination module is used to obtain the graphic information to be analyzed corresponding to the target user, and perform graphic preprocessing on the graphic information to be analyzed to determine the target graphic information corresponding to the target user; the target graphic information includes: target image information and target text information;

[0011] The image-text emotion prediction module is used to input the target image-text information into the target image-text emotion prediction model for feature extraction, feature cross-talk, feature fusion and emotion type prediction;

[0012] The target image-text emotion type determination module is used to determine the target image-text emotion type corresponding to the target user based on the output of the target image-text emotion prediction model.

[0013] In a third aspect, an embodiment of the present invention further provides an electronic device, the electronic device comprising:

[0014] one or more processors;

[0015] A memory for storing one or more programs;

[0016] When the one or more programs are executed by the one or more processors, the one or more processors implement the method for determining the image and text emotion type provided by any embodiment of the present invention.

[0017] In a fourth aspect, an embodiment of the present invention provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements a method for determining a graphic and text emotion type as provided in any embodiment of the present invention.

[0018] In a fifth aspect, an embodiment of the present invention provides a computer program product, including a computer program, which, when executed by a processor, implements the method for determining the emotion type of a picture and text provided in any embodiment of the present invention.

[0019] The technical solution of the embodiment of the present invention obtains the to-be-analyzed graphic and text information corresponding to the target user, and performs graphic and text preprocessing on the to-be-analyzed graphic and text information to determine the target graphic and text information corresponding to the target user; the target graphic and text information includes: target image information and target text information; the target graphic and text information is input into a target graphic and text emotion prediction model for feature extraction, feature cross-linking, feature fusion and emotion type prediction; based on the output of the target graphic and text emotion prediction model, the target graphic and text emotion type corresponding to the target user is determined, thereby accurately and conveniently determining the target graphic and text emotion type, and further realizing the comprehensive prediction of user emotion type based on multimodal information, thereby improving the efficiency and accuracy of determining the graphic and text emotion type.

[0020] It should be understood that the contents described in this section are not intended to identify the key or important features of the embodiments of the present invention, nor are they intended to limit the scope of the present invention. Other features of the present invention will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0022] Figure 1 This is a flow chart of a method for determining the emotion type of a picture and text provided in the first embodiment of the present invention;

[0023] Figure 2 This is a flow chart of a method for determining the emotion type of a picture and text provided in the second embodiment of the present invention;

[0024] Figure 3 This is an example diagram of a visualization of a model input representation involved in the second embodiment of the present invention;

[0025] Figure 4 is a structural example diagram of a core residual block involved in Embodiment 2 of the present invention;

[0026] Figure 5 is a structural example diagram of a multimodal feature extraction sub-model involved in the second embodiment of the present invention;

[0027] Figure 6 It is a structural schematic diagram of a device for determining the emotion type of a picture and text provided in Embodiment 3 of the present invention;

[0028] Figure 7 It is a structural schematic diagram of an electronic device for implementing the method for determining the image and text emotion type according to an embodiment of the present invention. DETAILED DESCRIPTION

[0029] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0030] It should be noted that the terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0031] Embodiment 1

[0032] Figure 1 A flowchart of a method for determining a text-image emotion type is provided for the first embodiment of the present invention. This embodiment is applicable to the case of comprehensively predicting a user's emotion type based on multimodal information. The method can be performed by a text-image emotion type determination device. The text-image emotion type determination device can be implemented in the form of hardware and / or software. The text-image emotion type determination device can be configured in an electronic device. Figure 1 As shown, the method includes:

[0033] S110: Acquire the to-be-analyzed graphic and text information corresponding to the target user, and perform graphic and text preprocessing on the to-be-analyzed graphic and text information to determine the target graphic and text information corresponding to the target user.

[0034] The target graphic information includes: target image information and target text information. The target user may refer to the user who uploads the graphic information to be analyzed. For example, the target user may be a user who evaluates a certain financial product. Alternatively, the target user may also be a staff member who conducts an evaluation survey on a certain financial product. The graphic information to be analyzed may refer to the original image information and original text information provided by the target user. The target graphic information may refer to the graphic information that can be input into the model after preprocessing. Preprocessing may include but is not limited to text cleaning and image resizing.

[0035] Specifically, in order to ensure the security of user data and personalized experience, the target user needs to log in before transmitting the image and text information to be analyzed. For registered users, the login interface provides an input box for the user name and password. The user can fill in the registered user name and password and click the "Login" button to log in. After that, identity authentication will be performed. After successful verification, the user will be directed to the main interface. For new users, the login interface provides a registration option. There is a "Register Account" button in the lower right corner of the login interface. After clicking it, a registration window will pop up. In this window, the user needs to fill in information such as the user name, password, and confirm password. After submission, the information entered by the user will be verified, such as whether the user name has been used and whether the password strength meets the requirements. After the verification passes, the user's account is successfully registered, and then the newly registered user name and password can be used to log in. After the target user successfully logs in to the system, he will enter the target user's homepage, which has an intuitive interface design and convenient operation. A text box is provided on the homepage, and the target user can directly enter the text to be analyzed in this text box. If the user wants to perform image and text sentiment analysis, he can click the "Select Input Image" button to upload a local image file. Supports multiple image formats, such as JPEG, PNG, etc. After uploading, the target user can preview the selected image on the same interface to ensure that the correct file is uploaded. After the image and text information to be analyzed is uploaded and submitted, the sentiment analysis process will be executed. After the target user clicks the "Submit" button, the image and text information to be analyzed uploaded by the target user will be passed to the backend for preprocessing, such as text cleaning, image size adjustment, etc., to obtain the target image and text information corresponding to the target user.

[0036] S120, inputting the target image-text information into the target image-text emotion prediction model for feature extraction, feature cross-talk, feature fusion and emotion type prediction.

[0037] The target image-text emotion prediction model may refer to a pre-trained image-text emotion prediction model. The target image-text emotion prediction model may be used to perform emotion prediction based on image-text features. Feature crossover may be understood as feature complementation, that is, complementing features under each modality.

[0038] Specifically, the target image and text information is input into the target image and text sentiment prediction model, image features are extracted for the target image information alone, and text features are extracted for the target text information alone, and then the extracted image features and text features are cross-featured, and then the cross-featured image and text features are fused with the extracted text features to obtain target image and text features that both deeply understand each modal information and complementarily fuse each modal feature. The target image and text features are used to predict sentiment types, thereby improving the accuracy and efficiency of sentiment analysis for financial products, and thus better serving the analysis and decision-making process of the financial market.

[0039] On the basis of the above technical scheme, "inputting the target graphic information into the target graphic emotion prediction model for unimodal feature extraction, multimodal feature intersection, feature fusion and emotion type prediction" may include: inputting the target graphic information into the multimodal feature extraction sub-model for unimodal feature extraction, multimodal feature intersection and multimodal feature fusion to obtain the target graphic features corresponding to the target graphic information; inputting the target graphic features into the emotion type prediction sub-model for emotion type prediction to obtain the target prediction result corresponding to the target graphic information.

[0040] Among them, the target image and text emotion prediction model includes: a multimodal feature extraction submodel and an emotion type prediction submodel. The multimodal feature extraction submodel can be used to extract features from multimodal information. The target image and text features may refer to features contained in the target image and text information. The advantage of such a setting is that the effect of emotion analysis is improved, that is, the target image and text emotion prediction model proposed in the present invention integrates the information of the two modes of image and text, and comprehensively considers the influence of multiple factors on emotion information compared to the unimodal model.

[0041] S130 : Determine the target image-text emotion type corresponding to the target user based on the output of the target image-text emotion prediction model.

[0042] The target image and text sentiment type may refer to the target user's usage experience category for a certain financial product. For example, the target image and text sentiment type may be, but is not limited to, positive, negative, and neutral, so as to help more accurately understand the sentiment tendency of the content uploaded by the target user (image and text information to be analyzed).

[0043] Specifically, after the target image and text sentiment prediction model is processed, the sentiment analysis result, i.e., the target image and text sentiment type, will be sent back to the front end and displayed to the target user.

[0044] The technical solution of the embodiment of the present invention obtains the to-be-analyzed graphic and text information corresponding to the target user, and performs graphic and text preprocessing on the to-be-analyzed graphic and text information to determine the target graphic and text information corresponding to the target user; the target graphic and text information includes: target image information and target text information; the target graphic and text information is input into a target graphic and text emotion prediction model for feature extraction, feature cross-linking, feature fusion and emotion type prediction; based on the output of the target graphic and text emotion prediction model, the target graphic and text emotion type corresponding to the target user is determined, thereby accurately and conveniently determining the target graphic and text emotion type, and then realizing the comprehensive prediction of the user emotion type based on multimodal information, thereby improving the efficiency and accuracy of the graphic and text emotion type determination.

[0045] Exemplarily, the training process of the target image-text emotion prediction model of the embodiment of the present invention is as follows.

[0046] Data processing and training are preliminary preparations. In actual application, different data can be collected for different scenarios, and special models for corresponding scenarios can be trained based on different data. Since customer information in the financial field is private data within the bank, there are very few open source financial-related Chinese text and image emotion recognition data sets on the Internet. The training data sets used in the demonstration of the present invention are MVSA and Financial PhraseBank. The data sets are introduced in detail as follows.

[0047] Financial PhraseBank consists of 4,845 texts randomly selected from financial news. These sentences were then annotated by 16 people with financial and business backgrounds. The annotators were asked to give labels based on how they thought the information in the sentence might affect the stock price of the company mentioned. The dataset also includes information about the level of agreement between the annotators.

[0048] MVSA is a picture and text dataset collected from Twitter, each piece of data contains a text and the corresponding image. MVSA contains two sub-datasets MVSA-Single and MVSA-Multiple. MVSA-Single contains 4870 text-image pairs, each piece of data contains a text-image emotion label pair. For each piece of data, if the image label is consistent with the text label, the data is retained and a multimodal emotion label is generated, otherwise the data is deleted. MVSA-Multiple contains 19600 text-image pairs, each piece of data contains three pairs of text-image emotion label pairs, totaling six emotion labels. Due to different annotations, the present invention first converts the text-image label pairs into corresponding multimodal emotion labels in the same way. If two or more of the three multimodal emotion labels are the same, the data is saved. If the three multimodal emotion labels contain both positive labels and negative labels, the data is directly deleted. Finally, by processing the above data, 4511 pieces of data in MVSA-Single and 17024 pieces of data in MVSA-Multiple are obtained. Then, the target image and text sentiment prediction model is obtained based on the data set training.

[0049] It should be noted that in order to cope with various application scenarios, different models can be replaced according to different usage scenarios. For example, in the scenario of Chinese text, the large-scale language model (Bidirectional EncoderRepresentations from Transformers, BERT) can be replaced with the Chinese BERT retrained model. Replacing the BERT model in the model with the Chinese BERT can achieve emotion recognition in Chinese scenarios.

[0050] Embodiment 2

[0051] Figure 2 This is a flowchart of a method for determining the emotion type of a picture and text provided in the second embodiment of the present invention. Based on the above embodiments, this embodiment describes in detail the process of determining the target picture and text features. The explanations of the terms that are the same or corresponding to the above embodiments are not repeated here. Figure 2 As shown, the method includes:

[0052] S210: Acquire the to-be-analyzed graphic and text information corresponding to the target user, and perform graphic and text preprocessing on the to-be-analyzed graphic and text information to determine the target graphic and text information corresponding to the target user.

[0053] The target graphic information includes: target image information and target text information.

[0054] S220 , inputting the target image and text information into the feature extraction layer to perform single-modal feature extraction, and obtaining reference image features and reference text features.

[0055] Among them, the multimodal feature extraction submodel includes: feature extraction layer, joint coding layer and residual coding layer. The feature extraction layer uses a single modal model to extract image and text features, the joint coding layer is used to obtain cross information between images and text, and the residual coding layer is used to fuse cross information with text information to obtain the final output features. The reference image feature may refer to the image feature extracted for the target image information. The reference text feature may refer to the text feature extracted for the target text information.

[0056] Specifically, the target image-text information is input into the feature extraction layer, and image features are extracted for the target image information in the target image-text information to obtain reference image features, and text features are extracted for the target text information in the target image-text information to obtain reference text features.

[0057] It should be noted that the embodiment of the present invention can also perform user sentiment analysis on all recent comments under a certain financial product. For example, the image and text sentiment recognition dataset under the financial scenario is defined as Where S i Represents an image / text pair in a user comment, Represents financial review text, Represents the corresponding image, y i Indicates the sentiment label of the financial image-text pair. The purpose of the embodiment of the present invention is to implement a model Map the financial image and text data pairs to reflect the customer's true emotional label y i .

[0058] Based on the above technical solution, "inputting the target image and text information into the feature extraction layer for unimodal feature extraction to obtain reference image features and reference text features" may include: inputting the target text information into the text feature extraction model for text modal feature extraction to obtain reference text features; inputting the target image information into the image feature extraction model for image modal feature extraction to obtain reference image features.

[0059] The feature extraction layer includes: a text feature extraction model and an image feature extraction model. The text feature extraction model can be used to extract text features from text information. The image feature extraction model can be used to extract image features from image information. The text feature extraction model can be a BERT model. The image feature extraction model can be a Deep Residual Network (ResNet) model.

[0060] Specifically, the present invention extracts features of the target text information based on the text vectorization representation method of BERT. Unlike traditional methods, this method pre-trains the text with the help of a deep neural network to obtain a text representation vector for each word, that is, a reference text feature. For the target image information, the present invention uses the ResNet model to extract image features. In traditional deep convolutional neural networks, the more network layers there are, the better the performance of the model tends to be. However, when the number of network layers increases to a certain extent, the performance of the model begins to decline. This is because the gradient in the deep neural network can easily become very small, making the training process very difficult. This phenomenon is called the "vanishing gradient" problem. The ResNet model solves the gradient vanishing problem by introducing residual blocks, thereby accurately obtaining reference image features.

[0061] On the basis of the above technical solution, "inputting the target text information into the text feature extraction model to extract text modal features to obtain reference text features" may include: inputting the target text information into the text feature extraction model, linearly mapping the target text information to obtain a first query vector, a first key vector and a first value vector; determining an attention score based on the first query vector, the first key vector and the first value vector; and performing weighted summation of the first value vector based on the attention score to determine the reference text features.

[0062] Among them, the query vector can represent the information that the current position is concerned about. In the self-attention mechanism, the query vector is used to calculate the degree of association with other elements in the input sequence, and it represents the element or information we want to focus on. The query vector is usually obtained by a linear transformation of the input data. For example, in a deep learning model (Transformer) based on the self-attention mechanism, this linear transformation is a fully connected layer, usually used with an activation function such as ReLU. The key vector can represent information at other positions. The key vector is used to calculate the correlation between the query vector and other elements in the input sequence. It can also be regarded as a representation that tells the model which input elements should receive more attention when calculating self-attention. The key vector is usually also obtained by a linear transformation of the input data and is calculated in the same way as the query vector. The value vector can represent the actual content of other positions. The value vector contains the information we want to assign to the query vector. In the self-attention mechanism, the query vector is dot-producted with the key vector of each position to obtain a scalar score that represents the relevance or importance of the current position to that position. Then, this attention score is cross-multiplied with the value vector of the corresponding position, and the sum is obtained to get the self-attention result of the current element. The attention score can be understood as the attention weight. The first query vector is the query vector corresponding to each element (vocabulary) in the target text information. The first key vector is the key vector corresponding to each element (word) in the target text information. The first value vector is the value vector corresponding to each element (word) in the target text information.

[0063] Specifically, in BERT, WordPiece embedding is first used to encode the text in the target text information, and then a multi-layer Transformer encoder is used to convert the text into a hidden layer representation. Each Transformer encoder consists of multiple self-attention mechanisms and a forward neural network. Among them, the self-attention mechanism can learn the relationship between each word and other words in a sentence, and it can automatically capture the dependencies and grammatical structures between words. Specifically, for the input word sequence, it first obtains the first query vector, the first key vector, and the first value vector through three linear mappings, calculates the attention score, and then performs a weighted summation on the value vector to obtain the final output vector, that is, the reference text feature. The above process can be expressed by the following formula.

[0064]

[0065] in, denote the first query vector, the first key vector, and the first value vector, respectively. n denotes the length of the input sequence. d krepresents the dimension of the first key vector, d v represents the dimension of the first value vector. Softmax(·) represents normalization of the result. Represents the attention score between values, and the n×n matrix represents the similarity between each word and other words.

[0066] It should be noted that in order for BERT to encode the entire sentence, it is also necessary to use positional encoding to encode the position of each word in the input sequence. Specifically, BERT uses a combination of sine and cosine functions for positional encoding, which can encode words at different positions in the sequence into different vectors, thereby solving the problem of different occurrences of words at the same position at different positions. Figure 3 An example diagram of a visualization of the model input representation is given. Figure 3 The reference text features generated after BERT training retain the word structure while also containing semantic information, which can help text parsing tasks to a certain extent. Specifically, the same word may have different meanings in different contexts, and the BERT model can distinguish these semantic differences in a fine-grained manner. The architecture of BERT makes it a powerful feature extractor. In the embodiment of the present invention, BERT is used to extract text features to obtain the text representation of the last hidden layer T = {E cls ,E1,E2,…,E n ,E SEP}, where n t represents the sequence length of the hidden layer, d t Indicates the dimension of the hidden layer. In BERT base Set to 768 in .

[0067] Based on the above technical solution, "inputting the target image information into the image feature extraction model to perform image modal feature extraction to obtain reference image features" may include: inputting the target image information into the image feature extraction model, performing image modal feature extraction on the target image information based on multiple stacked residual blocks to obtain reference image features.

[0068] Among them, the residual block is a cross-layer connection structure; the residual block includes two convolutional layers. The cross-layer connection structure can retain the information of the input features and avoid losing important features after multiple layers of nonlinear transformation. Figure 4 A structural example of a core residual block is given. Figure 4 , in the residual block, assuming the input is x, then the output y after the residual block can be expressed as follows:

[0069] y=F(x)+x

[0070] Among them, F(x) can be expressed as follows:

[0071] F(x)=f(x,W)

[0072] Among them, f(·) represents the mapping function of the residual block, and W represents the weight function of the residual block. In the implementation of the residual block, f(·) usually consists of two convolutional layers, and the process is expressed as follows

[0073] f(x)=W2σ(W1x)

[0074] Among them, σ(·) represents the activation function, W1 and W2 represent the weight parameters in convolution layer 1 and convolution layer 2 respectively. In this way, the input x can be directly passed to the subsequent layers through cross-layer connections, thus solving the problem of gradient disappearance.

[0075] Specifically, ResNet can extract image features by stacking multiple residual blocks. Assuming the input image is X, the output of the i-th residual block is Y i , then the process of ResNet extracting image features can be expressed as follows:

[0076] Y0=X;

[0077] Y i =Y i-1 +W i,2 σ(W i,1 ,Y i-1 );

[0078] Where W i,1 and W i,2 Respectively represent the weight parameters of the first and second convolutional layers in the ith residual block. By stacking multiple residual blocks, ResNet can extract image features. In the embodiment of the invention, ResNet152 is used to extract image features to obtain the image feature map I′ of the last output layer, as shown below;

[0079]

[0080] Among them, I′ represents the output of the last layer after ResNet, p×p×d image is the size of the feature map, in the specific implementation p = 7, d image =2048.

[0081] S230, inputting the reference image features and the reference text features into the joint coding layer for multimodal feature crossover to obtain image-text crossover features.

[0082] The image-text cross feature may refer to a feature obtained by complementing a reference image feature and a reference text feature.

[0083] Specifically, the reference image features and the reference text features are input into the joint coding layer, and one feature dimension is used as a reference, and the other feature is converted into the same dimension as the reference dimension to obtain the dimension-converted feature. For example, the reference image feature is converted into the same dimension as the reference text feature dimension to obtain the dimension-converted image feature. The reference text feature and the dimension-converted image feature are feature-fused to obtain the image-text cross feature.

[0084] On the basis of the above technical scheme, "inputting reference image features and reference text features into the joint coding layer for multimodal feature crossover to obtain image-text crossover features" may include: inputting reference image features and reference text features into the joint coding layer, performing dimensionality transformation on the reference image features to obtain candidate image features with the same dimension as the reference text features, and mapping the candidate image features to the feature space corresponding to the reference text features to obtain candidate image features; determining a second query vector, a second key vector and a second value vector based on the candidate image features and the reference text features, and determining a similarity score based on the second query vector, the second key vector and the second value vector; determining intermediate layer features based on the similarity score and the second value vector, and obtaining image-text crossover features by normalizing and residually connecting the intermediate layer features.

[0085] Specifically, the reference text feature T = {E cls ,E1,E2,…,E n ,E SEP}, and reference image features The two features are input into the joint encoding layer output to obtain the joint representation of image and text. However, since the image and text features belong to different feature spaces, the image features need to be processed before input. First, the dimension transformation is performed to obtain the candidate image features. The formula is shown below.

[0086]

[0087] Here, resize(·) represents a function for reconstructing two dimensions into one dimension, k=p×p, and k=49 in the specific implementation.

[0088] Reference image features It is also necessary to go through a layer of linear transformation to map the image features to the same space as the text features. The specific implementation is as follows.

[0089]

[0090] in, d t Represents the dimension of the reference text feature, which is similar to BERT baseThe same setting is 768 in the text. The result after dimensional transformation is in the same feature space as the reference text feature. ITE(·) stands for Image Transformer Encoder, which means that the transformed image feature will be output through a Transformer encoding layer to obtain the final candidate image feature representation.

[0091] After the above operations, the reference text features T and candidate image features I are obtained, and then they will pass through a joint encoding layer. The joint encoding layer uses the bimodal attention mechanism to obtain an attention-weighted bimodal representation, and then map it to another vector space through an intermediate layer. Finally, the intermediate representation is mapped to the original dimension, and a residual connection is added to obtain the final text-image representation (Text-Image Representations, TIR).

[0092] Specifically, for the reference text feature T and the candidate image feature I, the second query vector, the second key vector and the second value vector are obtained by the following formula.

[0093] Q=TW q +b q ;

[0094] K=IW k +b k ;

[0095] V=IW v +b v ;

[0096] The similarity score calculation formula after obtaining the second query vector Q, the second key vector K, and the second value vector V is as follows:

[0097]

[0098] After the above calculations, the formula for determining the intermediate layer features based on the similarity score and the second value vector is expressed as follows:

[0099] h = αV;

[0100] H=f(hW h +b h );

[0101] Finally, after a normalization layer and a residual connection, the image and text are jointly represented and the image and text cross features are obtained. The process is as follows:

[0102] TIR=LayerNorm(H+h)

[0103] In the above formula, W q, W k , W v , W h are the learnable weight parameters, b q 、b k 、b v 、b h are learnable bias parameters, d represents the feature dimension, I m is the mask representation of image features, f(·) represents the activation function, LayerNorm represents the normalization layer, and TIR represents the final image-text representation, i.e., the image-text cross feature.

[0104] S240: Input the image-text cross features and the reference text features into the residual coding layer for feature fusion to obtain target image-text features corresponding to the target image-text information.

[0105] Specifically, Figure 5 A structural example diagram of a multimodal feature extraction sub-model is given. Figure 5 , the multimodal feature extraction sub-model includes: feature extraction layer, joint encoding layer and residual encoding layer. In the model, the joint encoding layer outputs the final image-text representation, that is, the image-text cross feature, and then performs a residual connection with the reference text feature, and then passes through a residual encoding layer to encode the multimodal feature, that is, the target image-text feature. The residual encoding layer is similar to the joint encoding layer in structure, but the difference is that the self-attention mechanism is used to capture the dependency between different positions in the sequence, so that the representation of each element can fully utilize the information of the entire input sequence, thereby improving the representation and generalization capabilities of the model.

[0106] On the basis of the above technical solution, "inputting the image-text cross features and the reference text features into the residual coding layer for feature fusion to obtain the target image-text features corresponding to the target image-text information" may include: inputting the image-text cross features and the reference text features into the residual coding layer for feature connection to obtain the reference image-text features, and performing query vector calculation on the reference image-text features to obtain a third query vector; performing correlation calculation based on the third query vector and the reference image-text features to obtain the correlation weight, and performing weighted averaging processing based on the correlation weight and the reference image-text features to obtain the target image-text features corresponding to the target image-text information.

[0107] The reference graphic feature may refer to a graphic feature representation after feature connection.

[0108] Specifically, in the self-attention mechanism, each element of the sequence updates its representation by the weighted average of other elements, where the weight is determined by the similarity of each element to the third query vector. The third query vector is obtained by linearly transforming each element in the sequence. Specifically, the calculation process of the self-attention mechanism includes the following three steps: Calculation of the query vector: A query vector is obtained by linearly transforming each element in the input sequence. Calculation of the association degree: The similarity between the query vector and all elements in the input sequence is calculated to obtain the association weight. Commonly used similarity calculation methods include dot product and scaled dot product. Calculation of weighted average: The elements in the input sequence are weighted averaged using the association weight to obtain the output representation of the self-attention mechanism.

[0109] Specifically, the output of the joint coding layer (image-text cross feature) is firstly connected with the reference text feature, and then sent to the residual coding layer for encoding. The feature connection formula is as follows:

[0110] R′=contact(TIP,T)={r1,r2,…,r l}

[0111] Among them, R′ represents the feature representation after feature connection, and l represents the length of R′.

[0112] The calculation process of the self-attention mechanism in the multimodal encoder is as follows:

[0113]

[0114] Among them, W i,1 , W i,2 , W R are the learnable weight parameters, b i,1 , b i,2 , b R are learnable bias parameters, f(·) represents the activation function, α i is the normalized weight, and R is the final multimodal coding representation, that is, the target image and text features. The multimodal coding representation R is obtained and used in the multimodal emotion prediction task after normalization, so as to achieve richer emotion capture. The model proposed in the embodiment of the present invention uses a bimodal attention mechanism to extract the image-text cross-features between images and texts, and then reconnects the image-text cross-features with the reference text features, and automatically captures the relationship between sequences through the self-attention mechanism. Its advantage is that it utilizes the cross-information between images and texts, thereby obtaining richer emotional expressions.

[0115] S250: Determine the target image-text emotion type corresponding to the target user based on the output of the target image-text emotion prediction model.

[0116] The technical solution of the embodiment of the present invention, the multimodal feature extraction submodel includes: a feature extraction layer, a joint coding layer and a residual coding layer. By inputting the target image and text information into the feature extraction layer for single-modal feature extraction, reference image features and reference text features are obtained, so that accurate feature extraction is performed for each modality; the reference image features and reference text features are input into the joint coding layer for multimodal feature crossover to obtain image-text crossover features, so as to achieve complementarity between multimodal features; the image-text crossover features and the reference text features are input into the residual coding layer for feature fusion to obtain the target image-text features corresponding to the target image and text information, so as to obtain fusion features that can more comprehensively indicate the target user's emotions based on the features after fusion of multimodal features, and then the user's emotion type is comprehensively predicted through the fusion features, so as to further improve the accuracy of user emotion type prediction.

[0117] The following is an embodiment of a device for determining the emotion type of a picture and text provided by an embodiment of the present invention. The device and the method for determining the emotion type of a picture and text in the above-mentioned embodiments belong to the same inventive concept. For details not fully described in the embodiment of the device for determining the emotion type of a picture and text, please refer to the embodiment of the method for determining the emotion type of a picture and text mentioned above.

[0118] Embodiment 3

[0119] Figure 6 This is a schematic diagram of the structure of a device for determining the emotion type of a picture and text provided in Embodiment 3 of the present invention. Figure 6 As shown, the device includes: a target image and text information determination module 610, an image and text emotion prediction module 620 and a target image and text emotion type determination module 630.

[0120] Among them, the target image and text information determination module 610 is used to obtain the image and text information to be analyzed corresponding to the target user, and perform image and text preprocessing on the image and text information to be analyzed to determine the target image and text information corresponding to the target user; the target image and text information includes: target image information and target text information; the image and text emotion prediction module 620 is used to input the target image and text information into the target image and text emotion prediction model for feature extraction, feature cross-linking, feature fusion and emotion type prediction; the target image and text emotion type determination module 630 is used to determine the target image and text emotion type corresponding to the target user based on the output of the target image and text emotion prediction model.

[0121] The technical solution of the embodiment of the present invention obtains the to-be-analyzed graphic and text information corresponding to the target user, and performs graphic and text preprocessing on the to-be-analyzed graphic and text information to determine the target graphic and text information corresponding to the target user; the target graphic and text information includes: target image information and target text information; the target graphic and text information is input into a target graphic and text emotion prediction model for feature extraction, feature cross-linking, feature fusion and emotion type prediction; based on the output of the target graphic and text emotion prediction model, the target graphic and text emotion type corresponding to the target user is determined, thereby accurately and conveniently determining the target graphic and text emotion type, and then realizing the comprehensive prediction of the user emotion type based on multimodal information, thereby improving the efficiency and accuracy of the graphic and text emotion type determination.

[0122] Based on the above technical solution, the target image and text emotion prediction model includes: a multimodal feature extraction sub-model and an emotion type prediction sub-model;

[0123] The picture and text emotion prediction module 620 may include:

[0124] The target image-text feature determination submodule is used to input the target image-text information into the multimodal feature extraction submodel to perform single-modal feature extraction, multimodal feature crossover and multimodal feature fusion to obtain the target image-text features corresponding to the target image-text information;

[0125] The target prediction result determination submodule is used to input the target image and text features into the emotion type prediction submodel to perform emotion type prediction and obtain the target prediction result corresponding to the target image and text information.

[0126] Based on the above technical solution, the multimodal feature extraction sub-model includes: a feature extraction layer, a joint coding layer and a residual coding layer;

[0127] The target image and text feature determination submodule may include:

[0128] A reference feature determination unit is used to input the target image and text information into the feature extraction layer for single-modal feature extraction to obtain reference image features and reference text features;

[0129] An image-text cross feature determination unit is used to input reference image features and reference text features into a joint coding layer for multimodal feature crossover to obtain image-text cross features;

[0130] The target image-text feature determination unit is used to input the image-text cross feature and the reference text feature into the residual coding layer for feature fusion to obtain the target image-text feature corresponding to the target image-text information.

[0131] Based on the above technical solution, the feature extraction layer includes: a text feature extraction model and an image feature extraction model;

[0132] The reference feature determination unit may include:

[0133] A reference text feature determination subunit is used to input the target text information into the text feature extraction model to extract text modality features and obtain reference text features;

[0134] The reference image feature determination subunit is used to input the target image information into the image feature extraction model to extract the image modality features and obtain the reference image features.

[0135] Based on the above technical solution, the reference text feature determination subunit is specifically used to: input the target text information into the text feature extraction model, linearly map the target text information to obtain a first query vector, a first key vector and a first value vector; determine the attention score based on the first query vector, the first key vector and the first value vector; perform weighted summation of the first value vector based on the attention score to determine the reference text feature.

[0136] On the basis of the above technical scheme, the image-text cross-feature determination unit is specifically used to: input the reference image features and the reference text features into the joint coding layer, perform dimensional transformation on the reference image features to obtain candidate image features with the same dimension as the reference text features, and map the candidate image features to the feature space corresponding to the reference text features to obtain candidate image features; determine the second query vector, the second key vector and the second value vector based on the candidate image features and the reference text features, and determine the similarity score based on the second query vector, the second key vector and the second value vector; determine the intermediate layer features based on the similarity score and the second value vector, and obtain the image-text cross-features by normalizing and residually connecting the intermediate layer features.

[0137] On the basis of the above technical scheme, the target image and text feature determination unit is specifically used to: input the image and text cross features and the reference text features into the residual coding layer for feature connection to obtain the reference image and text features, and perform query vector calculation on the reference image and text features to obtain a third query vector; perform correlation calculation based on the third query vector and the reference image and text features to obtain the correlation weight, and perform weighted averaging processing based on the correlation weight and the reference image and text features to obtain the target image and text features corresponding to the target image and text information.

[0138] The apparatus for determining the emotion type of a picture and text provided in the embodiment of the present invention can execute the method for determining the emotion type of a picture and text provided in any embodiment of the present invention, and has the corresponding functional modules and beneficial effects for executing the method for determining the emotion type of a picture and text.

[0139] It is worth noting that in the above-mentioned embodiment of determining the emotional type of pictures and texts, the various units and modules included are only divided according to functional logic, but are not limited to the above-mentioned division, as long as the corresponding functions can be achieved; in addition, the specific names of the functional units are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of the present invention.

[0140] Embodiment 4

[0141] Figure 7 A schematic diagram of the structure of an electronic device 10 that can be used to implement an embodiment of the present invention is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices (such as helmets, glasses, watches, etc.) and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present invention described and / or required herein.

[0142] like Figure 7 As shown, the electronic device 10 includes at least one processor 11, and a memory connected to the at least one processor 11, such as a read-only memory (ROM) 12, a random access memory (RAM) 13, etc., wherein the memory stores a computer program that can be executed by at least one processor, and the processor 11 can perform various appropriate actions and processes according to the computer program stored in the read-only memory (ROM) 12 or the computer program loaded from the storage unit 18 to the random access memory (RAM) 13. In the RAM 13, various programs and data required for the operation of the electronic device 10 can also be stored. The processor 11, the ROM 12, and the RAM 13 are connected to each other through a bus 14. An input / output (I / O) interface 15 is also connected to the bus 14.

[0143] A number of components in the electronic device 10 are connected to the I / O interface 15, including: an input unit 16, such as a keyboard, a mouse, etc.; an output unit 17, such as various types of displays, speakers, etc.; a storage unit 18, such as a disk, an optical disk, etc.; and a communication unit 19, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 19 allows the electronic device 10 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0144] The processor 11 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the processor 11 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any appropriate processor, controller, microcontroller, etc. The processor 11 executes the various methods and processes described above, such as the method for determining the image and text emotion type.

[0145] In some embodiments, the method for determining the emotion type of the image and text may be implemented as a computer program, which is tangibly contained in a computer-readable storage medium, such as a storage unit 18. In some embodiments, part or all of the computer program may be loaded and / or installed on the electronic device 10 via the ROM 12 and / or the communication unit 19. When the computer program is loaded into the RAM 13 and executed by the processor 11, one or more steps of the method for determining the emotion type of the image and text described above may be performed. Alternatively, in other embodiments, the processor 11 may be configured to execute the method for determining the emotion type of the image and text by any other appropriate means (e.g., by means of firmware).

[0146] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0147] Computer programs for implementing the methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that when the computer program is executed by the processor, the functions / operations specified in the flow chart and / or block diagram are implemented. The computer program may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0148] In the context of the present invention, a computer-readable storage medium may be a tangible medium that may contain or store a computer program for use by or in combination with an instruction execution system, device or equipment. A computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices or equipment, or any suitable combination of the foregoing. Alternatively, a computer-readable storage medium may be a machine-readable signal medium. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0149] To provide interaction with a user, the systems and techniques described herein may be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and a pointing device (e.g., a mouse or trackball) through which the user can provide input to the electronic device. Other types of devices may also be used to provide interaction with the user; for example, the feedback provided to the user may be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user may be received in any form (including acoustic input, voice input, or tactile input).

[0150] The systems and techniques described herein may be implemented in a computing system that includes backend components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes frontend components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such backend components, middleware components, or frontend components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), a blockchain network, and the Internet.

[0151] A computing system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The client and server relationship is generated by computer programs running on the corresponding computers and having a client-server relationship with each other. The server may be a cloud server, also known as a cloud computing server or cloud host, which is a host product in the cloud computing service system to solve the defects of difficult management and weak business scalability in traditional physical hosts and VPS services.

[0152] An embodiment of the present invention further provides a computer program product, including a computer program, which, when executed by a processor, implements the method for determining the emotion type of a picture and text provided in any embodiment of the present application.

[0153] In the process of implementation, the computer program product can be written in one or more programming languages ​​or a combination thereof to perform the computer program code of the present invention, and the programming language includes an object-oriented programming language, such as Java, Smalltalk, C++, and also includes a conventional procedural programming language, such as "C" language or similar programming language. The program code can be executed completely on the user's computer, partially on the user's computer, as an independent software package, partially on the user's computer and partially on the remote computer, or completely on the remote computer or server. In the case of a remote computer, the remote computer can be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computer (for example, using an Internet service provider to connect through the Internet). The program product and the method for determining the graphic emotion type disclosed in each embodiment of the present application belong to the same inventive concept, so it will not be repeated here.

[0154] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps described in the present invention can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solution of the present invention can be achieved, and this document does not limit this.

[0155] The above specific implementations do not constitute a limitation on the protection scope of the present invention. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A method for determining the emotional type of a picture and text, characterized in that: include: Acquire the to-be-analyzed graphic information corresponding to the target user, and perform graphic preprocessing on the to-be-analyzed graphic information to determine the target graphic information corresponding to the target user; The target graphic information includes: target image information and target text information; Inputting the target image-text information into the target image-text emotion prediction model for feature extraction, feature cross-talk, feature fusion and emotion type prediction; Based on the output of the target image-text emotion prediction model, the target image-text emotion type corresponding to the target user is determined.

2. The method according to claim 1, characterized in that The target image-text emotion prediction model includes: a multimodal feature extraction sub-model and an emotion type prediction sub-model; The step of inputting the target image-text information into the target image-text emotion prediction model to perform single-modal feature extraction, multi-modal feature crossover, feature fusion, and emotion type prediction includes: Inputting the target graphic information into a multimodal feature extraction sub-model to perform single-modal feature extraction, multimodal feature crossover and multimodal feature fusion to obtain target graphic features corresponding to the target graphic information; The target image-text features are input into the emotion type prediction sub-model to perform emotion type prediction, and a target prediction result corresponding to the target image-text information is obtained.

3. The method according to claim 2, characterized in that The multimodal feature extraction sub-model includes: a feature extraction layer, a joint coding layer and a residual coding layer; The step of inputting the target graphic information into a multimodal feature extraction sub-model to perform single-modal feature extraction, multimodal feature crossover, and multimodal feature fusion to obtain target graphic features corresponding to the target graphic information includes: Inputting the target image and text information into a feature extraction layer for single-modal feature extraction to obtain reference image features and reference text features; Inputting the reference image features and the reference text features into a joint coding layer for multimodal feature crossover to obtain image-text crossover features; The image-text cross feature and the reference text feature are input into the residual coding layer for feature fusion to obtain the target image-text feature corresponding to the target image-text information.

4. The method according to claim 3, characterized in that The feature extraction layer includes: a text feature extraction model and an image feature extraction model; The step of inputting the target image and text information into a feature extraction layer to extract single-modal features to obtain reference image features and reference text features includes: Input the target text information into the text feature extraction model to extract text modality features and obtain reference text features; The target image information is input into the image feature extraction model to extract the image modality features and obtain the reference image features.

5. The method according to claim 4, characterized in that The step of inputting the target text information into the text feature extraction model to extract text modality features and obtain reference text features includes: Inputting the target text information into a text feature extraction model, and performing linear mapping on the target text information to obtain a first query vector, a first key vector, and a first value vector; determining an attention score based on the first query vector, the first key vector, and the first value vector; A weighted sum is performed on the first value vector based on the attention score to determine a reference text feature.

6. The method according to claim 3, characterized in that The step of inputting the reference image features and the reference text features into a joint coding layer for multimodal feature crossover to obtain image-text crossover features includes: Inputting the reference image features and the reference text features into a joint coding layer, performing dimension transformation on the reference image features to obtain candidate image features having the same dimension as the reference text features, and mapping the candidate image features to a feature space corresponding to the reference text features to obtain candidate image features; Determining a second query vector, a second key vector, and a second value vector based on the candidate image features and the reference text features, and determining a similarity score based on the second query vector, the second key vector, and the second value vector; Based on the similarity score and the second value vector, the intermediate layer features are determined, and the image-text cross features are obtained by normalizing and residual connecting the intermediate layer features.

7. The method according to claim 3, characterized in that The step of inputting the image-text cross feature and the reference text feature into the residual coding layer for feature fusion to obtain the target image-text feature corresponding to the target image-text information includes: Inputting the image-text cross feature and the reference text feature into the residual coding layer for feature connection to obtain a reference image-text feature, and performing query vector calculation on the reference image-text feature to obtain a third query vector; A correlation degree is calculated based on the third query vector and the reference image and text features to obtain a correlation degree weight, and a weighted average process is performed based on the correlation degree weight and the reference image and text features to obtain a target image and text feature corresponding to the target image and text information.

8. A device for determining the emotional type of a picture or text, characterized in that: The device comprises: The target graphic information determination module is used to obtain the graphic information to be analyzed corresponding to the target user, and perform graphic preprocessing on the graphic information to be analyzed to determine the target graphic information corresponding to the target user; the target graphic information includes: target image information and target text information; The image-text emotion prediction module is used to input the target image-text information into the target image-text emotion prediction model for feature extraction, feature cross-talk, feature fusion and emotion type prediction; The target image-text emotion type determination module is used to determine the target image-text emotion type corresponding to the target user based on the output of the target image-text emotion prediction model.

9. An electronic device, characterized in that: The electronic device comprises: one or more processors; A memory for storing one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors implement the method for determining the image and text emotion type as described in any one of claims 1-7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the method for determining the image-text emotion type as described in any one of claims 1-7 is implemented.

Images