Picture-text-oriented coarse-grained sentiment analysis method and device

Abstract

The application discloses a kind of picture text-oriented coarse-grained sentiment analysis method and device, method includes: identifying picture obtains text information and text structured information in picture;Fusion is carried out to preset label information and text information, respectively input to main task preset model and auxiliary task preset model to obtain corresponding main task text vector and auxiliary task text vector;Wherein, preset label information includes sentiment classification label and emotion recognition label;Main task preset model is used for coarse-grained sentiment analysis;Auxiliary task preset model is used for fine-grained sentiment analysis;Text structured information is converted to obtain picture vector, and picture vector and main task text vector are spliced, to obtain main task picture text vector;Using auxiliary task text vector and main task picture text vector, main task preset model is trained and adjusted, and the output result of the coarse-grained sentiment analysis of main task preset model is obtained.

Description

TECHNICAL FIELD

[0001] The embodiment of the present application relates to the technical field of natural language processing and image field, and particularly relates to a coarse-grained sentiment analysis method and device for picture text. BACKGROUND

[0002] Among many research directions of NLP (Natural Language Processing), sentiment analysis is a popular research topic. The sentiment analysis is to analyze and process subjective text with emotional color, and automatically judge the emotional tendency of the text. The sentiment analysis is an indispensable step in many natural language processing applications, such as evaluation monitoring, reputation detection, satisfaction analysis and other different fields. Sentiment includes coarse-grained (such as positive, neutral and negative sentiment) and fine-grained (happy, sad, angry, etc.), the coarse-grained sentiment analysis is called sentiment classification, and the fine-grained sentiment analysis is called emotion recognition.

[0003] The existing technology is different from the text-based sentiment analysis in that the text is not the traditional social media text, but the text information obtained from the picture, such as the text information in the picture extracted based on the OCR (Optical Character Recognition) recognition, and the sentiment analysis is performed on the recognized text information. However, the analysis result accuracy of the current method for performing sentiment analysis based on the text information is low. SUMMARY

[0004] In view of the above problems, the embodiment of the present application is proposed to provide a coarse-grained sentiment analysis method and device for picture text to overcome the above problems or at least partially solve the above problems.

[0005] According to one aspect of the embodiment of the present application, a coarse-grained sentiment analysis method for picture text is provided, the method comprising:

[0006] identifying the text information and the text structured information in the picture;

[0007] fusing the preset label information and the text information, and inputting the same into a main task preset model and an auxiliary task preset model to obtain a main task text vector and an auxiliary task text vector respectively; wherein, the preset label information comprises a sentiment classification label and an emotion recognition label; the main task preset model is used for coarse-grained sentiment analysis; and the auxiliary task preset model is used for fine-grained sentiment analysis;

[0008] converting the text structured information to obtain a picture vector, and splicing the picture vector and the main task text vector to obtain a main task picture text vector;

[0009] The similarity constraint module is adapted to train and adjust the preset model of the main task by using the preset text vector of the auxiliary task and the preset picture text vector of the main task, to obtain an output result of coarse-grained sentiment analysis of the preset model of the main task.

[0010] According to another aspect of the embodiment of the present application, there is provided a coarse-grained sentiment analysis device for picture text, which comprises:

[0011] The recognition module is adapted to recognize the picture to obtain text information and text structured information in the picture.

[0012] The fusion module is adapted to fuse preset label information and text information, and input the same into the preset model of the main task and the preset model of the auxiliary task respectively to obtain a corresponding preset text vector of the main task and a preset text vector of the auxiliary task; wherein, the preset label information comprises sentiment classification labels and emotion recognition labels; the preset model of the main task is used for coarse-grained sentiment analysis; and the preset model of the auxiliary task is used for fine-grained sentiment analysis.

[0013] The splicing module is adapted to convert the text structured information to obtain a picture vector, and splice the picture vector and the preset text vector of the main task to obtain a preset picture text vector of the main task.

[0014] The similarity constraint module is adapted to train and adjust the preset model of the main task by using the preset text vector of the auxiliary task and the preset picture text vector of the main task, to obtain an output result of coarse-grained sentiment analysis of the preset model of the main task.

[0015] According to still another aspect of the embodiment of the present application, there is provided a computing device, which comprises a processor, a memory, a communication interface and a communication bus, wherein the processor, the memory and the communication interface complete communication with each other through the communication bus.

[0016] The memory is used for storing at least one executable instruction, and the executable instruction makes the processor execute the operation corresponding to the above-mentioned coarse-grained sentiment analysis method for picture text.

[0017] According to still another aspect of the embodiment of the present application, there is provided a computer storage medium, wherein the storage medium stores at least one executable instruction, and the executable instruction makes the processor execute the operation corresponding to the above-mentioned coarse-grained sentiment analysis method for picture text.

[0018] The picture text-oriented coarse-grained sentiment analysis method and device provided by the embodiment of the present application make full use of the text structural information in the picture, such as text, to convert the text structural information into a feature vector and splice the feature vector with a main task text vector based on text information, expand the original text vector, help the preset model to mine the internal relationship, and improve the model training effect. The label information and the text information are fused to further mine the internal relationship between the text information and the label information, help to improve the analysis effect, and use the auxiliary task preset model to adjust the main task preset model by using the joint learning, so as to improve the sentiment analysis accuracy.

[0019] The above description is only a summary of the technical solutions of the embodiments of the present application, in order to more clearly understand the technical means of the embodiments of the present application, and can be implemented according to the content of the specification, and in order to make the above and other purposes, features and advantages of the embodiments of the present application more obvious and easy to understand, the following specific embodiments of the present application are described. BRIEF DESCRIPTION OF DRAWINGS

[0020] By reading the detailed description of the preferred embodiments below, various other advantages and benefits will become clear to those of ordinary skill in the art. The drawings are only for the purpose of illustrating the preferred embodiments and are not considered as limiting the embodiments of the present application. Moreover, the same reference symbols are used to represent the same parts throughout the drawings. In the drawings:

[0021] Figure 1 A flowchart of a picture text-oriented coarse-grained sentiment analysis method according to an embodiment of the present application is shown;

[0022] Figure 2 A schematic diagram of a sentiment analysis learning unified framework is shown;

[0023] Figure 3 A schematic diagram of each layer of the Bert model is shown;

[0024] Figure 4 A schematic diagram of a preset punctuation symbol randomly inserted into text information is shown;

[0025] Figure 5 A structural schematic diagram of a picture text-oriented coarse-grained sentiment analysis device according to an embodiment of the present application is shown;

[0026] Figure 6 A structural schematic diagram of a computing device according to an embodiment of the present application is shown. DETAILED DESCRIPTION

[0027] Exemplary embodiments of the present application will be described herein below with reference to the accompanying drawings. While exemplary embodiments of the present application are shown in the drawings, it is understood that the present application can be embodied in various forms and should not be limited by the embodiments set forth herein. Rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the scope of the application to those skilled in the art.

[0028] Figure 1 A flow chart of a picture-text oriented coarse-grained sentiment analysis method according to an embodiment of the present application is shown in FIG. 1, which includes the following steps: Figure 1

[0029] In step S101, text information and text structured information in the picture are identified.

[0030] In the prior art, when a picture is used, the picture is directly converted into a picture vector and a text vector, which are simply fused, or only the identified text information is used, so that the effective structured information in the picture, such as text position, font size, font color, etc., is not fully utilized.

[0031] To address the above problems, in the present embodiment, for a picture, text information contained in the picture is identified and extracted by using recognition calculation such as OCR. In addition to the text information, text structured information in the picture is also obtained. The text structured information includes, for example, text position information in the picture, text font size information, text color information, etc. It is considered that different position information of different texts in the picture also reflects different attention degrees of the different texts, for example, the importance of different texts set in the middle of the picture and the side of the picture is different, and the representative sentiment degree is also different. The size of the text font and the color of the text also carry the importance of the text, for example, the larger the text font, the darker the color, which indicates that the user is more likely to pay attention to the text content of this part, i.e., the text content of this part is more important, and the carried sentiment is heavier.

[0032] Specifically, when obtaining the text structured information, for the position information of the text in the picture, the position of the text in the picture can be detected based on a picture field target detection framework SSD (Single Shot Multibox Detector), and the coordinate information of the text paragraph is obtained, specifically, the coordinate information of two diagonal corners of the text paragraph, such as the coordinate information of the upper left corner and the lower right corner, is obtained, and the coordinate information of the text paragraph is taken as the position information of the text in the picture, which is represented as d IOC = { <x1, y1>, <x2, y2>}. d IOC ​The position information of the text in the picture, wherein <x1, y1> and <x2, y2> are respectively the coordinate information of two diagonal corners of the text paragraph.

[0033] For the text font size information, according to the coordinate information of the text paragraph, the coordinate information of each character in the text paragraph is determined, if the text font is larger, the corresponding coordinate distance is larger, and vice versa, the coordinate distance is smaller, the text font size information is obtained according to the coordinate information of each character, the font size information of each text is represented according to the coordinate information of each character, such as w size i i i , i ∈ n}. Wherein, w size is the font size information of the text, w i is the font size information of the i-th character in the text paragraph, and n represents that the text paragraph contains n characters. <x i , y i > is the coordinate information of the i-th character. For each character, the coordinate value of any corner position of the character can be selected as the coordinate information of the character, which is not limited here, and the text font size information can be calculated according to the coordinate distance of each character.

[0034] For the text color information, HSV (Hue, Saturation, Value, hue (H), saturation (S), and brightness (V)) color space can be used to calculate the chroma and saturation of the whole text paragraph, and the variance of each character and the chroma and saturation of the whole text paragraph is calculated according to each character, and the variance is the text color information, represented as w hsv i , i ∈ n}, wherein w hsv is the text color information, var i is the chroma and saturation of the i-th character in the text paragraph, and n represents that the text paragraph contains n characters.

[0035] In step S102, the preset label information and the text information are fused and input into the main task preset model and the auxiliary task preset model respectively to obtain the corresponding main task text vector and auxiliary task text vector.

[0036] ​​​​The embodiment is based on a sentiment analysis unified framework (SAUF) with fusion of label and vector constraints to perform recognition, wherein a preset model used is taken as an example of a Bert (Bidirectional Encoder Representations from Transformers) model to complete picture text recognition, and other natural language processing models can also be used, which are not limited here.

[0037] The sentiment of the same text information can include coarse granularity and fine granularity, such as positive sentiment of coarse granularity and happy emotion of fine granularity. Fine-grained emotion recognition can further help determine coarse-grained sentiment classification. For example, by jointly learning the intermediate layers (such as the output of the hidden layer) of the shared emotion recognition model and the sentiment classification model, sentiment classification and emotion recognition are performed on the same text information, wherein the main task preset model outputs coarse-grained sentiment analysis, and the auxiliary task preset model outputs fine-grained sentiment analysis. Through joint learning, the auxiliary task preset model helps the main task preset model learn the internal relationship between the two different tasks, thereby improving the effect of sentiment classification of the main task preset model. In joint learning, although the shared intermediate layer can help improve the effect of the main task preset model through the auxiliary task preset model to a certain extent, the relationship between coarse-grained and fine-grained sentiment analysis needs to be more fully utilized, such as the relationship between the positive label (sentiment classification label) and the happy label (emotion recognition label). In the same text information, there is an internal relationship between the two different label information, and the label information can be further fused in the training process to improve the overall training effect of the SAUF framework. Specifically, the text information is fused with the preset label information, which includes sentiment classification labels and emotion recognition labels. For example, the sentiment classification labels include three category labels: positive, neutral, and negative. The emotion recognition labels include 5 labels: happy, thank, sad, angry, and complain. The above is an example, and the label information is not limited according to the implementation. Taking the text information "It rained today, I feel bad, and I feel a little down" as an example, the label information is abstract for the text information whether it is sentiment classification or emotion recognition. When using the preset model for training, the sentiment classification labels and the emotion recognition labels can be input together, such as "positive neutral negative happy thank sad angry complain + It rained today, I feel bad, and I feel a little down (text information)", so that the model can learn the internal relationship between "feeling bad" and "a little down" and "negative" and "sad" label information, and use the text information with added sentiment labels to train the mapping logic of the specific label information. Here, the label information fusion input is all the label information involved in sentiment classification and emotion recognition, not just the specific label corresponding to the current text information. Inputting the sentiment classification labels and the emotion recognition labels together with the text information also facilitates the constraint of the intermediate vector. The SAUF framework is shown in Figure 2 The dashed box shown in ① of the figure represents the label information. After fusing the label information with the text information, they are input into the main task preset model and the auxiliary task preset model, respectively. The preset model takes the Bert model as an example, that is, Figure 2The fusion information of the label information and the text information is input to the Bert layer of the main task and the Bert layer of the auxiliary task, and the corresponding main task text vector and auxiliary task text vector are obtained. The main task Bert layer outputs the main task text vector pointed by the arrow outside the picture vector in the dashed box shown in ②, and the auxiliary task Bert layer outputs the auxiliary task text vector marked as h main2 or h aux . Wherein, h main2 = h aux = BERT(T input ), T input is the fusion information of the label information and the text information, and the corresponding output auxiliary task text vector is obtained after the auxiliary task Bert layer.

[0038] The Bert layer of the main task and the Bert layer of the auxiliary task can use the same Bert model. The input of the Bert model mainly includes the input of the Embedding embedding layer such as three parts, such as Token Embedding (word embedding layer), SegmentEmbedding (classification embedding layer) and Position Embedding (position embedding layer). As shown in Figure 3 , it can be seen from InputTokens (input word) that the input includes label information and text information, the starting position is set as [CLS] mark, the ending position is set as [SEP] mark, and [SEP] mark is not set between the label information and the text information, to indicate that the label information and the text information are not natural sentences, and are inconsistent with the NSP (Next Sentence Prediction, next sentence prediction) of the Bert model. Among them, L1…L C represents the c label information corresponding to the text information, D1…D K is the input text information with a length of k text information, the label information and the text information are converted into fixed dimension vectors through Token Embedding (word embedding layer), SegmentEmbedding (classification embedding layer) learns classification, such as E A , Position Embedding (position embedding layer) learns the order position, such as E1, Self-AttentionEncoder (self-attention mechanism encoder), and outputs ClassPrediction (class prediction), that is, the text vector after combining the label information and the text information. The above Token Embedding (word embedding layer), SegmentEmbedding (classification embedding layer), Position Embedding (position embedding layer), Self-AttentionEncoder (self-attention mechanism encoder) can be set according to the implementation, which is not expanded here.

[0039] Furthermore, regarding the issue of limited textual information extracted from images, most methods rely on EDA (easy data augmentation) for data augmentation, such as synonym replacement, random insertion, random swapping, and random deletion. Other methods include back-translation for Chinese-English translation or concatenating the overall image vector with the text information. While these methods achieve some data augmentation, they inevitably alter the original text's grammatical and semantic information, damaging the original emotional and semantic information. This data augmentation may mislead model training and affect overall performance.

[0040] To address the aforementioned issues, this embodiment utilizes random insertion of punctuation marks to augment text information, resulting in multiple augmented text entries. Specifically, preset punctuation marks can be randomly inserted into the text information to effectively increase the amount of text information available for sentiment analysis, while minimizing modification to the original text sequence. For example... Figure 4 As shown, the arrow points in the direction of the text to indicate the location where punctuation marks will be randomly inserted. Punctuation marks include, for example, ... Figure 4 The symbols [“.”,“;”,“?”,“:”,“!”,“,”] represent separators, semicolons, question marks, colons, exclamation marks, commas, etc., and may also include other punctuation marks; no limitation is made here. When randomly inserting punctuation marks, a random value range is used, from the value 1 to a certain length of the text information, such as one-third of the length (rounding can be performed if not divisible). A random value is selected as the number of punctuation marks to insert. The insertion position can also be randomly set, and any punctuation mark is randomly selected for insertion. For example, if the text information is a sentence with a length of 12, the random value range is 1-4. A random value is selected as the number of punctuation marks to insert, ensuring that each piece of text contains punctuation marks, increasing its complexity without adding too many punctuation marks that could excessively interfere with the semantic information of the sentence, thus reducing the negative impact of excessive noise.

[0041] After randomly inserting preset punctuation marks into the text information, multiple expanded text information containing the preset punctuation marks can be obtained. It is necessary to select multiple expanded text information pieces with high similarity to the original text information to facilitate subsequent analysis using these expanded text information pieces along with the original semantic text information. The similarity between the expanded text information and the original text information can be calculated using the following formula:

[0042]

[0043] Where, x i and y irespectively represent vectors obtained after linear transformation operation such as Bert-whitening of two different pieces of text information, wherein the vector dimension is d. x and y correspond to the expanded text information and the text information respectively. cos(x, y) is the similarity of the expanded text information and the text information, and the similarity of each expanded text and the text information is calculated respectively, and the similarities are sorted from high to low, and the top ones, such as the top 5 expanded text information with high similarity, are selected and combined with the text information to obtain the expanded text information, which is used as the input text information together.

[0044] In step S103, the text structured information is converted to obtain a picture vector, and the picture vector is spliced with the main task text vector to obtain a main task picture text vector.

[0045] For the text structured information, a vector conversion model can be used to convert to obtain a picture vector. The vector conversion model can be selected according to the implementation, such as a convolutional neural network (CNN), a transformer network, etc., which is not limited here. For example, taking the transformer network as an example, h add =transformer(T input1 +T input2 +T input3 ). Wherein, T input1 represents the position information of the text in the picture (i.e. d IOC ), T input2 represents the text font size information (i.e. w size ), T input3 represents the text color information (i.e. w hsv ), and the output obtained after the transformer conversion is the picture vector h add .

[0046] The picture vector is spliced with the main task text vector to obtain a main task picture text vector, which is used for coarse-grained sentiment analysis of the main task, as shown in Figure 2 h main1 is the picture text vector, and h main1 =BERT(T input )+h add . Wherein, T input is the fusion of the label information and the text information, and the main task text vector is obtained after the main task Bert layer. The main task text vector is spliced with the picture vector h add to obtain the main task picture text vector h main1 .

[0047] Step S104: Using the text vectors of the auxiliary task and the image text vectors of the main task, train and adjust the preset model of the main task to obtain the output results of the coarse-grained sentiment analysis of the preset model of the main task.

[0048] The main task image text vector h main1 and auxiliary task text vector h main2 After performing vector concatenation, the input is fed into the hidden layer of the main task's preset model to obtain the hidden layer output of the main task's preset model. Among them, dense main This is the hidden layer (fully connected layer) of the main task's preset model, where ⊕ represents a vector concatenation operation. The auxiliary task's text vector is input into the hidden layer of the auxiliary task's preset model to obtain the output of the auxiliary task's preset model's hidden layer. This represents the hidden layer (fully connected layer) output of the auxiliary task's preset model. The similarity between the hidden layer outputs of the main task's preset model and the auxiliary task's preset model is calculated using a cosine function. For example... Figure 2 The dashed box shown in ③ indicates that the main and auxiliary task preset models incorporate label information when inputting text information, and share weights through the BERT layer. and The vectors should be as consistent as possible. To achieve this, the similarity score is increased. The similarity score is calculated using the cosine function, such as... By adjusting the loss function of the main task's preset model using similarity, we obtain... in, The preset cross-entropy loss function is set according to the implementation situation. The similarity between the hidden layer outputs of the main task preset model and the auxiliary task preset model is used to constrain the hidden layer outputs of the main task and auxiliary task preset models to be as similar as possible. It is important to note that the calculation for the main task is consistent with that for the auxiliary task. For the overall SAUF framework, the loss functions of the main task preset model and the auxiliary task preset model are weighted and linearly fused to obtain a common loss function for both models, i.e., the overall SAUF framework loss function. The main task preset model and the auxiliary task preset model are trained and adjusted based on this overall SAUF framework loss function, finally yielding the coarse-grained sentiment analysis output of the main task preset model. During the weighted linear fusion, for example, loss = λloss. main +(1-λ)loss aux Where λ is a preset weight parameter, set according to the implementation situation, and loss main The loss function of the model preset for the main task, lossaux presetting a loss function for the auxiliary task.

[0049] Further, the preset model in the overall framework can use, for example, RMSprop (Root Mean Square Rrop) as an optimization function to accelerate gradient descent. The matrix and vector parameters involved in the overall framework can be initialized based on parameters in the range of [0, 1], and the parameters r and c represent the rows and columns of the matrix, respectively. The specific settings are not limited here.

[0050] The coarse-grained sentiment analysis method for picture text provided by the embodiment of the present application makes full use of the text structured information such as the position information of the text in the picture, the text font size information, and the text color information in the picture, converts the text structured information into a feature vector, splices the feature vector with the output vector of the BERT layer based on the text information input, expands the original text vector, helps the preset model to mine the internal relationship, and improves the model training effect. Further, in view of the problem that the text information obtained from the picture is insufficient, the expanded text information is obtained based on the preset punctuation symbol random insertion mode, the expanded text information with high similarity to the original text information is selected for text expansion, the loss of sentiment semantic information is small, and the text expansion is simple and fast. The label information and the text information are fused, the internal relationship between the text information and the label information is further mined, the analysis effect is helped to improve, the similarity of the hidden layer outputs of the main task preset model and the auxiliary task preset model is used to constrain the vector distribution, so that the vector distribution is as consistent as possible, and the effect of improving the sentiment analysis accuracy is achieved.

[0051] Figure 5 The structure of the coarse-grained sentiment analysis device for picture text provided by the embodiment of the present application is shown. As shown in the figure, Figure 5 The device comprises:

[0052] The recognition module 510 is adapted to recognize the text information and the text structured information in the picture.

[0053] The fusion module 520 is adapted to fuse the preset label information and the text information, and input the preset label information and the text information into the main task preset model and the auxiliary task preset model respectively to obtain the main task text vector and the auxiliary task text vector. The preset label information comprises sentiment classification labels and emotion recognition labels. The main task preset model is used for coarse-grained sentiment analysis. The auxiliary task preset model is used for fine-grained sentiment analysis.

[0054] The splicing module 530 is adapted to convert the text structured information to obtain a picture vector, splice the picture vector with the main task text vector, and obtain a main task picture text vector.

[0055] ​The similarity constraint module 540 is adapted to train and adjust the preset model of the main task by using the auxiliary task text vector and the main task picture text vector, to obtain an output result of coarse-grained sentiment analysis of the preset model of the main task.

[0056] Optionally, the text structured information comprises at least one of position information of the text in the picture, text font size information, and / or text color information.

[0057] The recognition module 510 is further adapted to:

[0058] detect the position of the text in the picture, and obtain coordinate information of the text paragraph as the position information of the text in the picture;

[0059] determine coordinate information of each character according to the coordinate information of the text paragraph, and obtain the text font size information according to the coordinate information of each character;

[0060] obtain the chroma and saturation of the text paragraph, and calculate the variance of each character and the chroma and saturation of the text paragraph as the text color information.

[0061] Optionally, the device further comprises an expansion module 550 adapted to perform text expansion on the text information by using preset punctuation symbols, to obtain a plurality of text information after text expansion; and fuse the preset label information with the plurality of text information after text expansion.

[0062] Optionally, the expansion module 550 is further adapted to:

[0063] randomly insert the preset punctuation symbols into the text information to obtain a plurality of expanded text information;

[0064] calculate the similarity between the plurality of expanded text information and the text information;

[0065] sort the expanded text information according to the similarity from high to low, merge a preset number of expanded text information ranked first with the text information to obtain the plurality of text information after text expansion.

[0066] Optionally, the similarity constraint module 540 is further adapted to:

[0067] determine the hidden layer output of the preset model of the main task and the hidden layer output of the preset model of the auxiliary task based on the auxiliary task text vector and the main task picture text vector, and calculate the similarity between the hidden layer output of the preset model of the main task and the hidden layer output of the preset model of the auxiliary task;

[0068] adjust the loss function of the preset model of the main task by using the similarity, and train and adjust the preset model of the main task based on the loss function of the preset model of the main task, to obtain the output result of coarse-grained sentiment analysis of the preset model of the main task.

[0069] Optionally, the similarity constraint module 540 is further adapted to:

[0070] vector connection processing is performed on the main task picture text vector and the auxiliary task text vector, and the main task preset model is input into the hidden layer to obtain the hidden layer output of the main task preset model;

[0071] The auxiliary task text vector is input into the hidden layer of the auxiliary task preset model to obtain the hidden layer output of the auxiliary task preset model.

[0072] The similarity between the hidden layer output of the main task preset model and the hidden layer output of the auxiliary task preset model is calculated by using a cosine function.

[0073] Optionally, the similarity constraint module 540 is further adapted to:

[0074] The loss function of the main task preset model is obtained by using the similarity and a preset cross-entropy loss function.

[0075] The loss function of the main task preset model and the loss function of the auxiliary task preset model are linearly fused by weighting to obtain a common loss function of the main task preset model and the auxiliary task preset model, and the main task preset model and the auxiliary task preset model are trained and adjusted based on the common loss function to obtain the output result of the coarse-grained sentiment analysis of the main task preset model.

[0076] The above description of each module refers to the corresponding description in the method embodiment, and will not be repeated here.

[0077] The embodiment of the application also provides a non-volatile computer storage medium, which stores at least one executable instruction, and the executable instruction can execute the picture text-oriented coarse-grained sentiment analysis method in any method embodiment described above.

[0078] Figure 6 The structure schematic diagram of a computing device according to an embodiment of the application is shown, and the specific implementation of the computing device is not limited in the specific embodiment of the application.

[0079] As shown in Figure 6 , the computing device can include a processor 602, a communications interface 604, a memory 606, and a communications bus 608.

[0080] characterized in that:

[0081] The processor 602, the communications interface 604, and the memory 606 complete mutual communication through the communications bus 608.

[0082] The communication interface 604 is configured to communicate with network elements such as clients or other servers.

[0083] The processor 602 is configured to execute the program 610, and particularly, can execute the steps in the above-mentioned coarse-grained sentiment analysis method for image text.

[0084] Specifically, the program 610 can include program codes including computer operation instructions.

[0085] The processor 602 can be a central processing unit (CPU), or an application-specific integrated circuit (ASIC), or one or more integrated circuits configured to implement the embodiments of the present application. The one or more processors included in the computing device can be processors of the same type, such as one or more CPUs; or processors of different types, such as one or more CPUs and one or more ASICs.

[0086] The memory 606 is configured to store the program 610. The memory 606 can include a high-speed RAM memory, and can also include a non-volatile memory such as at least one disk memory.

[0087] The program 610 can be specifically configured to cause the processor 602 to execute the coarse-grained sentiment analysis method for image text in any of the above-mentioned method embodiments. The specific implementation of each step in the program 610 can refer to the corresponding description in the above-mentioned coarse-grained sentiment analysis method for image text, and will not be described here. Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working process of the above-mentioned devices and modules can refer to the corresponding process description in the above-mentioned method embodiments, and will not be described here.

[0088] The algorithms or displays provided herein are not inherently related to any particular computer, virtual system, or other apparatus. Various general-purpose systems can be used with these teachings, based on the description as set forth above. In terms of structure, those skilled in the art will appreciate that the required structure for a particular application can be implemented using the above description as a guide. Moreover, the present embodiments are not intended to be limited to any particular programming language. It will be appreciated that there are many programming languages that can be used to implement the teachings of the present embodiments as described herein, and any such programming language can be used in connection with the various aspects of the present embodiments.

[0089] In the specification provided herein, a large number of specific details are described. However, it can be understood that the embodiments of the present application can be practiced without these specific details. In some examples, well-known methods, structures and techniques are not described in detail in order not to obscure the understanding of the present specification.

[0090] Similarly, it is to be understood that the embodiments of the application can be altered, combined, or modified and that any alteration, combination, or modification is intended to fall within the scope of the application. For example, it is intended that any steps laid out in the claims can be performed in any order or simultaneously, and are not limited to the specific order or sequence described herein. Moreover, while embodiments of the application have been described as being implemented in one or more specific examples, it will be apparent to those of ordinary skill in the art that numerous modifications can be made without departing from the techniques described herein. Accordingly, the particular implementation of the embodiments of the application described in the specification is intended to be illustrative only and not limiting of the scope of the application as described by the claims. Other embodiments will be apparent to those of ordinary skill in the art from consideration of the specification and practice of the techniques disclosed herein. For example, features of one embodiment can be incorporated into another embodiment. It is therefore intended that the specification and figures be considered as illustrative only presenting the spirit of the application and not limiting of the scope of the application as more fully expressed by the appended claims.

[0091] Those skilled in the art will appreciate that the modules in the apparatuses in the embodiments can be adapted and placed in one or more apparatuses other than the embodiments. The modules or units or components in the embodiments can be combined into one module or unit or component, and further can be divided into more sub-modules or sub-units or sub-components. Any combination of all the features disclosed in the specification (including the accompanying claims, abstract and drawings), and any method or apparatus so disclosed, can be taken in any combination, except that at least some of such features and / or processes or units are mutually exclusive. Unless explicitly stated otherwise, each feature disclosed in the specification (including the accompanying claims, abstract and drawings) can be replaced by alternative features that serve the same, equivalent or similar purpose.

[0092] Further, those skilled in the art will appreciate that the features of the different embodiments can be combined in any combination and that the combinations of features are within the scope of the application and form different embodiments. For example, in the following claims, any of the claims can be read limiting as the other claims in any combination.

[0093] The various component embodiments of the present application can be implemented in hardware, or as software modules running in one or more processors, or in combinations thereof. As will be appreciated by one skilled in the art, a microprocessor or digital signal processor (DSP) can be used in practice to implement some or all of the functionality of some or all of the components according to the embodiments of the present application. The embodiments of the present application can also be implemented as a program of instructions for performing part or all of the methods described herein, e.g., a computer program and a computer program product. Such program implementing the embodiments of the present application can be stored on a computer readable medium, or can be in the form of one or more signals. Such signals can be downloaded from an Internet website, or provided on a carrier signal, or in any other form.

[0094] It should be noted that the above-mentioned embodiments illustrate rather than limit the application, and that one skilled in the art will be able to design many alternative embodiments without departing from the scope of the appended claims. In the claims, any reference signs placed between parentheses shall not be construed as limiting the claim. The word 'comprising' does not exclude the presence of elements or steps other than those listed in a claim. The word 'a' or 'an' preceding an element does not exclude the presence of a plurality of such elements. The application can be implemented by means of both hardware and software, and any combination thereof. In a unit claim, several devices can be listed with a conjunction like 'or', but it is to be understood that each of these devices can be implemented by its own hardware item. The use of the word 'at least' followed by a list of one or more members does not preclude the presence of additional such members, nor does it preclude further members. The word 'first' or'second' does not have any order connotations. The terms 'comprise', 'comprising', 'comprises' and 'comprising' should not be interpreted as a limitation on the means of the patentable scope of the claims. The word'step' does not imply any order between steps. The word'step' does not imply any order between steps.

Claims

1. A coarse-grained sentiment analysis method for image-text, characterized in that, The method comprises: recognizing a picture to obtain text information and text structured information in the picture; fusing preset label information with the text information, and inputting the same into a main task preset model and an auxiliary task preset model respectively to obtain a main task text vector and an auxiliary task text vector; wherein the preset label information comprises sentiment classification labels and emotion recognition labels; the main task preset model is used for coarse-grained sentiment analysis; and the auxiliary task preset model is used for fine-grained sentiment analysis; converting the text structured information to obtain a picture vector, and splicing the picture vector with the main task text vector to obtain a main task picture text vector; training and adjusting the main task preset model by using the auxiliary task text vector and the main task picture text vector to obtain an output result of coarse-grained sentiment analysis of the main task preset model; wherein, based on the auxiliary task text vector and the main task picture text vector, hidden layer outputs of the main task preset model and the auxiliary task preset model are determined, and a similarity between the hidden layer outputs of the main task preset model and the auxiliary task preset model is calculated; the similarity is used to adjust a loss function of the main task preset model, and the main task preset model is trained and adjusted based on the loss function of the main task preset model to obtain the output result of coarse-grained sentiment analysis of the main task preset model.

2. The method of claim 1, wherein, The text structured information comprises at least one of position information of text in a picture, text font size information, and text color information; The recognizing a picture to obtain text structured information in the picture further comprises at least one of: detecting the position of text in the picture to obtain coordinate information of a text paragraph as the position information of text in the picture; determining coordinate information of each character according to the coordinate information of the text paragraph, and obtaining text font size information according to the coordinate information of each character; obtaining the chroma and saturation of the text paragraph, and calculating the variance of each character and the chroma and saturation of the text paragraph as text color information.

3. The method of claim 1, wherein, The fusing the preset label information with the text information comprises: text expansion of the text information by using preset punctuation symbols to obtain a plurality of text information after text expansion; fusing the preset label information with the plurality of text information after text expansion.

4. The method of claim 3, wherein, The text expansion of the text information by using preset punctuation symbols to obtain a plurality of text information after text expansion further comprises: randomly inserting preset punctuation symbols into the text information to obtain a plurality of expanded text information; calculating the similarity between the plurality of expanded text information and the text information; merging the preset number of expanded text information ranked first according to the similarity from high to low with the text information to obtain a plurality of text information after text expansion.

5. The method of claim 1, wherein, The determining the hidden layer output of the main task preset model and the hidden layer output of the auxiliary task preset model based on the auxiliary task text vector and the main task picture text vector, and calculating the similarity between the hidden layer output of the main task preset model and the hidden layer output of the auxiliary task preset model further comprises: performing vector connection processing on the main task picture text vector and the auxiliary task text vector, inputting the main task picture text vector and the auxiliary task text vector into the hidden layer of the main task preset model, and obtaining the hidden layer output of the main task preset model; inputting the auxiliary task text vector into the hidden layer of the auxiliary task preset model to obtain the hidden layer output of the auxiliary task preset model; calculating the similarity between the hidden layer output of the main task preset model and the hidden layer output of the auxiliary task preset model using a cosine function.

6. The method of claim 1, wherein, The adjusting the loss function of the main task preset model using the similarity, and training and adjusting the main task preset model based on the loss function of the main task preset model to obtain the output result of the coarse-grained sentiment analysis of the main task preset model further comprises: obtaining the loss function of the main task preset model using the similarity and a preset cross-entropy loss function; performing weighted linear fusion on the loss function of the main task preset model and the loss function of the auxiliary task preset model to obtain a common loss function of the main task preset model and the auxiliary task preset model; training and adjusting the main task preset model and the auxiliary task preset model based on the common loss function to obtain the output result of the coarse-grained sentiment analysis of the main task preset model.

7. A coarse-grained sentiment analysis device for picture text, characterized in that the device comprises: an identification module adapted to identify a picture to obtain text information and text structured information in the picture; a fusion module adapted to fuse preset label information with the text information, and input the preset label information and the text information into a main task preset model and an auxiliary task preset model respectively to obtain a main task text vector and an auxiliary task text vector; wherein the preset label information comprises sentiment classification labels and emotion recognition labels; the main task preset model is used for coarse-grained sentiment analysis; and the auxiliary task preset model is used for fine-grained sentiment analysis; a splicing module adapted to convert the text structured information to obtain a picture vector, and splice the picture vector with the main task text vector to obtain a main task picture text vector. The similarity constraint module is adapted to train and adjust the main task preset model by using the auxiliary task text vector and the main task picture text vector, to obtain an output result of coarse-grained sentiment analysis of the main task preset model; wherein, based on the auxiliary task text vector and the main task picture text vector, the hidden layer output of the main task preset model and the hidden layer output of the auxiliary task preset model are determined, and the similarity between the hidden layer output of the main task preset model and the hidden layer output of the auxiliary task preset model is calculated; the loss function of the main task preset model is adjusted by using the similarity, and the main task preset model is trained and adjusted based on the loss function of the main task preset model, to obtain the output result of coarse-grained sentiment analysis of the main task preset model.

8. A computing device, comprising: Comprise: a processor, a memory, a communication interface and a communication bus, the processor, the memory and the communication interface complete mutual communication through the communication bus; The memory is used to store at least one executable instruction, and the executable instruction makes the processor execute the operation corresponding to the picture text oriented coarse-grained sentiment analysis method in any one of claims 1-6.

9. A computer storage medium, characterized in that The storage medium has at least one executable instruction stored therein, and the executable instruction makes the processor execute the operation corresponding to the picture text oriented coarse-grained sentiment analysis method in any one of claims 1-6.

Images