Text information classification method and device and electronic equipment

Abstract

Embodiments of the invention disclose a text information classification method and apparatus, and an electronic device. The method comprises the steps of obtaining to-be-identified text information and multimedia object information commented by the to-be-identified text information; performing text feature extraction on the image frame and the voice information in the multimedia object information to obtain a first text feature and a second text feature of the multimedia object information; inputting the image frames in the multimedia object information into a pre-established abstract description generation model to obtain abstract description text features of the multimedia object information; and performing interactive fusion on the first text feature, the second text feature and the abstract description text feature with the comment text feature by using an emotion recognition model, and determining an emotion tag corresponding to the to-be-recognized text information based on the fused features. By utilizing the technical scheme provided by the specification, the classification accuracy of the text information is improved.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a method, device, and electronic device for classifying text information. Background Art

[0002] With the development of internet technology and the advancement of society, people are now able to express their personal opinions or comments on the information they see online. For example, they can comment on a short video, comment on content on social networking sites, or post comments on videos or live broadcasts using bullet screens. These comments often carry the user's personal emotions, such as happiness, anger, or sadness. Emotion recognition in user comments can help analyze the content that users are interested in and improve the conversion rate of information.

[0003] Sentiment analysis, also known as opinion mining, aims to extract opinions from large amounts of unstructured text and categorize sentiment into positive, neutral, and negative categories. Existing techniques for sentiment recognition primarily train multiple content sentiment classification models based on different modalities, then use a superposition of scores to determine the degree of preference of commenting users for the content. This process requires continuous adjustment of the thresholds of each sub-model to achieve the optimal combination. Alternatively, sentiment recognition is performed only on the content to be identified to obtain recognition results. However, human language and emotions are relatively rich, and single-modality sentiment recognition struggles to identify true human emotion categories, impacting the accuracy of both sentiment recognition results and comment information classification.

[0004] Therefore, how to provide a text information classification scheme that can accurately classify the text of comments and thereby improve the accuracy of sentiment recognition of comment text information is a technical problem that urgently needs to be solved in this field. Summary of the Invention

[0005] The embodiments of this specification provide a text information classification method, apparatus, device, storage medium, and computer program product, which improve the accuracy of text information classification and lay a data foundation for subsequent information recommendation.

[0006] In one aspect, an embodiment of this specification provides a method for classifying text information, the method comprising:

[0007] Acquire text information to be identified and multimedia object information commented on by the text information to be identified;

[0008] Performing text feature extraction on the image frame and the voice information in the multimedia object information respectively to obtain a first text feature and a second text feature of the multimedia object information;

[0009] Inputting the image frame in the multimedia object information into a pre-established summary description generation model to obtain summary description text features of the multimedia object information; the summary description generation model is used to convert the image frame into text description information;

[0010] The first text feature, the second text feature, the summary description text feature and the comment text feature in the text information to be identified are input into a pre-created emotion recognition model, and the first text feature, the second text feature, the summary description text feature and the comment text feature are interactively fused with the emotion recognition model to obtain a first interactive fusion feature, a second interactive fusion feature and a third interactive fusion feature respectively, and based on the first interactive fusion feature, the second interactive fusion feature and the third interactive fusion feature, the emotion label corresponding to the text information to be identified is determined.

[0011] Another aspect provides a text information classification device, the device comprising:

[0012] The module for obtaining information to be identified is used to obtain text information to be identified and multimedia object information commented on by the text information to be identified;

[0013] A first feature extraction module is configured to extract text features from image frames and voice information in the multimedia object information to obtain first and second text features of the multimedia object information;

[0014] a second feature extraction module, configured to input the image frames in the multimedia object information into a pre-established summary description generation model to obtain summary description text features of the multimedia object information; the summary description generation model is configured to convert the image frames into text description information;

[0015] The emotion recognition module is used to input the first text feature, the second text feature, the summary description text feature and the comment text feature in the text information to be identified into a pre-created emotion recognition model, and use the emotion recognition model to interactively fuse the first text feature, the second text feature, the summary description text feature with the comment text feature to obtain a first interactive fusion feature, a second interactive fusion feature and a third interactive fusion feature respectively, and determine the emotion label corresponding to the text information to be identified based on the first interactive fusion feature, the second interactive fusion feature and the third interactive fusion feature.

[0016] Another aspect provides an electronic device, comprising: a processor;

[0017] a memory for storing instructions executable by the processor;

[0018] The processor is configured to execute the instructions to implement any of the above-mentioned methods for classifying text information.

[0019] On the other hand, a computer-readable storage medium is provided. When instructions in the storage medium are executed by a processor of an electronic device, the electronic device can execute any of the above-mentioned text information classification methods.

[0020] Another aspect provides a computer program product or computer program, comprising computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the text information classification method provided in the various optional implementations described above.

[0021] The text information classification method, apparatus, device, storage medium, and computer program product provided in the embodiments of this specification have the following technical effects:

[0022] The text information classification method provided in the embodiment of this specification combines the features of the text information to be identified with the multimedia object information it comments on, and can use the content in the multimedia object information together in the process of identifying the emotional tag of the text information to be identified. Feature fusion is performed based on the multimodal features of the multimedia object information and the text features in the text information to be identified, and the emotions in the text information to be identified are identified based on the fused features. In addition, the context and meaning of the commented multimedia object information are combined to improve the accuracy of the classification of the text information. In addition, when extracting features from the multimedia object information, the embodiment of this specification combines the summary description generation model to understand the image frames in the multimedia object information and extracts the summary description text of the image frame. This can make up for the problem that when the text is directly extracted from the image or voice, the obtained text is confusing and may not accurately express the meaning of the multimedia object information. The summary description text corresponding to the multimedia object information and the text features in the image frame and voice are combined to more comprehensively cover the content of the multimedia object information, improve the accuracy of feature extraction, and further improve the accuracy of the emotion recognition and classification of the text information. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] In order to more clearly illustrate the technical solutions and advantages of the embodiments of this specification or the prior art, the following is a brief introduction to the drawings required for use in the embodiments or the prior art descriptions. Obviously, the drawings described below are only some embodiments of this specification. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0024] Figure 1 What is shown is merely a schematic diagram of an application environment of a text information classification method;

[0025] Figure 2 This is a schematic diagram of the principle of emotion recognition in the prior art;

[0026] Figure 3 This is a flowchart of a text information classification method provided by an embodiment of this specification;

[0027] Figure 4 This is a schematic diagram of the principle flow of feature interaction fusion in one embodiment of this specification;

[0028] Figure 5 This is a schematic diagram of a multi-label output framework in one embodiment of this specification;

[0029] Figure 6 This is a schematic diagram of the process of automatic model update in one embodiment of this specification;

[0030] Figure 7 is a schematic diagram of the structure of a text information classification system in one embodiment of this specification;

[0031] Figure 8 This is a schematic diagram comparing the effects of sentiment recognition on comment information in one embodiment of this specification;

[0032] Figure 9 This is a structural diagram of a text information classification device provided in an embodiment of this specification;

[0033] Figure 10 This is a block diagram of an electronic device for classifying text information provided by an embodiment of this specification;

[0034] Figure 11 This is a block diagram of another electronic device for classifying text information provided by an embodiment of this specification. DETAILED DESCRIPTION

[0035] The following will be combined with the drawings in the embodiments of this specification to clearly and completely describe the technical solutions in the embodiments of this specification. Obviously, the embodiments described are only part of the embodiments of this specification, not all of the embodiments. Based on the embodiments in this specification, all other embodiments obtained by ordinary technicians in this field without making any creative efforts are within the scope of protection of this specification.

[0036] It should be noted that the terms "first", "second", etc. in the description and claims of the embodiments of this specification and the above-mentioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of this specification described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, system, product or server that includes a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0037] See also Figure 1 , Figure 1 1 is a schematic diagram of an application environment of a text information classification method provided in an embodiment of this specification. The application environment may include at least a server 100 and a terminal 200.

[0038] In an optional embodiment, the server 100 can be used to receive data sent by the terminal 200, such as comments on a video, and perform emotion recognition on the comments. The server 100 can be an independent physical server, a server cluster or a distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms.

[0039] In an optional embodiment, terminal 200 can be a terminal device that sends data to server 100, or it can directly obtain local comment information and perform emotion recognition on the comment information. Specifically, terminal 200 can be an electronic device such as a smartphone, desktop computer, tablet computer, laptop computer, smart wearable device, or in-vehicle device; it can also be software running on such electronic devices, such as applications or applets. The operating systems running on the electronic devices in the embodiments of this specification may include, but are not limited to, Android, iOS, Linux, Windows, etc.

[0040] In addition, it should be noted that Figure 1 What is shown is merely a schematic diagram of an application environment of a text information classification method, and the embodiments of this specification are not limited to the above.

[0041] In the embodiments of this specification, the server 100 and the terminal 200 may be directly or indirectly connected via wired or wireless communication, which is not limited in this embodiment of this specification.

[0042] The text information classification method in the embodiments of this specification can be applied to some scenarios with comments, and perform emotion recognition on the text information of the user's comments to determine the emotional label of the text information. For example, emotion recognition is performed on the user's comment information under a certain video. The video can be a video on a short video platform or a video posted by certain users on a social networking site. General emotion recognition mainly trains multiple content emotion classification models based on different features, and then judges the degree of preference of the commenting user for the content by superimposing scores. During the process, the threshold of each sub-model needs to be continuously adjusted to achieve the optimal combination effect. Figure 2 This is a schematic diagram of the principle of emotion recognition in the existing technology, such as Figure 2 As shown, in this scenario, classification models are trained separately for titles, comments, and topic content. The prediction scores from each classification model are then combined to determine a comprehensive classification result. However, human language is complex, with various ironic and other expressions that contradict emotion. Modeling each type of feature separately is insufficient for multimodal emotion recognition, affecting the accuracy of emotion recognition results.

[0043] When performing emotion recognition on comment information, the embodiments of this specification combine the multimedia object information corresponding to the comment information, perform multimodal feature extraction of images, voice and video on the multimedia object information, and perform feature fusion based on the multimodal features of the multimedia object information and the text features in the comment information, so as to more accurately identify the emotion tags in the comment information.

[0044] The following describes a text information classification method according to an embodiment of this specification. Figure 3 It is a flow chart of the text information classification method provided by an embodiment of this specification. This specification provides method operation steps such as the embodiment or flow chart, but may include more or fewer operation steps based on conventional or non-creative labor. The order of steps listed in the embodiment is only one way of executing the order of many steps and does not represent the only execution order. When the actual system or server product is executed, it can be executed in sequence or in parallel according to the method shown in the embodiment or the accompanying drawings (for example, in a parallel processor or multi-threaded processing environment). The method can be applied to terminals such as computers, tablet computers, smart phones, smart wearable devices, and vehicle-mounted devices, or in terminal applications or applets. Of course, it can also be applied to servers according to actual needs, and the embodiments of this specification do not specifically limit this. Specifically, Figure 3 As shown, the method may include:

[0045] S302: Acquire the text information to be recognized and the multimedia object information commented on by the text information to be recognized.

[0046] In a specific implementation, the text information to be identified can be understood as comment information that requires emotion recognition. For example, if user A posts a comment S in the comment section of short video 1, and emotion recognition is required on comment S to determine user A's emotional state at the time of posting comment S or to identify user A's emotional state towards short video 1 from comment S, comment S can be used as the text information to be identified. The text information to be identified can generally be in the form of text, and can also contain pictures or emoticons, etc. The specific form can be determined according to actual needs and is not specifically limited in this embodiment of the specification. The text information to be identified can be the comments posted by users in the comment section of certain videos, or the comments posted by users in certain videos. Depending on actual use needs, it can also be the comments posted by users on news or articles on certain social networking sites. The specific application scenarios and forms are not specifically limited in this embodiment of the specification. Multimedia object information can be understood as the content or object commented on by the text information to be identified. For example, in the short video 1 in the above example, if the text information to be identified is a comment posted by a user while watching a movie, then the multimedia object information can be the movie or a movie clip within the movie within a preset time before and after the comment was posted. Multimedia object information generally includes multiple modalities such as text, voice, and image. The embodiments of this specification do not limit the specific form of the multimedia object information.

[0047] S304: Perform text feature extraction on the image frame and voice information in the multimedia object information respectively to obtain a first text feature and a second text feature of the multimedia object information.

[0048] In the specific implementation process, the multimedia object information in the embodiment of this specification can generally include image frames and voice information, which is a multimodal information. When classifying text information, the embodiment of this specification first performs text feature extraction on the image frame in the multimedia object information corresponding to the text information to be identified, extracts the text in the image frame, and obtains the first text feature of the image frame in the multimedia object information. At the same time, text feature extraction can also be performed on the voice information in the multimedia object information, and the voice information is converted into corresponding text to obtain the second text feature of the voice information in the multimedia object information. The image frame in the multimedia object information can be understood as the picture in the multimedia object information, such as: the video frame in the video information or the picture in the news, etc., and the voice information in the multimedia object information can be understood as the music or dubbing in the multimedia object information, etc. If the multimedia object information is video information, then the multimedia object information can be subjected to equal-frequency frame extraction, and the obtained video frame can be the image frame of the multimedia object information.

[0049] Among them, the specific method of text feature extraction can be selected according to actual needs, such as: pre-training a text extraction model and using the trained model to extract text features from image frames and voice in multimedia object information, or selecting other text feature extraction methods, which are not specifically limited in the embodiments of this specification.

[0050] In some embodiments of this specification, text feature extraction is performed on image frames and voice information in multimedia object information to obtain first and second text features of the multimedia object information, including:

[0051] Extracting text information from an image frame in the multimedia object information using optical character recognition technology to obtain a first text feature;

[0052] Automatic speech recognition technology is used to convert speech information in the multimedia object information into text information to obtain a second text feature.

[0053] In a specific implementation, when extracting text features from multimedia object information corresponding to the text information to be recognized, optical character recognition (OCR) can be used to extract text features from image frames within the multimedia object information, thereby obtaining a first text feature of the multimedia object information. OCR can be understood as a process of detecting dark and light patterns to determine shape, and then using character recognition methods to translate the shape into computer text. OCR can be used to recognize text within images of multimedia object information, thereby obtaining a first text feature of the multimedia object information. Furthermore, automatic speech recognition (ASR) can be used to extract text features from speech information within the multimedia object information, thereby obtaining a second text feature of the multimedia object information. ASR converts spoken language (audio signals) into written text, typically used as commands. ASR can be used to recognize text within speech within multimedia object information, thereby obtaining a second text feature of the multimedia object information.

[0054] Of course, if there is no image frame or voice in the multimedia object information, or there is no text in the image frame, then the corresponding extracted first text feature or second text feature may be empty.

[0055] The embodiments of this specification utilize OCR technology and ASR technology to quickly extract text features from image frames and speech in multimedia object information whose text information is to be recognized, and convert the image modality and speech modality in the multimedia object information into text form, thereby laying a data foundation for subsequent emotion recognition.

[0056] S306: Input the image frame in the multimedia object information into a pre-established summary description generation model to obtain summary description text features of the multimedia object information; the summary description generation model is used to convert the image frame into text description information based on the content of the image frame.

[0057] During specific implementation, when extracting text features from image frames and speech in multimedia object information, the embodiments of this specification may also understand the image frames in the multimedia object information and convert the image frames into summary description text features. The summary description text features can be used to describe the main content of the image frames, that is, to describe the content of the image frames using a paragraph of text. Specifically, a machine learning model can be used to convert the video information in the multimedia object information into text, extract text features from the video information, and obtain summary description text features of the video information. For example, a summary description generation model can be pre-constructed. The summary description generation model can convert the image frames into text description information based on the content of the image frames. For example, the summary description generation model can understand the image content in the image frames, thereby describing the content expressed by the image frames using a paragraph of text, and obtaining summary description text features of the image frames. The model structure of the summary description generation model can be selected according to actual needs. For example, a deep learning algorithm or other machine learning algorithm can be selected for model training. The embodiments of this specification do not specifically limit the model structure of the description generation model and the type of algorithm used.

[0058] For example, multimedia object information may be a video, which includes multiple image frames. When extracting text features from the video information, the multimedia object information can be extracted once every preset time. For example, when the multimedia object information is played, one frame of the multimedia object information is extracted every 1 second. In this way, multiple image frames can be obtained. The obtained multiple image frames are input into the summary description generation model. The summary description generation model can be used to understand the content of the image frames and convert the multiple image frames into corresponding summary description text features. The summary description generation model can be used to express the content of the image frame in the form of text. For example, the image frame can be converted into a corresponding text description based on the text or subtitles in the image frame or the positional relationship between the objects in the image frame. The summary description generation model is similar to a picture description function, which can convert the multimedia object information into corresponding text features based on the content of the multimedia object information.

[0059] Because when performing text feature extraction on the image frames and speech of multimedia object information, the text in the image frames is directly extracted and the speech is converted into text, the extracted text may be chaotic, incoherent or incoherent, resulting in an inability to accurately represent the meaning of the multimedia object information, or when the multimedia object information does not have pictures, the extracted feature content is limited. The embodiment of this specification uses a description generation model to understand the image frames in the multimedia object information corresponding to the text information to be identified, and converts them into corresponding summary description text features, so as to obtain relatively complete and accurate text features in the multimedia object information. By understanding the content of the multimedia object information and converting the multimedia object information into corresponding text features, it can make up for the defect that the extracted text cannot represent the meaning of the multimedia object information when text features are directly extracted from pictures and speech, thereby improving the accuracy of subsequent sentiment recognition of comment information.

[0060] S308: Input the first text feature, the second text feature, the summary description text feature and the comment text feature in the text information to be identified into a pre-created emotion recognition model, and use the emotion recognition model to interactively fuse the first text feature, the second text feature, the summary description text feature with the comment text feature to obtain a first interactive fusion feature, a second interactive fusion feature and a third interactive fusion feature respectively, and determine the emotion label corresponding to the text information to be identified based on the first interactive fusion feature, the second interactive fusion feature and the third interactive fusion feature.

[0061] In the specific implementation process, sample data can be collected in advance for training and constructing an emotion recognition model, such as: collecting some comment sample information and corresponding comment subject sample information, annotating the comment sample information, determining the emotion sample label corresponding to each comment sample information, and then using the method provided in the above embodiment to extract text features in the comment subject sample information, using the extracted text features together with the text features in the comment sample information as input, and using the emotion sample label corresponding to each comment sample information as output, to perform model training and construct an emotion recognition model. The emotion recognition model can be used to perform emotion recognition on the input comment information and determine the emotion label of the comment information. The model structure and the algorithm used in the emotion recognition model can be selected according to actual needs and are not specifically limited in the embodiments of this specification. In the model training stage, based on the deepspeed (a model training tool) framework, CPU offload and zero stage 3 training mode can be used to divide the model into 8 A100 GPUs to improve the training speed of the model. The training parameters can be set according to actual needs and are not specifically limited in the embodiments of this specification.

[0062] The text information to be identified is generally in text form, and it is not necessary to extract text features separately. After extracting the text features of the multimedia object information corresponding to the text information to be identified, the obtained first text features, second text features, and summary description text features can be input into a pre-trained emotion recognition model together with the comment text features in the text information to be identified. The emotion recognition model can interactively fuse the input first text features, second text features, and summary description text features with the comment text features in the text information to be identified, such as: interactively fusing the first text features with the comment text features to obtain a first interactive fusion feature, interactively fusing the second text features with the comment text features to obtain a second interactive fusion feature, and interactively fusing the summary description text features with the comment text features to obtain a third interactive fusion feature.

[0063] Among them, the specific method of feature interaction fusion can be selected according to actual needs, such as: after converting two features into vectors, performing the dot product of the vectors to realize the interactive fusion of the two features, or selecting a dedicated feature fusion algorithm or model structure for feature fusion, such as: using attention mechanisms such as Cross-Attention or Self-Attention or Multi-HeadAttention for feature fusion. The embodiments of this specification do not specifically limit the specific method of feature interaction fusion.

[0064] In some embodiments of this specification, the sentiment recognition model is used to interactively fuse the first text feature, the second text feature, and the summary description text feature with the comment text feature to obtain a first interactive fusion feature, a second interactive fusion feature, and a third interactive fusion feature, respectively, including:

[0065] Using the embedded vector layer in the sentiment recognition model, the first text feature, the second text feature, the summary description text feature, and the comment text feature are respectively converted into vector features to obtain the first text vector feature, the second text vector feature, the summary description text vector feature, and the comment text vector feature;

[0066] The first text vector feature, the second text vector feature, and the summary description text vector feature are interactively fused with the comment text vector feature to obtain a first interactive fusion feature, a second interactive fusion feature, and a third interactive fusion feature, respectively.

[0067] In the specific implementation process, after obtaining the text features in the comment information and multimedia object information, the embedding vector layer in the emotion recognition model can be used to convert each text feature into a vector feature. Specifically, an embedding vector layer, i.e., an embedding layer, can be set in the emotion recognition model. After the extracted text features are input into the emotion recognition model, the embedding layer can perform vector conversion on the input first text features, second text features, summary description text features, and comment text features, and obtain the first text vector features, second text vector features, summary description text vector features, and comment text vector features, respectively. Among them, the specific structure of the embedding layer can be set according to actual needs, and the embodiments of this specification do not make specific limitations. For example, the small version of RoBerta can be used as the embedding layer for vector conversion. The RoBerta (A Robustly Optimized BERT) model is an improved version of BERT (Bidirectional Encoder Representations from Transformers, a pre-trained language model). RoBerta can improve the accuracy of vector conversion. Figure 4 This is a schematic diagram of the principle flow of feature interaction fusion in one embodiment of this specification. Figure 4 As shown, after obtaining the vector features corresponding to each text feature, the first text vector feature corresponding to the image frame in the multimedia object information can be interactively fused with the comment text vector feature corresponding to the text information to be identified to obtain the first interactive fusion feature. Similarly, the second text vector feature corresponding to the voice information in the multimedia object information can be interactively fused with the comment text vector feature corresponding to the text information to be identified to obtain the second interactive fusion feature. The summary description text vector feature corresponding to the video information in the multimedia object information can be interactively fused with the comment text vector feature corresponding to the text information to be identified to obtain the third interactive fusion feature. Figure 4 As shown, Cross-Attention can be used as the interactive feature fusion method. Generally, before feature fusion, each vector feature is aligned, and then feature fusion is performed. The fused features are input into the concat layer, where the first, second, and third interactive fusion features are concatenated to obtain a complete feature set for emotion recognition.

[0068] In the embodiments of this specification, text features of comment information and multimedia object information can be extracted first, and then the extracted text features can be converted into vector features that can be recognized by the computer. Feature fusion is then performed separately to lay a data foundation for subsequent classification of text information.

[0069] After obtaining the fusion features, the first, second, and third interactive fusion features can be combined to perform emotion recognition and determine the emotion label corresponding to the text information to be identified. This emotion label can be used to characterize the emotion contained in the text information to be identified. Furthermore, it can also be used to characterize the emotion of the user when posting the text information to be identified, or the emotion of the multimedia object information commented on by the user in the text information to be identified, such as happiness, anger, sadness, etc.

[0070] In some embodiments of the present specification, there are multiple emotion tags corresponding to the text information to be identified, and the multiple emotion tags are used to represent different emotion types in the text information to be identified.

[0071] In the specific implementation process, Figure 5 This is a schematic diagram of a multi-label output framework in one embodiment of this specification. Figure 5 As shown, in the embodiment of this specification, when using the emotion recognition model to identify the emotion label of the text information to be identified, a multi-emotion label output method can be adopted. When outputting the emotion label corresponding to the text information to be identified based on the interactive fusion features, multiple emotion labels can be output at the same time, such as: happy, disappointed, scared, etc. Multiple emotion labels can be used to represent different emotion types in the text information to be identified. Human emotions are complex, and opinions on a thing are likely to be complex. A single emotion label may not be able to express the emotions contained in the text information to be identified. Through the form of multiple labels, different emotion types in the text information to be identified can be more accurately identified, and then the user's mood when posting the comment can be covered as much as possible, and the emotions in the comment information posted by the user can be accurately captured, laying a data foundation for the recommendation of subsequent information and the improvement of information conversion rate.

[0072] The embodiment of this specification provides a method for classifying text information, which combines the features of the text information to be identified with the multimedia object information it comments on, and can use the content in the multimedia object information together in the process of identifying the emotional tag of the text information to be identified. Feature fusion is performed based on the multimodal features of the multimedia object information and the text features in the text information to be identified, and the emotions in the text information to be identified are identified based on the fused features. In addition, the context and meaning of the commented multimedia object information are combined to improve the accuracy of the classification of the text information. In addition, when extracting features from the multimedia object information, the embodiment of this specification combines the summary description generation model to understand the image frames in the multimedia object information and extracts the summary description text of the image frame. This can make up for the problem that the text obtained when the image and voice are directly extracted is confusing and may not accurately express the meaning of the multimedia object information. The summary description text corresponding to the multimedia object information and the text features in the image frame and voice can be combined to more comprehensively cover the content of the multimedia object information, improve the accuracy of feature extraction, improve the accuracy of emotion recognition of comment information, and thus improve the accuracy of text information classification.

[0073] In some embodiments of this specification, the method further includes:

[0074] According to the emotion label corresponding to the text information to be identified, the multimedia object information corresponding to the text information to be identified with the emotion label as a positive label is obtained as the recommended reference information of the target object corresponding to the text information to be identified;

[0075] obtaining recommended information of the target object based on the recommended reference information, where the recommended information is at least one of related information of the recommended reference information and information of the same category as the recommended reference information;

[0076] Push the recommended information to the target object.

[0077] In the specific implementation process, after determining the emotion label corresponding to the text information to be identified, the multimedia object information corresponding to the text information to be identified in which the emotion label is a positive label can be obtained based on the emotion label corresponding to the text information to be identified as the recommended reference information of the target object corresponding to the text information to be identified. Among them, the positive label can be understood as the emotion type represented by the emotion label is a positive or active emotion, such as: happy, glad, joyful, excited, etc. Different emotion labels can be divided into positive and negative according to actual needs. The embodiments of this specification do not specifically limit the specific type and content of the positive label. The information corresponding to the positive label can generally bring positive ideas to the user. The multimedia object information corresponding to the text information to be identified based on the positive label can be used as the recommended reference information of the target object corresponding to the text information to be identified, and relevant recommendation information can be obtained, such as: using the associated information of the reference information to be recommended corresponding to the positive label as the recommended information, or using the same category information of the reference information to be recommended corresponding to the positive label as the recommended information. Among them, the related information of the reference information to be recommended can be understood as information that has a certain connection with the reference information to be recommended, such as: video information of the same event or information posted by the same user or information that has a certain connection with the content, etc. The same category of information as the reference information to be recommended can be understood as information that belongs to the same type as the reference information to be recommended, such as: all information about travel or all information about food, etc.

[0078] Information associated with or of the same category as the reference information to be recommended usually has a positive emotional impact on the target object corresponding to the comment information to be detected, and is generally information that the target object is more interested in. In the embodiments of this specification, after screening out reference information to be recommended with positive emotional tags posted by the target object, the information associated with or of the same category as the reference information to be recommended is pushed to the corresponding target object as recommendation information. For example, the recommendation information can be sent to the target object's terminal, or to the target object's account in a designated application. The embodiments of this specification do not specifically limit the method of pushing and the time of pushing information.

[0079] In addition, in the embodiments of this specification, a multi-label output method is adopted. When screening text information to be identified with positive emotional labels, if the emotional types of the emotional labels in the comment letters to be identified are inconsistent, then the comment information with a number of emotional labels representing positive emotions greater than a preset number can be selected as the text information to be identified with positive emotional labels. For example, if the text information S to be identified has 5 emotional labels, 3 of which are positive emotional labels, and the preset number is assumed to be 2, then the text information to be identified S can be considered to be text information to be identified with positive emotional labels, and the associated information and / or information of the same category of the multimedia object information corresponding to the text information S to be identified can be pushed to the corresponding target object as recommendation information.

[0080] The embodiments of this specification achieve the goal of recommending information that is of interest to the target object and will have a positive emotional impact on the target object, thereby improving the accuracy of information recommendation and the information conversion rate, while also being able to bring more positive emotions to the user and improving the user experience.

[0081] In some embodiments of this specification, the method further includes:

[0082] Obtain feedback from each target audience on the recommended information pushed within a specified time range;

[0083] Determine the accuracy of the recommended information based on the feedback information;

[0084] If the accuracy is less than the preset threshold, the emotion recognition model is updated.

[0085] In the specific implementation process, the embodiments of this specification can also automatically update the emotion recognition model, calculate the accuracy of the recommended information in real time or at regular intervals, monitor the emotion tags predicted by the current emotion recognition model, and perform information recommendation accuracy. For example, feedback information on the recommended information pushed by each target object within a specified time range can be obtained at regular intervals, and the accuracy of the recommended information can be determined based on the feedback information. If the accuracy is less than a preset threshold, it means that the current recommended information does not meet the preset accuracy requirements, indicating that the accuracy of the emotion tags predicted by the emotion recognition model is no longer sufficient, and the emotion recognition model can be automatically updated. The specified time range can be set according to actual needs, such as one day, one week, or ten days before the current time, and this specification embodiment does not specifically limit it. The feedback information of each target object on the pushed recommended information can be understood as the actions of each target object on the pushed recommended information, such as clicking, commenting, browsing, or forwarding. If the target object does not click, comment, browse, or forward the pushed recommended information, it can be considered that the target object has no corresponding feedback information for the recommended information.

[0086] Generally, the recommended information determined by the emotion label of the evaluation information to be detected based on the emotion recognition model is the information that the target object is interested in. The target object will usually perform certain feedback operations on the information that he is interested in. If, within a specified time range, the feedback information of each target object on the pushed recommendation information is counted, it shows that most target objects have not made corresponding feedback on the pushed recommendation information, such as: after statistics, the proportion of feedback operations of each target object on the recommended information within the specified time range is less than the preset proportion, then it means that the emotion label of the comment information to be detected predicted by the emotion recognition model may be inaccurate, and the model needs to be adjusted and updated.

[0087] In the embodiments of this specification, the accuracy of the recommended information can be calculated by the feedback information of the target object on the recommended information, and further judgment can be made as to whether the emotion recognition model needs to be updated, thereby realizing the automatic update of the emotion recognition model, thereby improving the accuracy of emotion recognition of comment information and the accuracy of information recommendation, and improving the conversion rate of information.

[0088] The automatic update method of the emotion recognition model can be selected based on actual needs. For example, a sample database for emotion recognition model training can be constructed, and the sample data in the sample database can be updated regularly. When the emotion recognition model needs to be updated, the sample data in the sample database can be directly used to retrain and optimize the emotion recognition model. Alternatively, the recommendation information pushed to each target object and the feedback information of the target object on the recommendation information can be directly used as sample data to retrain and optimize the emotion recognition model. The embodiments of this specification do not specifically limit the update method of the emotion recognition model.

[0089] In some embodiments of this specification, the method further includes:

[0090] Selecting a preset number of recommendation information as updated multimedia object information at regular intervals, and selecting a specified number of comment information from the comment information of the updated multimedia object information as updated comment sample information;

[0091] Label the selected update comment sample information and determine the labeled sentiment label corresponding to each update comment sample information;

[0092] Constructing an updated sample subset based on the updated multimedia object information, the updated comment sample information, and the annotated sentiment labels corresponding to the updated comment sample information;

[0093] If the accuracy is less than the preset threshold, the emotion recognition model is updated, including:

[0094] When the accuracy rate is less than the preset threshold, an updated sample subset is obtained, and the updated comment sample information and updated multimedia object information in the updated sample subset are used as the input of the emotion recognition model. The annotated emotion label corresponding to the updated comment sample information is used as the output of the emotion recognition model to train and update the emotion recognition model.

[0095] During the specific implementation process, in the embodiments of this specification, a preset number of recommendation information can be periodically selected from the recommendation information pushed to each target object as updated multimedia object information. At the same time, a specified number of comment information can be selected from the comment information corresponding to the updated multimedia object information as updated comment sample information. Generally, the updated comment sample information is the comment information of the target object on the updated multimedia object information. Some additional comment information corresponding to the updated multimedia object information can also be selected together as updated comment sample information. The selected updated comment sample information is labeled, and the labeled emotion tags corresponding to each updated comment sample information are determined. When labeling the updated comment sample information, expert review can be involved to improve the accuracy of the labeled emotion tags. Then, based on the selected updated multimedia object information, the updated comment sample information, and the labeled emotion tags corresponding to the updated comment sample information, an updated sample subset is constructed.

[0096] When the emotion recognition model needs to be updated, the constructed update sample subset can be used as sample data for model update to update the emotion recognition model. Specifically, the updated comment sample information and updated multimedia object information in the update sample subset can be used as the input of the emotion recognition model, and the annotated emotion labels corresponding to the updated comment sample information can be used as the output of the emotion recognition model to update the emotion recognition model.

[0097] The embodiments of this specification periodically select some data from the recommendation information pushed to the target object as sample data for model update training. When the emotion recognition model needs to be updated, the emotion recognition model is trained and updated, so that the emotion recognition model can learn the recognized comment information again, which can improve the accuracy of the emotion recognition model and further improve the accuracy of emotion recognition classification of text information and the accuracy of information recommendation.

[0098] In some other embodiments of this specification, if the accuracy is less than a preset threshold, the emotion recognition model is updated, further comprising:

[0099] When the accuracy rate is less than the preset threshold, a model training sample set is obtained, where the model training sample set includes comment sample information, comment subject sample information corresponding to the comment sample information, and annotated sentiment labels;

[0100] The updated sample subset is added to the model training sample set, and the updated comment sample information, updated multimedia object information in the updated sample subset and the comment sample information and comment subject sample information in the model training sample set are used as the input of the emotion recognition model. The updated comment sample information and the annotated emotion labels corresponding to the comment sample information are used as the output of the emotion recognition model to train and update the emotion recognition model.

[0101] In the specific implementation process, when the accuracy of the recommended information is calculated and it is found that the accuracy is less than the preset threshold, and the emotion recognition model needs to be updated, a model training sample set can be obtained. The model training sample set may include comment sample information and the comment subject sample information and the annotated emotion tags corresponding to the comment sample information. The model training sample set can be constructed in the following way: after the emotion recognition model training is completed, a certain number of comment information and multimedia object information are randomly selected from the comment information and multimedia object information in the application or website, and then the selected comment information is manually annotated to determine the annotated emotion tags corresponding to the comment information. The model training sample set is constructed based on the selected comment information, multimedia object information and the annotated emotion tags of the comment information. Among them, when annotating the comment information in the sample data, refined annotation can be performed, and after the annotation, the quality inspection team will conduct a second review to ensure that the credibility of the sample is high enough.

[0102] In addition, the model training sample set can be updated regularly, that is, some new comment information and multimedia object information can be randomly selected as sample data at regular intervals to update the comment information and multimedia object information in the model training sample set. When the emotion recognition model needs to be updated, the constructed update sample subset can be added to the model training sample set and used together as sample data for model update to update the emotion recognition model. Specifically, the updated comment sample information, updated multimedia object information, comment sample information, and comment subject sample information in the update sample subset and the model training sample set can be used as the input of the emotion recognition model, and the updated comment sample information and the annotated emotion labels corresponding to the comment sample information can be used as the output of the emotion recognition model to update the emotion recognition model.

[0103] Over time, the coverage of the model continues to decline after it goes online. Figure 6 This is a flow chart of automatic model update in one embodiment of this specification. Figure 6As shown, in the embodiments of the present specification, the accuracy of the model can be automatically monitored, that is, the accuracy of the recommended information can be calculated in real time to determine whether the emotion recognition model needs to be updated and iterated. After determining that an update and iteration is needed, the emotion recognition model is trained and updated, and then the indicators of the trained and updated model are verified. For example, the trained and updated model can be tested with test samples. After the model accuracy meets the preset requirements, the updated model is put online, and the updated emotion recognition model is used to perform emotion recognition on the comment information.

[0104] The embodiments of this specification periodically select some data from the recommendation information pushed to the target object as sample data for model update training, and train and update the emotion recognition model together with other sample data when the emotion recognition model needs to be updated, so that the emotion recognition model can learn the recognized comment information again, and at the same time, it can also learn other relatively new comment information, which can improve the accuracy of the emotion recognition model and further improve the accuracy of text information classification and information recommendation.

[0105] Figure 7 is a structural diagram of a classification system for text information in one embodiment of this specification. Figure 7 As shown, some embodiments of this specification may also provide a text information classification system, which may include:

[0106] The perception layer is configured to extract text features from the image frame and voice information in the multimedia object information commented on by the text information to be recognized, obtain first and second text features of the multimedia object information, and extract summary description text features of the image frame in the multimedia object information using a pre-established summary description generation model; the summary description generation model is configured to convert the image frame into text description information based on the content of the image frame;

[0107] The understanding layer is used to interactively fuse the first text feature, the second text feature, and the summary description text feature with the comment text feature in the text information to be identified, respectively, to obtain a first interactive fusion feature, a second interactive fusion feature, and a third interactive fusion feature;

[0108] The recognition layer is used to determine the emotion label corresponding to the text information to be recognized based on the first interactive fusion feature, the second interactive fusion feature and the third interactive fusion feature.

[0109] In the specific implementation process, the perception layer is designed to extract the content of multimedia object information and map the image mode and voice mode to the text mode through cross-modal technology. Figure 7As shown in the figure, in addition to using OCR technology and ASR technology to extract text features from image modalities and speech modalities in multimedia object information, generative technology is also used to extract text features from video modalities in multimedia object information. For example, frames can be extracted from the video, and then the extracted image frames are fed into the generative model. The model outputs text descriptions of these image frames, that is, the summary description text features of the video information are obtained.

[0110] The understanding layer is primarily responsible for media information conversion and multimodal information alignment and interaction. Media information conversion primarily involves digitizing extracted media information as input for the sentiment recognition model, converting the extracted text into vector features. For text embedding, the RoBerta small version can be used, which has moderate resource requirements and excellent performance. Its main advantage over BERT is its use of dynamic masks, which improves the model's generalization capabilities. Before performing text embedding, the understanding layer pre-trains the RoBerta base model using MLM to ensure sufficient semantic capabilities in commentary scenarios. This dynamically generates masks for each input provided to the model.

[0111] After digitizing the information of each modality, it is necessary to align the data of each modality and interact with each other to learn the potential interactive information. In some embodiments of this specification, Cross Attention can be used to asymmetrically combine two embedding sequences of the same dimension, where one sequence is used as the query Q input and the other sequence is used as the key K and value V input. Figure 4 As shown in the figure, the OCR embedding (i.e., the first text vector feature), ASR embedding (i.e., the second text vector feature), and KW embedding (i.e., the summary description text vector feature) are input into the Cross Attention layer together with the CMT embedding (i.e., the comment text vector feature), and finally pass through a concat layer to summarize the information.

[0112] Considering that a user may have multiple emotion types in a comment text and there is label intersection, the recognition layer mainly uses multi-label output to facilitate accurate identification of different emotion types.

[0113] Among them, the specific methods of feature extraction, feature fusion and emotion recognition can refer to the records of the above embodiments. The perception layer, understanding layer and recognition layer can be designed as a model structure, that is, the emotion recognition system can be an emotion recognition model. Referring to the records of the above embodiments, the model can also be automatically iterated and updated.

[0114] Figure 8This is a schematic diagram comparing the effect of emotion recognition of comment information in one embodiment of this specification. Figure 8 As shown, in order to verify the technical effect of the solution provided in the embodiment of this specification, the solution of cross-modal recognition provided in the embodiment of this specification and Figure 2 The single modality recognition scheme was evaluated online. Figure 8 As shown, the cross-modal recognition solution provided by the embodiment of this specification is better than Figure 2 The single-modality recognition solution has increased coverage by over 10%. Furthermore, the multi-label output framework significantly reduces machine resources and time costs compared to building multiple models. Automatic model iteration, which triggers a switch when model coverage declines to a threshold, improves coverage by over 3% after iteration. Figure 8 The coverage rate in the above embodiment can be understood as the accuracy of the recommended information, which can be used to characterize the accuracy of the emotion labels predicted by the emotion recognition model.

[0115] Based on the above-mentioned text information classification method, one or more embodiments of this specification also provide a terminal and a server for the classification processing of text information. The terminal and the server may include a device (including a distributed system), software (application), module, component, server, terminal, etc. that uses the method described in the embodiment of this specification and is combined with a device for implementing the necessary hardware. Based on the same innovative concept, the device in one or more embodiments provided in the embodiment of this specification is as described in the following embodiments. Since the implementation scheme and method for solving the problem of the device are similar, the implementation of the specific device in the embodiment of this specification can refer to the implementation of the aforementioned method, and the repetitions will not be repeated. As used below, the term "unit" or "module" can be a combination of software and / or hardware that implements a predetermined function. Although the device described in the following embodiments is preferably implemented in software, the implementation of hardware, or a combination of software and hardware, is also possible and conceived.

[0116] As can be seen from the technical solutions provided in the above embodiments of this specification, this embodiment of this specification also provides a text information classification device. Figure 9 This is a structural diagram of a text information classification device provided by an embodiment of this specification. Figure 9 As shown, the above device includes:

[0117] The to-be-identified information acquisition module 910 is configured to acquire the to-be-identified text information and the multimedia object information commented on by the to-be-identified text information;

[0118] A first feature extraction module 920 is configured to extract text features from image frames and voice information in the multimedia object information to obtain first and second text features of the multimedia object information;

[0119] A second feature extraction module 930 is configured to input the image frames in the multimedia object information into a pre-established summary description generation model to obtain summary description text features of the multimedia object information; the summary description generation model is configured to convert the image frames into text description information based on the content of the image frames;

[0120] The emotion recognition module 940 is used to input the first text feature, the second text feature, the summary description text feature and the comment text feature in the text information to be identified into a pre-created emotion recognition model, and use the emotion recognition model to interactively fuse the first text feature, the second text feature, the summary description text feature with the comment text feature to obtain a first interactive fusion feature, a second interactive fusion feature and a third interactive fusion feature respectively, and determine the emotion label corresponding to the text information to be identified based on the first interactive fusion feature, the second interactive fusion feature and the third interactive fusion feature.

[0121] In some embodiments, the device further includes an information recommendation module for:

[0122] According to the emotion label corresponding to the text information to be identified, obtaining multimedia object information corresponding to the text information to be identified whose emotion label is a positive label as recommended reference information of the target object corresponding to the text information to be identified;

[0123] obtaining recommended information of the target object based on the recommended reference information, where the recommended information is at least one of associated information of the recommended reference information and information of the same category as the recommended reference information;

[0124] Push the recommendation information to the target object.

[0125] In some embodiments, the apparatus further comprises a model automatic updating module, configured to:

[0126] Obtaining feedback from each target object on the pushed recommendation information within a specified time range;

[0127] Determining the accuracy of the recommendation information based on the feedback information;

[0128] If the accuracy is less than a preset threshold, the emotion recognition model is updated.

[0129] In some embodiments, the model automatic update module is specifically used to:

[0130] Selecting a preset number of recommendation information as updated multimedia object information at regular intervals, and selecting a specified number of comment information from the comment information of the updated multimedia object information as updated comment sample information;

[0131] Annotate the selected update comment sample information and determine the annotated sentiment label corresponding to each update comment sample information;

[0132] Constructing an updated sample subset based on the updated multimedia object information, the updated comment sample information, and the annotated sentiment tags corresponding to the updated comment sample information;

[0133] If the accuracy is less than a preset threshold, updating the emotion recognition model includes:

[0134] When the accuracy rate is less than the preset threshold, the updated sample subset is obtained, and the updated comment sample information and updated multimedia object information in the updated sample subset are used as the input of the emotion recognition model. The annotated emotion label corresponding to the updated comment sample information is used as the output of the emotion recognition model to train and update the emotion recognition model.

[0135] In some embodiments, the model automatic update module is further configured to:

[0136] When the accuracy rate is less than the preset threshold, a model training sample set is obtained, wherein the model training sample set includes comment sample information and comment subject sample information and annotated emotion labels corresponding to the comment sample information;

[0137] The updated sample subset is added to the model training sample set, and the updated comment sample information and updated multimedia object information in the updated sample subset and the comment sample information and comment subject sample information in the model training sample set are used as the input of the emotion recognition model. The updated comment sample information and the annotated emotion labels corresponding to the comment sample information are used as the output of the emotion recognition model to train and update the emotion recognition model.

[0138] In some embodiments, the first feature extraction module 920 is specifically configured to:

[0139] Extracting text information from the image frame in the multimedia object information using optical character recognition technology to obtain the first text feature;

[0140] Automatic speech recognition technology is used to convert the speech information in the multimedia object information into text information to obtain the second text feature.

[0141] In some embodiments, the emotion recognition module 940 is specifically configured to:

[0142] Using the embedded vector layer in the emotion recognition model, the first text feature, the second text feature, the summary description text feature, and the comment text feature are respectively converted into vector features to obtain first text vector features, second text vector features, summary description text vector features, and comment text vector features;

[0143] The first text vector feature, the second text vector feature, the summary description text vector feature and the comment text vector feature are interactively fused to obtain the first interactive fusion feature, the second interactive fusion feature and the third interactive fusion feature respectively.

[0144] In some embodiments, the emotion recognition module 940 is specifically configured to:

[0145] The determined emotion tags corresponding to the text information to be identified are multiple, and the multiple emotion tags are used to represent different emotion types in the text information to be identified.

[0146] Regarding the apparatus in the above embodiment, the specific manner in which each module performs operations has been described in detail in the embodiment of the method and will not be elaborated on here. The apparatus in the above embodiment may also include other implementation methods according to the description of the method embodiment. The specific implementation methods can refer to the description of the relevant method embodiment and will not be described in detail here.

[0147] Figure 10 This is a block diagram of an electronic device for text information classification provided by an embodiment of this specification. The electronic device may be a terminal, and its internal structure diagram may be as shown in FIG. Figure 10 As shown. The electronic device includes a processor, a memory, a network interface, a display screen and an input device connected via a system bus. The processor of the electronic device is used to provide computing and control capabilities. The memory of the electronic device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the electronic device is used to communicate with an external terminal via a network connection. When the computer program is executed by the processor, a method for classifying text information is implemented. The display screen of the electronic device can be a liquid crystal display screen or an electronic ink display screen, and the input device of the electronic device can be a touch layer covering the display screen, or a button, trackball or touchpad provided on the housing of the electronic device, or an external keyboard, touchpad or mouse, etc.

[0148] Figure 11This is a block diagram of another electronic device for text information classification provided by an embodiment of this specification. The electronic device may be a server, and its internal structure diagram may be as shown in FIG. Figure 11 As shown. The electronic device includes a processor, a memory, and a network interface connected via a system bus. The processor of the electronic device is used to provide computing and control capabilities. The memory of the electronic device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and computer program in the non-volatile storage medium. The network interface of the electronic device is used to communicate with an external terminal via a network connection. When the computer program is executed by the processor, it implements a method for classifying text information.

[0149] Those skilled in the art will understand that Figure 10 or Figure 11 The structure shown in the figure is merely a block diagram of a portion of the structure related to the embodiment scheme of this specification, and does not constitute a limitation on the electronic device to which the embodiment scheme of this specification is applied. The specific electronic device may include more or fewer components than shown in the figure, or combine certain components, or have a different component arrangement.

[0150] In an exemplary embodiment, an electronic device is further provided, including: a processor; and a memory for storing instructions executable by the processor; wherein the processor is configured to execute the instructions to implement the text information classification method as described in the embodiments of this specification.

[0151] In an exemplary embodiment, a computer-readable storage medium is further provided. When instructions in the storage medium are executed by a processor of an electronic device, the electronic device is enabled to perform the text information classification method in the embodiment of this specification.

[0152] In an exemplary embodiment, a computer program product or computer program is also provided, comprising computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the text information classification method provided in the various optional implementations described above.

[0153] It is understandable that in the specific implementation of this specification, when user-related data is involved, when the above embodiments of this specification are applied to specific products or technologies, user permission or consent is required, and the collection, use and processing of relevant data must comply with relevant laws, regulations and standards of relevant countries and regions.

[0154] Those skilled in the art will appreciate that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through a computer program. The computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this specification can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0155] Those skilled in the art will readily recognize alternative embodiments of the embodiments described herein after considering the specification and practicing the inventions disclosed herein. This specification is intended to cover any variations, uses, or adaptations of the embodiments described herein that follow the general principles of the embodiments described herein and include common knowledge or customary techniques in the art not disclosed in the embodiments described herein. The description and examples are to be considered as exemplary only, with the true scope and spirit of the embodiments described herein being indicated by the following claims.

[0156] It should be understood that the embodiments of the present invention are not limited to the precise structures described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the embodiments of the present invention is limited only by the appended claims.

Claims

1. A text information classification method, characterized in that: The method comprises: Acquire text information to be identified and multimedia object information commented on by the text information to be identified; Performing text feature extraction on the image frame and the voice information in the multimedia object information respectively to obtain a first text feature and a second text feature of the multimedia object information; Inputting the image frame in the multimedia object information into a pre-established summary description generation model to obtain summary description text features of the multimedia object information; the summary description generation model is used to convert the image frame into text description information based on the content of the image frame; The first text feature, the second text feature, the summary description text feature and the comment text feature in the text information to be identified are input into a pre-created emotion recognition model, and the first text feature, the second text feature, the summary description text feature and the comment text feature are interactively fused with the emotion recognition model to obtain a first interactive fusion feature, a second interactive fusion feature and a third interactive fusion feature respectively, and based on the first interactive fusion feature, the second interactive fusion feature and the third interactive fusion feature, the emotion label corresponding to the text information to be identified is determined.

2. The method according to claim 1, characterized in that The method further comprises: According to the emotion label corresponding to the text information to be identified, obtaining multimedia object information corresponding to the text information to be identified whose emotion label is a positive label as recommended reference information of the target object corresponding to the text information to be identified; obtaining recommended information of the target object based on the recommended reference information, where the recommended information is at least one of associated information of the recommended reference information and information of the same category as the recommended reference information; Push the recommendation information to the target object.

3. The method according to claim 2, characterized in that The method further comprises: Obtaining feedback from each target object on the pushed recommendation information within a specified time range; Determining the accuracy of the recommendation information based on the feedback information; If the accuracy is less than a preset threshold, the emotion recognition model is updated.

4. The method according to claim 3, characterized in that The method further comprises: Selecting a preset number of recommendation information as updated multimedia object information at regular intervals, and selecting a specified number of comment information from the comment information of the updated multimedia object information as updated comment sample information; Annotate the selected update comment sample information and determine the annotated sentiment label corresponding to each update comment sample information; Constructing an updated sample subset based on the updated multimedia object information, the updated comment sample information, and the annotated sentiment tags corresponding to the updated comment sample information; If the accuracy is less than a preset threshold, updating the emotion recognition model includes: When the accuracy rate is less than the preset threshold, the updated sample subset is obtained, and the updated comment sample information and updated multimedia object information in the updated sample subset are used as the input of the emotion recognition model. The annotated emotion label corresponding to the updated comment sample information is used as the output of the emotion recognition model to train and update the emotion recognition model.

5. The method according to claim 4, characterized in that If the accuracy rate is less than a preset threshold, updating the emotion recognition model further includes: When the accuracy rate is less than the preset threshold, a model training sample set is obtained, wherein the model training sample set includes comment sample information and comment subject sample information and annotated emotion labels corresponding to the comment sample information; The updated sample subset is added to the model training sample set, and the updated comment sample information and updated multimedia object information in the updated sample subset and the comment sample information and comment subject sample information in the model training sample set are used as the input of the emotion recognition model. The updated comment sample information and the annotated emotion labels corresponding to the comment sample information are used as the output of the emotion recognition model to train and update the emotion recognition model.

6. The method according to claim 1, wherein The extracting text features of the image frame and the voice information in the multimedia object information respectively to obtain the first text feature and the second text feature of the multimedia object information includes: Extracting text information from the image frame in the multimedia object information using optical character recognition technology to obtain the first text feature; Automatic speech recognition technology is used to convert the speech information in the multimedia object information into text information to obtain the second text feature.

7. The method according to claim 1, characterized in that The method of using the emotion recognition model to interactively fuse the first text feature, the second text feature, the summary description text feature, and the comment text feature to obtain a first interactive fusion feature, a second interactive fusion feature, and a third interactive fusion feature, respectively, includes: Using the embedded vector layer in the emotion recognition model, the first text feature, the second text feature, the summary description text feature, and the comment text feature are respectively converted into vector features to obtain first text vector features, second text vector features, summary description text vector features, and comment text vector features; The first text vector feature, the second text vector feature, the summary description text vector feature and the comment text vector feature are interactively fused to obtain the first interactive fusion feature, the second interactive fusion feature and the third interactive fusion feature respectively.

8. The method according to claim 1, characterized in that There are multiple emotion tags corresponding to the text information to be identified, and the multiple emotion tags are used to represent different emotion types in the text information to be identified.

9. A text information classification device, characterized in that: The device comprises: The module for obtaining information to be identified is used to obtain text information to be identified and multimedia object information commented on by the text information to be identified; A first feature extraction module is configured to extract text features from image frames and voice information in the multimedia object information to obtain first and second text features of the multimedia object information; a second feature extraction module configured to input the image frames in the multimedia object information into a pre-established summary description generation model to obtain summary description text features of the multimedia object information; the summary description generation model is configured to convert the image frames into text description information based on the content of the image frames; The emotion recognition module is used to input the first text feature, the second text feature, the summary description text feature and the comment text feature in the text information to be identified into a pre-created emotion recognition model, and use the emotion recognition model to interactively fuse the first text feature, the second text feature, the summary description text feature with the comment text feature to obtain a first interactive fusion feature, a second interactive fusion feature and a third interactive fusion feature respectively, and determine the emotion label corresponding to the text information to be identified based on the first interactive fusion feature, the second interactive fusion feature and the third interactive fusion feature.

10. A text information classification system, characterized in that: The system comprises: The perception layer is configured to extract text features from the image frame and voice information in the multimedia object information commented on by the text information to be recognized, obtain first and second text features of the multimedia object information, and extract summary description text features of the image frame in the multimedia object information using a pre-established summary description generation model; the summary description generation model is configured to convert the image frame into text description information based on the content of the image frame; An understanding layer is used to interactively fuse the first text feature, the second text feature, and the summary description text feature with the comment text feature in the text information to be identified, respectively, to obtain a first interactive fusion feature, a second interactive fusion feature, and a third interactive fusion feature; The recognition layer is used to determine the emotion label corresponding to the text information to be recognized based on the first interactive fusion feature, the second interactive fusion feature and the third interactive fusion feature.

11. An electronic device, characterized in that: include: processor; a memory for storing instructions executable by the processor; The processor is configured to execute the instructions to implement the text information classification method according to any one of claims 1 to 8.

12. A computer-readable storage medium, characterized in that When the instructions in the storage medium are executed by a processor of an electronic device, the electronic device is enabled to execute the text information classification method according to any one of claims 1 to 8.

Images