Mongolian emotion recognition method based on multi-modal comparative learning

Abstract

The invention discloses a Mongolian emotion recognition method based on multi-modal comparative learning, and the method specifically comprises the steps: introducing a multi-modal open set zero sample classification technology for learning visual emotion from natural language supervision, and pre-training an image encoder and a Mongolian text encoder through a large number of image text pairs, by means of the constructed double-tower structure, a two-stage fine adjustment method is adopted, and feature alignment of the image and the Mongolian text is achieved by comparing learning loss. And in the aspect of data processing, filtering the translated image-text pair data by utilizing a Chine-CLIP model so as to obtain a high-quality image-text pair data set. According to the method, diversity and inclusiveness in the field of artificial intelligence are promoted, intelligent services are provided for Mongolian users, and application and development of related technologies in the global range are promoted at the same time.

Description

Technical Field

[0001] The present application relates to the fields of information technology and communication technology, and mainly to a Mongolian emotion recognition method based on multimodal contrastive learning. Background Art

[0002] In the field of artificial intelligence, emotion recognition has become a very challenging and valuable research area. This is especially true for multimodal emotion recognition, which involves extracting emotional signals from multiple data sources such as video, audio, and text. Developing effective multimodal emotion recognition technology is particularly important for low-resource languages ​​such as Mongolian, as this can promote diversity and inclusiveness in language technology, while also supporting practical applications such as education, media monitoring, and customer service. The problem of emotion classification is usually divided into fixed-category and open-set zero-shot classification. Fixed-category classification usually relies on a large amount of labeled training data to train deep neural networks such as convolutional neural networks (CNNs) or recurrent neural networks (RNNs) to identify standard emotion labels in images. The challenge of fixed-category classification is how to deal with non-standard expressions or culturally specific emotional expressions, which are particularly common in Mongolian videos. Open-set zero-shot classification technology enables the model to recognize emotion categories that have not been seen during the training phase, usually relying on the generalization ability of the algorithm. One of the techniques to achieve this goal is zero-shot learning, which infers unknown categories by understanding the high-level semantic properties of emotions. In addition, introducing external knowledge bases and semantic networks, such as lexical networks (WordNet) or knowledge graph networks (ConceptNet), can help the model understand the relationship between emotions and thus improve its ability to recognize new emotions.

[0003] There are two main forms of sentiment classification problems: one is fixed-category classification, and the other is open-set zero-shot classification. Fixed-category classification methods mainly rely on a large amount of labeled training data. By training deep neural networks, standard sentiment labels in images can be identified. However, this method is difficult to handle non-standard expressions in Mongolian video images or emotional expressions in a specific cultural context. On the other hand, open-set zero-shot classification technology enables the model to recognize sentiment categories that have not been seen in the training phase. This method infers unknown sentiment categories by understanding and grasping the high-level semantic attributes of sentiment. Summary of the invention

[0004] In view of the problems existing in the current prior art, the present application proposes a highly reliable programmable intelligent message processing method and device.

[0005] According to one aspect of the present invention, a Mongolian emotion recognition method based on multimodal contrastive learning is proposed, comprising:

[0006] S1. Collect Mongolian data, construct a picture-text pair data set, use the BPE byte pair encoding algorithm to segment the Mongolian data into words to generate a language model, and use the RoBERTa model structure to pre-train the language model to obtain a pre-trained model;

[0007] S2, the first stage of the pre-training model is initialized as the Chinese-CLIP image tower, and the image parameters in the training data set are frozen using a frozen image encoder to obtain image data of the frozen parameters; the high-quality text representation in the training data set is read out using a text encoder, and the image data of the frozen parameters and the high-quality text representation are optimized by contrast loss function;

[0008] S3. The second stage of the pre-trained model is initialized as a Mongolian language pre-trained RoBERTa model, and the image parameters frozen in the first stage are unfrozen using an open image encoder, and the contrast loss function is used again to perform contrast loss optimization on the image data and Mongolian text data.

[0009] Furthermore, the specific steps of collecting the image-text pair dataset include: traversing the images in the Mongolian material, combining the selected image with its paired text as a positive sample pair, and combining it with all other texts as a negative sample pair, thereby constructing the image-text pair dataset.

[0010] In the pre-training stage, Mongolian materials are collected from Mongolian news websites to construct a large-scale picture-text pair dataset. The picture-text pair dataset combines two different information sources, vision and text, so that the model can learn from images and related text descriptions at the same time, so as to understand more complex concepts and contexts, that is, multimodal information fusion.

[0011] Furthermore, the image-text pair dataset is further used to construct a training dataset. The specific steps include: using Chinese-CLIP to delete samples in the image-text pair dataset with a score lower than 0.3, less than 5 characters or more than 50 characters, and listing samples with specific text expressions as blacklisted, and adjusting the resolution of the remaining sample data to 336*336 supported by the ViT-L / 14@336px model structure to form a training dataset.

[0012] The ViT-L / 14@336px model refers to a specific variant of the Vision Transformer (ViT), where "L" stands for Large, "14" refers to the fact that the image is split into 14×14 patches, and "@336px" means that the model accepts an input image of size 336 pixels × 336 pixels.

[0013] ViT-L / 14 is a larger model configuration with more layers and larger parameters to achieve higher accuracy. The input image size is 336 pixels × 336 pixels, which can provide more detailed image information compared to the standard 224 × 224 or other sizes.

[0014] The working principle of Chinese-CLIP is to encode text and images into the same vector space through contrastive learning, so that semantically related text and images are close to each other in the vector space. In this way, the model can learn to associate text and images at the semantic level.

[0015] Furthermore, the contrast loss function adopts InfoNCE Loss, and the specific formula is expressed as:

[0016]

[0017] Among them, q represents positive samples, p represents negative samples, k represents the number of negative samples, represents the temperature coefficient, represents the i-th negative sample, i ranges from 0 to k, and exp represents the exponential function.

[0018] The contrastive loss function uses InfoNCE Loss in order to learn a common embedding space where text and images can be mapped together so that they can be retrieved or matched with each other based on semantic similarity.

[0019] At the same time, we can calculate the probability and convert these similarity scores into probability distribution using the softmax function. In this way, positive samples will have higher probability values, while negative samples will have lower probability values.

[0020] According to a second aspect of the present invention, a computer-readable storage medium is provided, on which one or more computer programs are stored. When the one or more computer programs are executed by a computer processor, the above method is implemented.

[0021] The above one or more technical solutions in the embodiments of the present application have at least one of the following technical effects:

[0022] The method of the present invention can efficiently transfer Chinese semantics to Mongolian semantics, and at the same time realize the alignment of Mongolian semantics with image semantics in the feature space. The development of Mongolian multimodal emotion recognition technology will greatly promote the diversity and inclusiveness of the field of artificial intelligence, provide Mongolian speakers with more intimate and intelligent services, and promote the application and development of related technologies on a global scale. Through continuous technological innovation and application practice, we can expect that the field of human-computer interaction will be more humane and intelligent in the future. Applying these technologies to practice, such as online customer service can improve service quality through emotion recognition, and in the field of education, teachers can adjust teaching strategies according to students' emotional state. In the field of security monitoring, emotion recognition can help predict and prevent potential conflicts and dangerous behaviors. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] The accompanying drawings are included to provide a further understanding of the embodiments and are incorporated into and constitute a part of this specification. The accompanying drawings illustrate the embodiments and are used together with the description to explain the principles of the present invention. It will be easy to recognize other embodiments and many expected advantages of the embodiments because they become better understood by reference to the following detailed description. The elements of the drawings are not necessarily to scale with each other. The same reference numerals refer to corresponding similar parts.

[0024] Figure 1 A schematic flow chart of a Mongolian emotion recognition method based on multimodal contrastive learning according to an embodiment of the present invention is shown.

[0025] Figure 2 It is a structural diagram of a computer system suitable for implementing an electronic device of an embodiment of the present application. DETAILED DESCRIPTION

[0026] The present application will be further described in detail below in conjunction with the accompanying drawings and embodiments. It is to be understood that the specific embodiments described herein are only used to explain the relevant invention, rather than to limit the invention. It should also be noted that, for ease of description, only the parts related to the relevant invention are shown in the accompanying drawings.

[0027] It should be noted that, in the absence of conflict, the embodiments and features in the embodiments of the present application can be combined with each other. The present application will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0028] Figure 1 FIG. 4 shows a flow chart of a Mongolian emotion recognition method based on multimodal contrastive learning according to an embodiment of the present invention. Figure 1 As shown:

[0029] S1. Collect Mongolian data, construct a picture-text pair data set, use the BPE byte pair encoding algorithm to segment the Mongolian data into words to generate a language model, and use the RoBERTa model structure to pre-train the language model to obtain a pre-trained model;

[0030] The specific steps of collecting the image-text pair dataset include: traversing the images in the Mongolian material, combining the selected image with its paired text as a positive sample pair, and combining it with all other texts as a negative sample pair, thereby constructing the image-text pair dataset.

[0031] In this embodiment, a Mongolian news material is extracted, for example, the extracted text is: шинэжилийнмендхургэе (translated into Chinese as: Happy New Year), the text is paired with the corresponding picture to form a positive sample, and paired with other pictures to form a negative sample.

[0032] At the same time, the sentence is segmented, and the segmentation specifically includes: word segmentation and part-of-speech tagging; word segmentation is to decompose the sentence into words, and part-of-speech tagging is to mark the part of speech of each word, such as noun, verb, etc.; in this embodiment, менд represents a verb, and жилийн represents a noun.

[0033] S2, the first stage of the pre-training model is initialized as the Chinese-CLIP image tower, and the image parameters in the training data set are frozen using a frozen image encoder to obtain image data of the frozen parameters; the high-quality text representation in the training data set is read out using a text encoder, and the image data of the frozen parameters and the high-quality text representation are optimized by contrast loss function;

[0034] Chinese-CLIP is used to delete samples with scores lower than 0.3, less than 5 characters or more than 50 characters in the image-text pair dataset, and samples with specific text expressions are blacklisted. The remaining sample data are adjusted to a resolution of 336*336 supported by the ViT-L / 14@336px model structure to form a training dataset.

[0035] In this embodiment, the image size is 336×336 pixels, the color channel is RGB, that is, 3 channels, and all samples are organized into a training data set.

[0036] In this embodiment, the text encoder converts the text into a vector representation and uses an attention mechanism or a gating mechanism to fuse the visual and text features into a multimodal vector representing the emotion.

[0037] S3. The second stage of the pre-trained model is initialized as a Mongolian language pre-trained RoBERTa model, and the image parameters frozen in the first stage are unfrozen using an open image encoder, and the contrast loss function is used again to perform contrast loss optimization on the image data and Mongolian text data.

[0038] The contrast loss function adopts InfoNCE Loss, and the specific formula is expressed as:

[0039]

[0040] Among them, q represents positive samples, p represents negative samples, k represents the number of negative samples, τ represents the temperature coefficient, and p i represents the i-th negative sample, i ranges from 0 to k, and exp represents the exponential function.

[0041] It is not enough to pre-train the Mongolian model with a frozen image encoder, because the image encoder needs to learn the image information unique to the Mongolian dataset and model the distribution of these data.

[0042] In this embodiment, when the model performs emotion recognition on шинэжилийнмендхургэе, it will output a positive emotion accompanied by a high probability value, such as 95%.

[0043] At the same time, the text feature vector to be classified is created through the text prompts output by the model; when performing forward reasoning, for the test image, the category with the greatest similarity is selected for output.

[0044] Experimental results show that the two-stage pre-training method outperforms pre-training from scratch or fine-tuning directly from a pre-trained model. Gradient accumulation techniques are used in both stages of fine-tuning to simulate larger batch sizes.

[0045] In this embodiment, the gradient accumulation technique accumulates gradients on multiple small batches of data, and then performs a weight update, specifically including:

[0046] Initialize gradients: Clear the gradient buffer to zero at the beginning of a new accumulation cycle.

[0047] Multiple forward and backward passes: For multiple consecutive mini-batches, forward and backward passes are performed to calculate the gradients, but the weights are not updated immediately. Instead, these gradients are accumulated in the gradient buffer.

[0048] Weight update: When the gradients of all specified mini-batches are accumulated, the accumulated gradients are divided by the number of mini-batches (i.e., the length of the accumulation cycle) to obtain the average gradient, and then the model weights are updated based on this average gradient.

[0049] Repeat the above process until the iteration of the entire training data set is completed.

[0050] In summary, based on the method of the present invention, a multimodal open set zero-shot classification technology for learning visual emotions from natural language supervision is introduced, a large number of image-text pairs are used to pre-train an image encoder and a Mongolian text encoder, and the dual-tower structure of this construction is used to adopt a two-stage fine-tuning method, and the contrastive learning loss (InfoNCE Loss) is used to achieve feature alignment of images and Mongolian texts. In data processing, the translated image-text pair data is filtered using the Chinese-CLIP model to obtain a high-quality image-text pair data set. This method can efficiently migrate Chinese semantics to Mongolian semantics, and at the same time achieve alignment of Mongolian semantics with image semantics in the feature space. In both stages of fine-tuning, the gradient accumulation technology is used to simulate a larger batch size. This method can obtain performance improvement of the contrastive learning model under limited hardware conditions. In the forward prediction stage, the text feature vector to be classified is created by constructing text prompts; during reasoning, for the test image, the category with the largest similarity is selected for output.

[0051] Reference below Figure 2 , which shows a schematic diagram of the structure of a computer system 200 suitable for implementing an electronic device of an embodiment of the present application. Figure 2 The electronic device shown is merely an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0052] like Figure 2 As shown, the computer system 200 includes a central processing unit (CPU) 201, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 202 or a program loaded from a storage part 208 into a random access memory (RAM) 203. In the RAM 203, various programs and data required for the operation of the system 200 are also stored. The CPU 201, the ROM 202, and the RAM 203 are connected to each other via a bus 204. An input / output (I / O) interface 205 is also connected to the bus 204.

[0053] The following components are connected to the I / O interface 205: an input section 206 including a keyboard, a mouse, etc.; an output section 207 including a liquid crystal display (LCD), etc. and a speaker, etc.; a storage section 208 including a hard disk, etc.; and a communication section 209 including a network interface card such as a LAN card, a modem, etc. The communication section 209 performs communication processing via a network such as the Internet. A drive 210 is also connected to the I / O interface 205 as needed. A removable medium 211, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 210 as needed, so that a computer program read therefrom is installed into the storage section 208 as needed.

[0054] In particular, according to an embodiment of the present disclosure, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present disclosure includes a computer program product, which includes a computer program carried on a computer-readable storage medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from the network through the communication part 209, and / or installed from the removable medium 211. When the computer program is executed by the central processing unit (CPU) 201, the above functions defined in the method of the present application are executed. It should be noted that the computer-readable storage medium of the present application can be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium can be, for example, - but not limited to - an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to, an electrical connection with one or more conductors, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present application, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, device, or device. In the present application, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, in which a computer-readable program code is carried. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. A computer-readable signal medium may also be any computer-readable storage medium other than a computer-readable storage medium, which may send, propagate, or transmit a program for use by or in conjunction with an instruction execution system, device, or device. The program code contained on the computer-readable storage medium may be transmitted using any appropriate medium, including but not limited to: wireless, wireline, optical cable, RF, etc., or any suitable combination of the foregoing.

[0055] Computer program code for performing the operations of the present application may be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages, such as Java, Smalltalk, C++, and conventional procedural programming languages, such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0056] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present application. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0057] The modules involved in the embodiments of the present application may be implemented by software or by hardware.

[0058] As another aspect, the present application also provides a computer-readable storage medium, which may be included in the electronic device described in the above embodiment; or it may exist independently and not be assembled into the electronic device. The above computer-readable storage medium carries one or more programs, and when the above one or more programs are executed by the electronic device, the electronic device: collects Mongolian data, constructs a picture-text pair data set, uses the BPE byte pair encoding algorithm to segment the Mongolian data to generate a language model, and uses the RoBERTa model structure to pre-train the language model to obtain a pre-trained model; the first stage of the pre-trained model is initialized to the Chinese-CLIP image tower, and the image parameters in the training data set are frozen using the frozen image encoder to obtain the image data of the frozen parameters; the high-quality text representation in the training data set is read out using the text encoder, and the image data of the frozen parameters and the high-quality text representation are optimized by contrast loss function; the second stage of the pre-trained model is initialized to the Mongolian language pre-trained RoBERTa model, and the image parameters frozen in the first stage are unfrozen using the open image encoder, and the image data and Mongolian text data are optimized by contrast loss again using the contrast loss function.

[0059] The above description is only a preferred embodiment of the present application and an explanation of the technical principles used. Those skilled in the art should understand that the scope of the invention involved in the present application is not limited to the technical solution formed by a specific combination of the above technical features, but should also cover other technical solutions formed by any combination of the above technical features or their equivalent features without departing from the above invention concept. For example, the above features are replaced with the technical features with similar functions disclosed in this application (but not limited to) by each other to form a technical solution.

Claims

1. A Mongolian emotion recognition method based on multimodal contrastive learning, characterized in that: The following steps are involved: S1. Collect Mongolian data, construct a picture-text pair data set, use the BPE byte pair encoding algorithm to segment the Mongolian data into words to generate a language model, and use the RoBERTa model structure to pre-train the language model to obtain a pre-trained model; S2, the first stage of the pre-training model is initialized as the Chinese-CLIP image tower, and the image parameters in the training data set are frozen using a frozen image encoder to obtain image data of the frozen parameters; the high-quality text representation in the training data set is read out using a text encoder, and the image data of the frozen parameters and the high-quality text representation are optimized by contrastive learning loss function; S3. The second stage of the pre-trained model is initialized as a Mongolian language pre-trained RoBERTa model, and the image parameters frozen in the first stage are unfrozen using an open image encoder, and the contrastive learning loss function is used again to perform contrastive learning loss optimization on the image data and the Mongolian text data.

2. The Mongolian emotion recognition method according to claim 1, characterized in that: The specific steps of collecting the image-text pair dataset include: traversing the images in the Mongolian material, combining the selected image with its paired text as a positive sample pair, and combining it with all other texts as a negative sample pair, thereby constructing the image-text pair dataset.

3. The Mongolian language emotion recognition method according to claim 1 or 2, characterized in that: The image-text pair dataset is used to further construct a training dataset. The specific steps include: using Chinese-CLIP to delete samples with scores lower than 0.3, less than 5 characters or more than 50 characters in the image-text pair dataset, and listing samples with specific text expressions as blacklists. The remaining sample data is adjusted to a resolution of 336*336 supported by the ViT-L / 14@336px model structure to form a training dataset.

4. The Mongolian language emotion recognition method according to claim 1, characterized in that: The contrastive learning loss function adopts InfoNCE Loss, and the specific formula is expressed as: Among them, q represents positive samples, p represents negative samples, k represents the number of negative samples, τ represents the temperature coefficient, and p i represents the i-th negative sample, i ranges from 0 to k, and exp represents the exponential function.

5. A computer program product, characterized in that A computer program is stored thereon, and when the computer program is executed by a processor, the method according to any one of claims 1 to 4 is implemented.

6. A computing system, characterized in that: The method comprises a processor and a memory, wherein the processor is configured to execute the method according to any one of claims 1 to 4.

Images