Method and system for automatic detection and interpretation of semantic changes in social media network language

Abstract

The application discloses a kind of social media network slang semantic change automatic detection and interpretation method and system, two stages of entire scheme, through two stages automatic detection and understanding social media comment in network slang semantic change phenomenon, in the first stage, by the distance of the corresponding vector representation of word in different corpus, can find the word that occurs changes in semantics (i.e. Network slang);Second stage, use multi-modal information can generate network slang explanation text, so as to accurately translate the real meaning of network slang.

Description

Technical Field

[0001] This invention relates to the field of social media analytics, and more particularly to a method and system for automatically detecting and interpreting semantic changes in social media online terminology. Background Technology

[0002] With the rise of online social media platforms such as TikTok and Bilibili, it has become incredibly convenient for users to share their personal opinions and feelings while browsing videos. A fascinating phenomenon can be observed in the numerous video comments: many users have created a variety of semantically modified internet slang terms to express their unique thoughts or emotions. Clearly, for new users unfamiliar with these subcultures, understanding the meanings expressed by these internet slang terms is very difficult, leading to a poor user experience. Therefore, for platform administrators, providing a tool to automatically detect these semantically modified internet slang terms and offer standardized semantic interpretations is of significant practical value.

[0003] However, fully understanding the information conveyed by these subcultures remains a highly challenging task for intelligent systems. In recent years, with the rapid development of natural language processing technology, some techniques have achieved preliminary detection of semantic changes in words throughout historical records. However, these methods lack the ability to effectively integrate multimodal information, thus failing to capture relevant information about internet slang in videos. Furthermore, they do not consider the subsequent task of standardizing the language interpretation of detected semantically changed words, severely limiting the application scope of these technologies. Worse still, the semantics of these internet slang terms may subtly change depending on different visual contexts. Therefore, if the visual context of the usage scenarios of internet slang cannot be fully understood, and a general dictionary is simply constructed, it may be impossible to accurately translate the true meaning of internet slang, and may even lead to misunderstandings of subtle semantic changes by users. Clearly, a more effective and accurate detection and interpretation method that combines multimodal semantic information is needed. Summary of the Invention

[0004] The purpose of this invention is to provide a method and system for automatically detecting and interpreting semantic changes in social media online slang, which can accurately detect online slang and generate corresponding explanatory text by combining multimodal information.

[0005] The objective of this invention is achieved through the following technical solution:

[0006] An automatic detection and interpretation method for semantic changes in social media online terminology includes:

[0007] Internet slang detection phase: For the social media comment corpus C u Social media comment texts in the text, through a given general corpus C gThe corresponding word vector representation set is used to obtain the vector representations of all words in the social media comment text, forming the first text vector sequence, and the vector representations are obtained from the social media comment corpus C. u The corresponding set of word vector representations is used to obtain the vector representations of all words in the social media comment text, forming a second text vector sequence; the distance between the vector representations of the same words in the first and second text vector sequences is measured, and the K words with the largest distances are selected from the social media comment text as online slang;

[0008] The explanation stage of internet slang involves encoding the visual representation of the video frame image corresponding to the social media comment text, the textual representation of internet slang and social media comment text, and the speech representation of internet slang and social media comment text using an encoder based on the Transformer model. The encoded image features of the video frame image, the textual representation of internet slang and social media comment text, and the speech features of internet slang and social media comment text are then input into the decoder based on the Transformer model to generate natural language explanation text.

[0009] An automatic detection and interpretation system for semantic changes in social media online terminology includes:

[0010] An image-enhanced web slang detection module is applied to the web slang detection stage, which includes: processing a social media comment corpus C. u Social media comment texts in the text, through a given general corpus C g The corresponding word vector representation set is used to obtain the vector representations of all words in the social media comment text, forming the first text vector sequence, and the vector representations are obtained from the social media comment corpus C. u The corresponding set of word vector representations is used to obtain the vector representations of all words in the social media comment text, forming a second text vector sequence; the distance between the vector representations of the same words in the first and second text vector sequences is measured, and the K words with the largest distances are selected from the social media comment text as online slang;

[0011] The Internet slang explanation task module is applied to the Internet slang explanation stage. The Internet slang explanation stage includes: encoding the visual representation of the video frame image corresponding to the social media comment text, the text representation of the Internet slang and the social media comment text, and the speech representation of the Internet slang and the social media comment text respectively through an encoder based on the Transformer model; inputting the encoded image features of the video frame image, the text representation of the Internet slang and the social media comment text, and the speech features of the Internet slang and the social media comment text into the decoder based on the Transformer model to generate natural language explanation text.

[0012] A processing device includes: one or more processors; and a memory for storing one or more programs;

[0013] When the one or more programs are executed by the one or more processors, the one or more processors implement the aforementioned method.

[0014] A readable storage medium storing a computer program that, when executed by a processor, implements the aforementioned method.

[0015] As can be seen from the technical solution provided by the present invention, the two-stage approach can automatically detect and understand the semantic changes of online terms in social media comments. In the first stage, by measuring the distance between the vector representations of words in different corpora, words that have undergone semantic changes (i.e., online terms) can be found. In the second stage, multimodal information can be used to generate explanatory texts of online terms, thereby accurately translating the true meaning of online terms. Attached Figure Description

[0016] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the following description of the embodiments will be briefly introduced. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0017] Figure 1 A flowchart illustrating an automatic detection and interpretation method for semantic changes in social media online terminology provided in this embodiment of the invention;

[0018] Figure 2 This is an overall framework diagram of a method for automatically detecting and interpreting semantic changes in social media network terms provided in an embodiment of the present invention;

[0019] Figure 3 A schematic diagram of an automatic detection and interpretation system for semantic changes in social media network terms provided in an embodiment of the present invention;

[0020] Figure 4 This is a schematic diagram of a processing device provided in an embodiment of the present invention. Detailed Implementation

[0021] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the protection scope of the present invention.

[0022] First, the following explanations are provided for the terms that may be used in this article:

[0023] The terms “including,” “comprising,” “containing,” “having,” or other similar semantic descriptions should be interpreted as non-exclusive inclusion. For example, “including a technical feature element (such as raw material, component, ingredient, carrier, dosage form, material, size, part, component, mechanism, device, step, process, method, reaction conditions, processing conditions, parameter, algorithm, signal, data, product or article of manufacture, etc.)” should be interpreted as including not only the expressly listed technical feature element, but also other technical feature elements that are not expressly listed and are well-known in the art.

[0024] The following is a detailed description of a method and system for automatically detecting and interpreting semantic changes in social media online terminology provided by this invention. Contents not described in detail in the embodiments of this invention belong to prior art known to those skilled in the art. Where specific conditions are not specified in the embodiments of this invention, they should be performed according to conventional conditions in the art or conditions recommended by the manufacturer.

[0025] Example 1

[0026] This invention provides an automatic detection and interpretation method for semantic changes in social media online terminology, such as... Figure 1 As shown, it mainly includes the following two stages:

[0027] I. Detection of Internet Slang

[0028] In this phase, for the social media comment corpus C u The text of social media comments (i.e., the text to be detected) is processed through a given general corpus C. g The corresponding word vector representation set is used to obtain the vector representations of all words in the social media comment text, forming the first text vector sequence, and the vector representations are obtained from the social media comment corpus C. u The corresponding set of word vector representations is used to obtain the vector representations of all words in the social media comment text, forming a second text vector sequence. The distance between the vector representations of the same words in the first and second text vector sequences is measured, and the K words with the largest distances are selected from the social media comments as online slang.

[0029] In this embodiment of the invention, the word vector representation set contains vector representations of multiple words. The vector representation of each word is extracted using a pre-trained model for the text. For the word w′, its vector representation is v, and the extraction method is as follows:

[0030] v = W g w

[0031] Among them, Wg Denote the parameters of the pre-trained model for text as, and denote the one-hot vector of word w′ as w.

[0032] Denote the set of corresponding word vector representations of the general corpus C g as where denotes the representation vector of the l-th word in the general corpus C g , l = 1, 2, …; Denote the set of corresponding word vector representations of the social media comment corpus C u as where denotes the representation vector of the t-th word in the social media comment corpus C u , t = 1, 2, ….

[0033] In an embodiment of the present invention, denote the first text vector sequence as Denote the second text vector sequence as where n is the number of words in the social media comment text, i is the serial number of the word in the social media comment text, denotes the vector representation of the i-th word obtained through the set of corresponding word vector representations of the given general corpus C g , denotes the vector representation of the i-th word obtained through the set of corresponding word vector representations of the social media comment corpus C u ; If the i-th word does not appear in the given general corpus C g , then the corresponding vector representation is the average of the representation vectors of all characters constituting the i-th word.

[0034] Those skilled in the art can understand that when obtaining Chinese word representations through a pre-trained model for text (for example, glove), representations of single characters will also be obtained. For example, when obtaining the representation of the word "hello", representations of the two characters "你" and "好" will also be obtained separately. Generally, if it can be determined that it is a word and there is a corresponding word representation, it can be directly used; otherwise, the average of all character representations is used.

[0035] Measure the distance between the vector representations and with the same serial number i in the first text vector sequence and the second text vector sequence through a distance function.

[0036] In an embodiment of the present invention, the network term detection stage is implemented through an image-enhanced network term detection module, and the image-enhanced network term detection module includes: a pre-trained model for text, a pre-trained model for images, an encoder, a decoder, and an extractor; where:

[0037] The pre-trained model for text is used to extract from a given general corpus C. g and the social media comment corpus C u The corresponding set of word vector representations;

[0038] The extractor is used to combine social media comment text to obtain a first text vector sequence and a second text vector sequence, measure the distance between the vector representations of the same words in the first text vector sequence and the second text vector sequence, and select the K words with the largest distance from the social media comment text as online slang.

[0039] The pre-trained model, encoder, and decoder for images are only used in the training phase. During training, the encoder takes a second text vector sequence as input and outputs a sentence-level vector representation. The decoder generates reconstructed text based on the sentence-level vector representation and calculates the reconstruction loss by using the difference between the reconstructed text and the social media comment text. The pre-trained model for images takes video frame images corresponding to social media comment text as input and outputs the overall visual representation. After transforming the sentence-level vector representation from the sentence semantic space to the visual space using an orthogonal matrix G, the distance loss between the vector representation and the overall visual representation is calculated using a distance function. Combined with reconstruction loss Distance loss The first training loss is constructed using the constraint terms of the orthogonal matrix G, and the pre-trained model, encoder, and decoder for text in the image-enhanced network language detection module are trained using the first training loss.

[0040] During the testing phase, the pre-trained model, encoder, and decoder for images are removed. Instead, a pre-trained model and extractor for text are used to detect online language. Typically, the social media comment text input for both the training and testing phases comes from the social media comment corpus C. u .

[0041] II. Explanation of Internet Slang

[0042] In this stage, an encoder based on the Transformer model encodes the visual representation of the video frame image corresponding to the social media comment text, the textual representation of the internet slang and social media comment text, and the speech representation of the internet slang and social media comment text. The encoded image features of the video frame image, the textual representation of the internet slang and social media comment text, and the speech features of the internet slang and social media comment text are then input into the decoder based on the Transformer model to generate natural language explanation text.

[0043] In this embodiment of the invention, the internet slang explanation stage is implemented through an internet slang explanation task module, which includes: a Transformer-based encoder and a Transformer-based decoder; during the training stage, natural language explanation text and explanation text in the dataset are generated based on the Transformer-based decoder, and cross-entropy loss is calculated. Furthermore, a macroscopic text representation is obtained by processing text features through an average pooling layer, and an alignment loss is calculated based on the distance difference between the macroscopic text representation and the overall visual representation. Combined with cross-entropy loss Alignment loss A second training loss is constructed, and the network is trained using the second training loss to explain the task module; wherein, the overall visual representation is part of the visual representation of the video frame image (details will be introduced later).

[0044] In this embodiment of the invention, the encoder based on the Transformer model includes three encoding units that process information from the visual modality, text modality, and speech modality, respectively.

[0045] For visual modalities, namely video frame images corresponding to social media comment text, a pre-trained model for images is used to obtain an overall visual representation and a series of local visual representations, which constitute the visual representation of the video frame image. This representation is then input into the first encoding unit in the encoder based on the Transformer model to obtain visual features.

[0046] For the text modality, internet slang and social media comment text are concatenated, and a delimiter is set in the concatenation part to obtain a text sequence. Using the pre-trained model for text, the vector representation of each word in the text sequence is obtained, which constitutes the text representation of internet slang and social media comment text. This representation is then input into the second encoding unit in the encoder based on the Transformer model to obtain text features.

[0047] For the speech modality, each word is translated into its corresponding phonetic symbol. The phonetic symbols of words in online slang are concatenated with the phonetic symbols of words in social media comment text, and a separator is set in the concatenation part to obtain a phonetic symbol sequence. A variable linear layer is used to obtain the feature representation of each phonetic symbol in the phonetic symbol sequence, which constitutes the speech representation of online slang and social media comment text. This representation is then input into the third encoding unit in the encoder based on the Transformer model to obtain speech features.

[0048] In this embodiment of the invention, by setting a separator, the internet slang explanation task module can learn during training that it is a special symbol used to separate internet slang from the social media comment text in which it appears.

[0049] The encoder based on the Transformer model outputs image features, text features, and speech features. The three types of features are concatenated and fused using an attention mechanism. Then, the decoder based on the Transformer model uses an autoregressive approach to generate words at each time step. The words are combined in time step order to form a natural language explanatory text.

[0050] To more clearly demonstrate the technical solution and its effects provided by the present invention, the following describes in detail a method for automatic detection and interpretation of semantic changes in social media network terms provided by the present invention, using specific embodiments.

[0051] I. Problem Definition.

[0052] Given a general corpus C g And a corpus of user-generated social media comments, C u A word w′ is defined as internet slang when it meets the following two conditions:

[0053] Condition 1: The word w′ or its variants exist simultaneously in C g and C u In, that is, w′∈C g ∩C u In other words, condition one is satisfied only when a word exists in both a general corpus and a social media comment corpus, and generally, condition one will be satisfied.

[0054] Condition 2: C u The meaning of the word w′ in C should be different from that of the word or phrase in C. g The variant in the text, namely, given the distance function Dis(.), changes significantly compared to the original meaning vector representation, i.e., Dis(v g ,v u ) is relatively large, where v g and v u Is w′ in C g and C u The corresponding vector representation in the diagram.

[0055] Considering that the general corpus refers to the corpus used in daily life, in fact, it is impossible to cover all possible words in any case. For example, the word "Tsinghua University" may not be included in the data that may be used to train the general corpus representation, but this word obviously exists in daily use. Therefore, in normal processing, the average value of characters is used to represent these words that may not be covered exactly. Similarly, for the above reasons, because the general corpus is too large to be completely covered, the present invention defaults that all words exist in the general corpus. For the social media comment corpus, when detecting, the detected sentences all come from social media, so it is also defaulted that these words all belong to the social media comment corpus. Based on the above reasons, the sentences obtained in the invention are defaulted to meet Condition 1.

[0056] Those skilled in the art can understand that a variant means that intuitively two different words represent the same meaning, one of which is the standard word and the other is a variant of the standard word. For example, "elder brother" and "old brother" represent the same meaning, "elder brother" is the standard word, and "old brother" is a variant of "elder brother".

[0057] II. Semantic Change Automatic Detection and Explanation Framework.

[0058] As Figure 2 shown, it shows the semantic change automatic detection and explanation framework. Among them, the upper dotted box part is the image-enhanced network term detection module described above, which is used for semantic change automatic detection; the lower dotted box part is the network term explanation task module described above, which is used to generate the explanation text of network terms. It should be noted that Figure 2 The text content presented in the upper left part and the content of the video frame image are only for illustration. The following will introduce the upper and lower parts in detail.

[0059] 1. Image-enhanced Network Term Detection Module.

[0060] In the embodiment of the present invention, the image-enhanced network term detection module is an unsupervised module, which learns the true meaning of each word in the social media comment sentence by iteratively reconstructing the text sequence. Then, by reducing the distance between the sentence representation and the image representation, visual information is extracted and integrated. In this way, a well-trained detector will obtain the word representation in the real context, and compare the word feature representation obtained by training with the word representation obtained by training in the corresponding general corpus to find the words with semantic changes (i.e., network terms). The following will introduce the training method of this module.

[0061] First, it is necessary to initialize the representation of words. For the general corpus C gThe text uses a pre-trained model to obtain the vector representation of each word; for example, a pre-trained Chinese GloVe model can be used. Meanwhile, data from the social media comment corpus C... u The words are also initialized using the same feature format, i.e., input into a pre-trained model for text to obtain a vector representation of each word. More formally, this is done by taking a sentence (i.e., social media comment text) from a corpus of social media comments. Each word is represented as a pre-trained universal vector, all of which are trainable:

[0062]

[0063] Among them, W glove These are the trainable parameters of the linear layer, obtained after initialization by the GloVe model. This is the one-hot vector of the word. If the word does not appear in the general corpus, then... It is set to the average value of the representation vectors of all characters that make up the i-th word, which is the average value of the representation vectors of the characters that make up the word.

[0064] In this embodiment of the invention, for the general corpus C g and the social media comment corpus C u By using different pre-trained models specifically for text, a general corpus C can be obtained through training. g The corresponding pre-trained model for text, and the social media comment corpus C u The corresponding pre-trained models for text; the two pre-trained models for text work on the same principle and are trained in the same way. The following example uses the social media comment corpus C. u Taking this as an example, we will introduce the training process of a pre-trained model for text.

[0065] Following word encoding, sentence representation is introduced. Since sentence representation is a typical type of sequence data, a feature vector based on an encoder-decoder structure is chosen. The encoder reads the text vector sequence and transforms it into the sentence hidden layer state h. s This is used as a sentence vector (a vector representation at the sentence level). Then, the decoder reconstructs the original sentence based on this semantic vector. The computational process can be represented as follows:

[0066]

[0067] z i =Decoder(w1,w2,…,w i-1 ,h s )

[0068]

[0069] Where Encoder represents the encoder. w1, w2, ..., w represents the vector representation of all words in the sentence; Decoder represents the decoder. i-1 Let z represent the 1st, 2nd, and (i-1th)th reconstructed words. i It is the intermediate feature obtained when decoding the i-th word output by the decoder, f(.) is the activation function, and W is the intermediate feature obtained when decoding the i-th word output by the decoder. out and b out These are the parameters of the output layer. This represents the probability that the i-th word predicted by the output layer is any word in the vocabulary.

[0070] Next, the reconstruction loss is calculated. The reconstruction loss is the sum of the negative log-likelihoods of the correct words at each step, as shown below:

[0071]

[0072] After obtaining the semantic vector of a sentence, the true representation of each word can be learned through the process of reconstructing the text. However, simply using sentence reconstruction to learn representations ignores an important assumption: sentences appearing in the same video frame usually contain similar topics. To address this limitation, this invention proposes another loss that measures the distance between the sentence semantics and the video frame semantics. It is worth noting that the reconstruction loss and the distance loss are somewhat contradictory. This is because as the reconstruction loss decreases, the semantic vector of the sentence becomes more diverse, which may hinder the reduction of the distance loss. Specifically, after obtaining the semantic vector of the sentence, the visual information is transformed into a semantic representation of the image, called the overall visual representation, through processing by a pre-trained model for the image (e.g., a pre-trained Vision Transformer model) and pooling layers. Here, the visual vectors are fixed to obtain an invariant image semantic space, meaning the parameters of the pre-trained model for the image are not updated. Then, an orthogonal matrix G is used as an efficient transformation from the sentence semantic space to the image. The final distance loss formula is as follows:

[0073]

[0074] For example, the distance function dist(.) can select Euclidean distance.

[0075] Finally, the first training loss is shown below:

[0076]

[0077] Where α1 and β1 are both hyperparameters, T is the matrix transpose symbol, and the third term β1‖G T GI‖ F This is used to constrain the orthogonal matrix G. Due to the antagonistic relationship, the degree of their aggregation can be controlled by adjusting the hyperparameter α1, which is a key factor affecting the final performance.

[0078] After several training periods (e.g., two periods), image information can be incorporated into the web language detection module. Similarly, the general corpus C... g The corresponding pre-trained models for text are also trained using the above method. Finally, through the two pre-trained models for text, the vector representation sets of each word in the corresponding corpus are obtained. Then, the vector representations of each word in the social media comment text can be analyzed. sum vector representation Words with semantic changes are detected by sorting them by distance, and the K words with the largest distances are selected as internet slang.

[0079] In this embodiment of the invention, training a pre-trained model for text can be understood as updating the vector representations of each word, where each word's vector representation is the parameter of the pre-trained model for text. If the two vector representations corresponding to the same word differ significantly, it is identified as internet slang.

[0080] 2. Internet slang explanation task module.

[0081] In this embodiment of the invention, the internet slang explanation task module is a speech-enhanced Transformer Explanation Model (ProTE). In this part, ProTE takes the internet slang filtered by the detection module as input, and also takes the original corresponding video frame and social media comment text as input. First, the representation of multimodal contextual information in the context of the internet slang is introduced, which consists of three parts: the corresponding image or video frame, the real-time comment containing the internet slang, and the pronunciation corresponding to the real-time comment.

[0082] For video frame images, a pre-trained model for images was also used, mainly to extract the overall visual representation and a series of local visual representations, denoted as... in, For the overall visual representation, A series of local visual representations are given, where m is the number of image patches, and each image patch corresponds to a local visual representation. These representations are then input into an encoder based on the Transformer model.

[0083] like Figure 2As shown, the encoder based on the Transformer model contains three encoding units, which process information from the visual modality, text modality, and speech modality, respectively. The three encoding units have the same structure, each containing a self-attention module and a forward propagation network connected in sequence. Specifically, the visual representation... The data is first adjusted using a linear transformation and an average pooling layer before being input into the first encoding unit.

[0084] The overall processing flow for video frame images is represented as follows:

[0085]

[0086] FFN1(x) = max(0, xW1+b1)W2+b2

[0087] Where I represents a video frame image, ViT represents a pre-trained model for the image, and the VisionTransformer (Vit) model is used as an example here. AvgPool represents the operation of the average pooling layer, W I and b I These are the trainable parameters of the linear transformation layer. denoted as the visual representation obtained after adjustment by linear transformation and average pooling layer, SelfAttn1 represents the self-attention module in the first coding unit, FFN1 represents the forward propagation network in the first coding unit, and the formula FFN1(x) shows the processing of the forward propagation network, where x is the output of the self-attention module SelfAttn1. It's a visual feature, the first one. These are the macroscopic features of a video frame image, while each of the other items represents a local feature of an image patch.

[0088] For the text modality, internet slang and social media comment text are first concatenated, using a special symbol (e.g., [SEP]) as a separator. Each word is encoded into a vector through a trainable linear layer. After the initial encoding, the second encoding unit of the feature is mapped to the hidden state H. T The calculation process is as follows:

[0089]

[0090] Among them, w r Let be the unique vector of the r-th word in the concatenated text sequence, and R be the total number of words in the text sequence (i.e., the total number of words in internet slang and social media comment text). Here, represents the character representation matrix; SelfAttn2 represents the self-attention module in the second coding unit, FFN2 represents the forward propagation network in the second coding unit, and the hidden state H...T This refers to textual features.

[0091] For the speech modality, analysis of social media comments reveals a significant proportion of homophones or near-homophones in user-created content. To leverage common homophones in online slang, this invention introduces pronunciation as an additional modality to enhance the model's capabilities. Specifically, in this invention, each word in the online slang and social media comment text is translated into its corresponding phonetic transcription. Then, similar to the process applied in the text modality, the phonetic transcriptions of the online slang and social media comment text are concatenated to form a speech representation, denoted as P = (p1, p2, ... p...). R ), p r These are the phonetic symbols of the r-th word; then, they are converted into feature representations through a variable linear layer. Finally, each of their feature representations will be converted into a hidden representation H. P It should be noted that both text and speech symbols are encoded with positional codes before being processed by the self-attention module. The calculation process is as follows:

[0092]

[0093] in, Here is the pronunciation representation matrix; SelfAttn3 represents the self-attention module in the third coding unit, FFN3 represents the forward propagation network in the third coding unit, and the hidden representation H... P This refers to speech features.

[0094] It should be noted that, for ease of representation, the above text representation... With the speech representation P=(p1,p2,...p R The separator is omitted in the original text. Furthermore, in language processing, different characters in different positions can have different meanings, but the self-attention module does not consider the position of the text. Therefore, positional encodings are added to both text and speech representations before they are processed by the corresponding self-attention modules to provide positional information. The positional encoding used here is consistent with the positional encoding method commonly used in transformer-based models.

[0095] Through the aforementioned processing, three parts of features are obtained from the outputs of the three coding units. These three parts of features are concatenated and then processed by a Transformer module. Figure 2 The components of the bottom Transformer are used to apply a self-attention mechanism to fuse their representations and information. Finally, a Transformer-based decoder is utilized. Figure 2 The components of the Transformer at the bottom are used to generate natural language interpreted text.

[0096]

[0097] P(e t ) = softmax(y t W proj )

[0098] Among them, W proj It is a linear mapping matrix, Transformer represents the Transformer module, y t Let represent the intermediate features of the t-th word. and Let P(e) represent the first and (t-1)th words generated by the model. t ) indicates that the t-th word is generated as e t The probability is calculated, and the word with the highest probability is selected as the t-th word generated by the model.

[0099] During the training phase, the output of the Transformer-based decoder is used to guide language generation using the cross-entropy loss function. Formally, it can be described as follows:

[0100]

[0101] Where K is the number of explanations in the dataset, n k It is the number of words in the k-th explanatory text.

[0102] However, due to the significant semantic gap between online slang and its corresponding images, models struggle to capture useful regional or global information, as comments typically focus on only one or two regions. To address this limitation and further explore visual information, a macroscopic alignment loss function for text and images is constructed to enhance the model's ability to bridge the semantic gap between online slang text and images. Specifically, an average pooling layer is used to process text features H. T And obtain the macroscopic text representation h T Measuring the overall representation of macroscopic text and images. The distance between them, as the alignment loss

[0103]

[0104] Through this supervision, the model can gain a deeper understanding of visual information and select the correct information guided by text. Finally, the second training loss is defined as follows:

[0105]

[0106] Here, α2 is a hyperparameter used to balance the two losses.

[0107] After training, natural language explanation text is generated using a Transformer-based decoder, as described above.

[0108] Example 2

[0109] This invention also provides an automatic detection and interpretation system for semantic changes in social media online terminology, which is mainly based on the method provided in the foregoing embodiments, such as... Figure 3 As shown, the system mainly includes:

[0110] An image-enhanced web slang detection module is applied to the web slang detection stage, which includes: processing a social media comment corpus C. u Social media comment texts in the text, through a given general corpus C g The corresponding word vector representation set is used to obtain the vector representations of all words in the social media comment text, forming the first text vector sequence, and the vector representations are obtained from the social media comment corpus C. u The corresponding set of word vector representations is used to obtain the vector representations of all words in the social media comment text, forming a second text vector sequence; the distance between the vector representations of the same words in the first and second text vector sequences is measured, and the K words with the largest distances are selected from the social media comment text as online slang;

[0111] The Internet slang explanation task module is applied to the Internet slang explanation stage. The Internet slang explanation stage includes: encoding the visual representation of the video frame image corresponding to the social media comment text, the text representation of the Internet slang and the social media comment text, and the speech representation of the Internet slang and the social media comment text respectively through an encoder based on the Transformer model; inputting the encoded image features of the video frame image, the text representation of the Internet slang and the social media comment text, and the speech features of the Internet slang and the social media comment text into the decoder based on the Transformer model to generate natural language explanation text.

[0112] Those skilled in the art will understand that, for the sake of convenience and brevity, the above-described division of functional modules is used as an example. In practical applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the system can be divided into different functional modules to complete all or part of the functions described above.

[0113] Example 3

[0114] The present invention also provides a processing device, such as Figure 4As shown, it mainly includes: one or more processors; a memory for storing one or more programs; wherein, when the one or more programs are executed by the one or more processors, the one or more processors implement the method provided in the foregoing embodiments.

[0115] Furthermore, the processing device also includes at least one input device and at least one output device; in the processing device, the processor, memory, input device, and output device are connected via a bus.

[0116] In this embodiment of the invention, the specific types of the memory, input device, and output device are not limited; for example:

[0117] Input devices can be touchscreens, image acquisition devices, physical buttons, or mice, etc.

[0118] The output device can be a display terminal;

[0119] The memory can be random access memory (RAM) or non-volatile memory, such as disk storage.

[0120] Example 4

[0121] The present invention also provides a readable storage medium storing a computer program that, when executed by a processor, implements the method provided in the foregoing embodiments.

[0122] In this embodiment of the invention, the readable storage medium is a computer-readable storage medium and can be disposed in the aforementioned processing device, for example, as a memory in the processing device. Furthermore, the readable storage medium can also be any medium capable of storing program code, such as a USB flash drive, portable hard drive, read-only memory (ROM), magnetic disk, or optical disk.

[0123] The above description is merely a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A method for automatically detecting and interpreting semantic changes in social media online terminology, characterized in that, include: Internet slang detection phase: For the social media comment corpus C u Social media comment texts in the text, through a given general corpus C g The corresponding word vector representation set is used to obtain the vector representations of all words in the social media comment text, forming the first text vector sequence, and the vector representations are obtained from the social media comment corpus C. u The corresponding set of word vector representations is used to obtain the vector representations of all words in the social media comment text, forming a second text vector sequence; the distance between the vector representations of the same words in the first and second text vector sequences is measured, and the K words with the largest distances are selected from the social media comment text as online slang; The explanation stage of internet slang: The visual representation of the video frame image corresponding to the social media comment text, the text representation of internet slang and social media comment text, and the speech representation of internet slang and social media comment text are encoded by an encoder based on the Transformer model. The image features of the video frame image, the text representation of internet slang and social media comment text, and the speech features of internet slang and social media comment text are then input into the decoder based on the Transformer model to generate natural language explanation text. The word vector representation set contains vector representations of multiple words. The vector representation of each word is extracted using a pre-trained model for the text. For word w′, its vector representation is v, and the extraction method is as follows: v=W g w Among them, W g This represents the parameters of the pre-trained model for the text, where w represents the one-hot vector of word w′. The online slang detection stage is implemented through an image-enhanced online slang detection module, which includes: a pre-trained model for text, a pre-trained model for images, an encoder, a decoder, and an extractor; wherein: the pre-trained model for text is used to extract from a given general corpus C. g and the social media comment corpus C u The corresponding set of word vector representations; the extractor is used to combine social media comment text to obtain a first text vector sequence and a second text vector sequence, measure the distance between the vector representations of the same words in the first text vector sequence and the second text vector sequence, and select the K words with the largest distance from the social media comment text as online slang; the pre-trained model, encoder, and decoder for images are only applied to the training phase. During the training phase, the encoder's input is the second text vector sequence, and its output is a sentence-level vector representation. The decoder generates reconstructed text based on the sentence-level vector representation and calculates the reconstruction loss using the difference between the reconstructed text and the social media comment text. The pre-trained model for images takes video frame images corresponding to social media comment text as input and outputs the overall visual representation. After transforming the sentence-level vector representation from the sentence semantic space to the visual space using an orthogonal matrix G, the distance loss between the vector representation and the overall visual representation is calculated using a distance function. Combined with reconstruction loss Distance loss The first training loss is constructed using the constraint terms of the orthogonal matrix G. This first training loss is used to train the text-based pre-trained model, encoder, and decoder in the image-enhanced web terminology detection module. During the testing phase, the image-based pre-trained model, encoder, and decoder are removed, and the text-based pre-trained model and extractor are used to detect web terms. The web terminology explanation phase is implemented through a web terminology explanation task module, which includes a Transformer-based encoder and a Transformer-based decoder. During the training phase, natural language explanation text and explanation text from the dataset are generated based on the Transformer-based decoder, and the cross-entropy loss is calculated. Furthermore, a macroscopic text representation is obtained by processing text features through an average pooling layer, and an alignment loss is calculated based on the distance difference between the macroscopic text representation and the overall visual representation. Combined with cross-entropy loss Alignment loss A second training loss is constructed, and the network is trained using the second training loss to explain the task module; wherein the overall visual representation is part of the visual representation of the video frame image.

2. The method for automatic detection and interpretation of semantic changes in social media online terminology according to claim 1, characterized in that, The general corpus C g The corresponding set of word vector representations is denoted as in, Represents the general corpus C g The representation vector of the l-th word in the text, l = 1, 2, ...; and the social media comment corpus C u The corresponding set of word vector representations is denoted as in, The social media comment corpus C u The representation vector of the t-th word in the text, where t = 1, 2, ...

3. The method for automatic detection and interpretation of semantic changes in social media online terminology according to claim 2, characterized in that, The measurement of the distance between the vector representations of the same words in the first text vector sequence and the second text vector sequence includes: Let the first text vector sequence be denoted as Let the second text vector sequence be denoted as Where n is the number of words in the social media comment text, and i is the index of the word in the social media comment text. This indicates that the given general corpus C is used. g The corresponding word vector representation is the vector representation of the i-th word obtained from the set. This indicates that the data was obtained from the social media comment corpus C. u The corresponding word vector representation is the vector representation of the i-th word obtained from the set; if the i-th word does not appear in the given general corpus C g In the middle, the corresponding vector representation It is the average of the representation vectors of all the characters that make up the i-th word; The vector representations of the same index i in the first and second text vector sequences are obtained by using a distance function. and The distance between them is measured.

4. The method for automatic detection and interpretation of semantic changes in social media online terminology according to claim 1, characterized in that, The encoding of the visual representation of the video frame image corresponding to the social media comment text, the textual representation of internet slang and social media comment text, and the speech representation of internet slang and social media comment text using an encoder based on the Transformer model includes: For video frame images corresponding to social media comment text, a pre-trained model for images is used to obtain the overall visual representation and a series of local visual representations, which constitute the visual representation of the video frame image and are input into the first encoding unit in the encoder based on the Transformer model to obtain visual features. The text sequence is obtained by concatenating internet slang with social media comment text and setting a separator in the concatenation part; using the pre-trained model for the text, the vector representation of each word in the text sequence is obtained, which constitutes the text representation of internet slang and social media comment text, and is input into the second encoding unit in the encoder based on the Transformer model to obtain text features; Each word is translated into its corresponding phonetic symbol. The phonetic symbols of words in internet slang are concatenated with the phonetic symbols of words in social media comment text, and a separator is set in the concatenation part to obtain a phonetic symbol sequence. A variable linear layer is used to obtain the feature representation of each phonetic symbol in the phonetic symbol sequence, which constitutes the speech representation of internet slang and social media comment text. This representation is then input into the third encoding unit in the encoder based on the Transformer model to obtain speech features.

5. The method for automatic detection and interpretation of semantic changes in social media online terminology according to claim 1, characterized in that, The steps to generate natural language interpretation text include: Image features, text features, and speech features are concatenated and fused using an attention mechanism. Then, words at each time step are generated using an autoregressive method through a decoder based on a Transformer model. The words are combined in time sequence to form a natural language explanatory text.

6. An automatic detection and interpretation system for semantic changes in social media online terminology, characterized in that, Based on the method described in any one of claims 1 to 5, the system comprises: An image-enhanced web slang detection module is applied to the web slang detection stage, which includes: processing a social media comment corpus C. u Social media comment texts in the text, through a given general corpus C g The corresponding word vector representation set is used to obtain the vector representations of all words in the social media comment text, forming the first text vector sequence, and the vector representations are obtained from the social media comment corpus C. u The corresponding set of word vector representations is used to obtain the vector representations of all words in the social media comment text, forming a second text vector sequence; the distance between the vector representations of the same words in the first and second text vector sequences is measured, and the K words with the largest distances are selected from the social media comment text as online slang; The Internet slang explanation task module is applied to the Internet slang explanation stage. The Internet slang explanation stage includes: encoding the visual representation of the video frame image corresponding to the social media comment text, the text representation of the Internet slang and the social media comment text, and the speech representation of the Internet slang and the social media comment text respectively through an encoder based on the Transformer model; inputting the encoded image features of the video frame image, the text representation of the Internet slang and the social media comment text, and the speech features of the Internet slang and the social media comment text into the decoder based on the Transformer model to generate natural language explanation text.

7. A processing device, characterized in that, include: One or more processors; Memory, used to store one or more programs; Wherein, when the one or more programs are executed by the one or more processors, the one or more processors cause the one or more processors to implement the method as described in any one of claims 1 to 5.

8. A readable storage medium storing a computer program, characterized in that, When a computer program is executed by a processor, it implements the method as described in any one of claims 1 to 5.

Images