Granule calculation-based multi-mode co-evolution anti-interference identification method and device

Abstract

The invention provides a multi-modal co-evolution anti-recognition method and device based on granular computing, and relates to the technical field of natural language process.The method comprises the steps that firstly, sample data is extracted from multiple modals, preprocessing and anti-label labeling are completed, and a multi-modal anti-data set with cross-modal alignment is formed; performing feature clustering in each modal feature space by utilizing hierarchical granularity analysis, and constructing a multi-modal multi-granularity knowledge representation; a plurality of granularities with the highest dependency degree are screened from all the modals based on a rough set dependency function, and a multi-modal optimal granularity matrix is generated; extracting a prototype mode vector of a known antitag and a test mode vector of a to-be-tested sample from the matrix, and constructing a heat sequence parameter depicting the similarity of the prototype mode vector and the test mode vector; and finally, constructing a collaborative neural network model based on a synergetics principle, enabling each modal sequence parameter to co-evolve under a self-excitation, self-suppression and side suppression mechanism, and outputting an anti-vital mark mode corresponding to the highest sequence parameter after weight fusion to realize anti-vital mark recognition.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of natural language processing, and particularly relates to a multi-modal collaborative evolution irony recognition method and device based on granular computing. BACKGROUND

[0002] In the booming development of natural language processing technology today, irony recognition as a key branch of sentiment computing and semantic understanding, its research value and application demand is increasingly prominent. Irony as a special way of language expression, its expression often shows a significant deviation between the literal meaning and the true intention. This "double-dealing" semantic feature makes the traditional recognition strategy based on surface text matching easily ineffective. With the explosive growth of social media and short video platforms, the spread of irony information has evolved rapidly from single text form to complex form of interweaving of multiple modal elements such as text, pictures, voice, expressions, editing rhythm and even picture atmosphere, which greatly increases the complexity of irony recognition task; a seemingly flattering text may constitute a strong irony effect if it is combined with exaggerated expressions, special tone or contrast strong pictures. However, the existing technology still stays at the level of shallow analysis of isolated text semantics or single visual symbol of image, lacks deep modeling of cross-modal information complementarity and synergy, and cannot effectively process the rich information contained in multi-modal data, resulting in frequent misrecognition and missed recognition in real open scenarios. For example, on social media platforms, users often express irony through text and pictures or short videos, and it is difficult to accurately understand the true semantics by relying on single modal analysis. On the other hand, in the feature extraction and representation learning process, there is a lack of multi-level and multi-angle analysis of data, which makes the model unable to fully capture the subtle differences and deep semantics in irony expression.

[0003] Meanwhile, there are problems such as incomplete knowledge representation and insufficient feature fusion in the sarcasm recognition. Traditional methods often use fixed feature extraction strategies, which are difficult to adapt to the diversity of sarcasm expression in different scenarios. In terms of multi-modal information fusion, the existing technology lacks effective collaborative evolution mechanism, and cannot fully utilize the complementary relationship between modalities, which seriously affects the stability and accuracy of the recognition effect. Specifically, sarcasm expression naturally has distinct context dependence and dynamic evolution characteristics: the same sentence may show completely opposite emotional polarity under different cultural backgrounds, narrative setting or emotional tone; and the rapid change of network slang and meme culture makes the sarcasm mode show a fragmented and instantaneous evolution trend. Traditional schemes based on fixed feature templates or single granularity representation learning are difficult to capture such subtle differences, often leading to rigid knowledge representation and insufficient generalization ability. In addition, in the multi-modal feature fusion link, existing methods generally use simple strategies such as early splicing or late voting, ignoring the dynamic changes of the weight of each modality feature at different granularity levels, and lack effective collaborative evolution mechanism to real-time correct the confidence of each channel, thereby limiting the robustness and accuracy of the overall recognition system.

[0004] More difficultly, sarcasm expression has strong context dependence and dynamic change characteristics, which requires the model to have strong context understanding ability. The sarcasm signal often shows rhythm changes of "first raising and then suppressing" or "catching by feigning to let go" in the time dimension, which requires the model to have long-distance dependence modeling ability across modalities and granularity. However, existing deep learning frameworks perform poorly when dealing with such high-order cognitive tasks (i.e. long-distance dependence and dynamic context change), still face problems such as poor explainability, sensitivity to noise, and huge demand for training data, making it difficult to accurately grasp the semantic conversion and emotional change in sarcasm expression. The existence of these problems seriously restricts the practical application effect of sarcasm recognition technology. Therefore, how to adaptively mine the most discriminative granularity features in multi-modal heterogeneous data and dynamically fuse the evidence of each channel through collaborative evolution reasoning has become a core problem that needs to be broken through in the field of sarcasm recognition.

[0005] In view of the above, the present application is proposed. SUMMARY

[0006] The present application provides a multi-modal collaborative evolution sarcasm recognition method and device based on granular computing, which can at least partially improve the above problems.

[0007] To achieve the above object, the present application adopts the following technical scheme: A multi-modal collaborative evolution sarcasm recognition method based on granular computing, comprising: extracting sample data of different modalities from a preset sample source library, and constructing a multi-modal sarcasm dataset based on the sample data; The hierarchical granularity analysis and division processing are performed on the multi-modal irony dataset, and a multi-modal multi-granularity knowledge representation is constructed; According to the multi-modal multi-granularity knowledge representation, the dependency of each mode is calculated, and the optimal multiple granularity levels of each mode are screened out, and a multi-modal optimal granularity matrix is obtained; Prototype mode vectors and test mode vectors are extracted from the multi-modal optimal granularity matrix, and irony sequence parameters are constructed according to the prototype mode vectors and the test mode vectors; The collaborative neural network model of the irony sequence parameter is used for collaborative recognition preprocessing of the sequence parameter, and an irony recognition result is generated.

[0008] The application also provides a multi-modal collaborative evolution irony recognition device based on granular computing, which comprises: A data processing unit is used for extracting sample data of different modes from a preset sample source library, and constructing a multi-modal irony dataset based on the sample data; A multi-granularity unit is used for hierarchical granularity analysis and division processing on the multi-modal irony dataset, and a multi-modal multi-granularity knowledge representation is constructed; An optimal granularity unit is used for calculating the dependency of each mode according to the multi-modal multi-granularity knowledge representation, screening out the optimal multiple granularity levels of each mode, and obtaining a multi-modal optimal granularity matrix; A sequence parameter unit is used for extracting prototype mode vectors and test mode vectors from the multi-modal optimal granularity matrix, and constructing irony sequence parameters according to the prototype mode vectors and the test mode vectors; A collaborative recognition unit is used for using the collaborative neural network model of the irony sequence parameter to perform collaborative recognition preprocessing on the sequence parameter, and generating an irony recognition result.

[0009] In summary, the multi-modal collaborative evolution irony recognition method based on granular computing combines the granular computing theory with multi-modal data, improves the robustness and accuracy of irony recognition through multi-granularity division and collaborative evolution mechanism, and provides a technical scheme for intelligent irony understanding. Specifically, the method integrates multi-modal data such as text, image, audio and video, and combines hierarchical granularity analysis method to realize all-around recognition of irony samples. The method innovatively uses the granular computing theory for multi-modal knowledge representation, screens the optimal granularity combination through the dependency function, constructs the irony sequence parameter and uses the collaborative neural network for reasoning, and finally obtains the recognition result based on weight fusion, effectively solving the limitation problem of traditional single mode recognition method.

[0010] By means of the technical scheme, the complementarity and synergy of multi-modal data are fully utilized, the subtle features of irony expression are captured through multi-granularity modeling, and information loss caused by single-granularity analysis is avoided. Meanwhile, the collaborative evolution mechanism effectively integrates the recognition results of different modalities, significantly enhancing the accuracy and generalization ability of irony recognition. The method not only achieves good performance in irony recognition tasks, but also provides an innovative technical idea for multi-modal fusion analysis in the field of natural language processing, and has important theoretical value and practical significance. BRIEF DESCRIPTION OF DRAWINGS

[0011] Figure 1 is a flowchart of a multi-modal collaborative evolution irony recognition method based on granular computing provided by the first embodiment of the present application; Figure 2 is a schematic diagram of multi-modal multi-granularity modeling and optimal granularity selection provided by the embodiment of the present application; Figure 3 is a schematic diagram of collaborative evolution provided by the embodiment of the present application; Figure 4 is a module schematic diagram of a multi-modal collaborative evolution irony recognition device based on granular computing provided by the second embodiment of the present application. DETAILED DESCRIPTION

[0012] In order to make the purpose, technical scheme and advantages of the present application clearer, further detailed description will be made to the present application in combination with embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application, and are not used to limit the present application.

[0013] Reference Figures 1 to 3 As shown in the figure, the first embodiment of the present application discloses a granular computing multi-modal collaborative evolution irony recognition method, which can be executed by a granular computing multi-modal collaborative evolution irony recognition device (hereinafter referred to as a recognition device), in particular, by one or more processors in the recognition device to implement the following method: S1, extracting sample data of different modalities from a preset sample source library, and constructing a multi-modal irony data set based on the sample data; Specifically, step S1 further comprises: extracting sample data of different modalities from a preset sample source library to obtain multi-modal sample data, wherein the modalities include text, image, audio and video, and the preset sample source library can be a well-known talk show program, such as stand-up comedy performance, variety show, etc. The multi-modal sample data is pre-processed, and the pre-processed data is sarcastic annotation processing, and the annotation results are two categories of sarcasm and non-sarcasm, wherein the pre-processing includes word segmentation processing, stop word removal processing, standardization processing of text data, size normalization processing, light adjustment processing, face detection and alignment processing of image data, noise reduction processing, volume standardization processing, frame processing of audio data, frame rate uniform processing, resolution adjustment processing, key frame extraction processing of video data; The multi-modal sample data is aligned according to the timestamp, and a multi-modal sarcasm data set is constructed.

[0014] In this embodiment, first, the sample source library interface is started, and the original text, image, audio and video stream of the single comedy special and variety show are read in batches; after the text stream is segmented, the stop words are removed, and the standardization is performed, the clean text segment is obtained, the image stream is normalized in size, the light is adjusted, the face is detected and aligned, the standard frame is obtained, the audio stream is de-noised, the volume is standardized, and the frame is obtained, the uniform energy spectrum is obtained after the video stream is uniformly processed, the resolution is adjusted, and the key frame is extracted, and the compact frame sequence is obtained; then, the four modal data is artificially annotated as sarcasm / non-sarcasm binary, and is accurately aligned according to the timestamp, to form a structured multi-modal sarcasm data set, which is used for subsequent granularity analysis and cooperative evolution identification.

[0015] Specifically, for text data, use jieba and other word segmentation tools for Chinese word segmentation, remove punctuation, numbers and other stop words, uniformly convert to lowercase, and perform text length normalization processing. For image data, all images are uniformly scaled to 224x224 pixels, the image brightness and contrast are adjusted using the histogram equalization method, face detection is performed using OpenCV and other tools, and the detected face area is aligned. For audio data, use a band-pass filter to remove environmental noise, standardize the volume to a uniform decibel level, and frame the audio signal according to a fixed window length of 20ms. For video data, the video frame rate is unified to 30fps, the resolution is adjusted to 1280x720, and a frame is extracted every 0.5 seconds as a key frame. In addition, the pre-processed original sample is manually annotated, and the annotation content includes whether the sample contains sarcastic expression (0 / 1), thereby establishing a corpus containing emotion and metaphor double annotation.

[0016] Subsequently, a unified data indexing system is established, and a unique identification ID is assigned to each sample. Then, the data of different modalities are aligned according to the timestamps, ensuring that the text, image, audio and video data remain synchronized in time sequence. For each sample, its text content, corresponding image frames, audio segments and video segments are organized into a structured data format, and the sample's various modality data paths and annotation information are stored in JSON format. Finally, the processed dataset is randomly divided into a training set and a validation set in a 7:3 ratio, ensuring that the positive and negative sample proportions in the two datasets are basically consistent, and the division results are saved as a configuration file.

[0017] S2, hierarchical granularity analysis and division processing are performed on the multi-modal irony dataset to construct a multi-modal multi-granularity knowledge representation; Specifically, step S2 further includes: extracting features from the multi-modal irony dataset to obtain feature spaces of each modality, and obtaining different feature subsets from the feature spaces of each modality through feature clustering technology, to construct knowledge representations of different granularities; Based on different feature subsets, the multi-modal irony dataset is divided to obtain multi-granularity feature subspaces of each modality, and the multi-granularity feature subspaces of each modality are aligned to construct a multi-modal multi-granularity knowledge representation.

[0018] In this embodiment, first, for the text modality, the system extracts the text sequence after word segmentation and standardization from the data set one by one, and maps it to a continuous vector space using a pre-trained language model; in this vector space, the system performs unsupervised feature clustering to automatically divide the text features into several non-overlapping feature subsets, each of which corresponds to a knowledge representation at a certain granularity level, such as word level, phrase level, sentence level, etc. For the image modality, the system extracts color, texture and shape features from each key frame to form a high-dimensional feature vector; the same clustering algorithm is run in this feature space to obtain three types of granularity descriptions: pixel level, region level and whole image level. For the audio modality, the system extracts Mel frequency cepstrum coefficients and fundamental frequency features from the audio segment, and then clusters them to obtain feature subsets at the frame level, phoneme level and sentence level. For the video modality, temporal and spatial features are extracted from key frames and optical flow information, and are divided into frame level, shot level and segment level granularity through clustering. After completing the internal granularity division of each modality, the system aligns the same level granularity features of text, image, audio and video using the original timestamp as the index: if there are sentence-level text, region-level image, phoneme-level audio and shot-level video features at the same time, then mark this four-tuple as the corresponding sample at the same granularity level; if a modality is missing at this time, use a zero vector to occupy it to ensure the consistency of the matrix operation dimension. Through this alignment mechanism, the system finally forms a "multimodal and multi-granularity knowledge representation" covering text, image, audio and video, and containing multiple levels of granularity in each modality. This representation directly inherits the time alignment relationship of the original data, without introducing additional parameters, preserving the granularity structure of each modality itself and achieving one-to-one mapping across modalities. It lays a data foundation for subsequent dependency calculation and optimal granularity selection, and significantly reduces the error propagation risk caused by granularity mismatch.

[0019] Specifically, as shown in Figure 2 text features are extracted from text modality data, wherein the text features include part-of-speech features, and / or semantic role features, and / or phonological features, etc.; image features are extracted from image modality data, wherein the image features include color features, and / or posture features, and / or atmosphere features, etc.; audio features are extracted from audio modality data, wherein the audio features include intonation features, and / or pitch features, and / or tone features, etc.; video features are extracted from video modality data, wherein the video features include plot features, and / or motion features, and / or narrative features, etc.

[0020] Among them, single-granularity knowledge representation refers to dividing data by a specific set of feature subsets. In order to fully capture sample information, multi-granularity knowledge representation is introduced. Multi-granularity knowledge representation refers to obtaining multiple feature subsets based on the extracted feature set through clustering, and different feature subsets have different divisions of the data set. As shown in Figure 2As shown, the process of multimodal multi-granularity modeling and optimal granularity selection is illustrated.

[0021] S3. Calculate the dependency of each mode based on the multimodal multigranularity knowledge representation, select the optimal multiple granularity levels for each mode, and obtain the multimodal optimal granularity matrix. Specifically, step S3 further includes: the granularity level list constructed from the modality M in the multimodal multigranularity knowledge representation is as follows: Where I is the number of granularities in the granularity level list of mode M. The granularity is for the i-th mode M; Based on the decision set D and the granularity of the i-th mode M The dependency of mode M is calculated using the following formula: ; The n granularities with the highest dependence in mode M are selected, and the optimal granularity matrix of the multimodal model is obtained based on the optimal n granularity levels of each mode, where n is the preset number of granularities.

[0022] In this embodiment, based on multimodal and multigranular knowledge representation, for each modality M (text, image, audio, video), a granularity level list has been formed internally according to the clustering results. Using the decision set (ironic / non-ironic tag set) as a benchmark, the contribution of each granularity to the classification decision is directly calculated using the rough set dependency formula; the positive domain in this formula consists of a set of samples where the granularity is unambiguous on the decision set. After calculation, all granularities of the current modality are sorted in descending order of dependency, and the top n granularities with the highest dependency are automatically selected as the optimal granularity combination for that modality; the value of n is preset by the user. After the above process is performed on the text, image, audio, and video modalities respectively, the system concatenates their respective optimal granularity subsets column-wise to form a multimodal optimal granularity matrix with consistent dimensions. This matrix retains the most discriminative granularity information for each modality and filters out low-contribution or noisy granularities through dependency quantification, demonstrating the dual advantages of adaptive granularity selection in terms of accuracy and efficiency.

[0023] Specifically, such as Figure 2 As shown, the dependency function is a mathematical tool in rough set theory used to measure the classification ability of conditional features (or granularity) on decision features. It quantifies the degree to which "equivalence classes partitioned by conditional features or granularity levels can accurately predict the decision class." Dependency is the specific value of the dependency function, representing the degree of dependence of the conditional attribute on the decision attribute, and its value ranges from [0,1]. Figure 2 The diagram illustrates the process of multimodal, multi-granularity modeling and optimal granularity selection. The granularity level lists for text, image, audio, and video modal data are as follows: , , and .

[0024] According to the decision set D and granularity level Calculate the text dependency , select the n granularities with the largest text dependency. According to the decision set D and granularity level Calculate the image dependency , select the n granularities with the largest image dependency. According to the decision set D and granularity level Calculate the audio dependency , select the n granularities with the largest audio dependency. According to the decision set D and granularity level Calculate the video dependency , select the n granularities with the largest video dependency.

[0025] Wherein, the calculation formula of text dependency is , the calculation formula of image dependency is , the calculation formula of audio dependency is , the calculation formula of video dependency is . , , and are the positive domains (i.e. the sample set determined to belong to a certain decision class) of decision attribute D at granularity , , and , denotes the cardinality of the set.

[0026] Further, according to the decision D, all samples in the sample set U are divided, denoted as division U / D; according to , , and , all samples in the sample set U are divided, respectively denoted as division U / T i , U / P i , U / A i , U / V i . The calculation formula of the positive domain of decision attribute D at granularity is ; the calculation formula of the positive domain of decision attribute D at granularity is ; the calculation formula of the positive domain of decision attribute D at granularity is ; the calculation formula of the positive domain of decision attribute D at granularity is The multi-modal optimal granularity matrix includes a text optimal granularity matrix, an image optimal granularity matrix, an audio optimal granularity matrix, and a video optimal granularity matrix.

[0027] S4, extracting a prototype mode vector and a test mode vector from the multi-modal optimal granularity matrix, and constructing a sarcasm order parameter according to the prototype mode vector and the test mode vector; Specifically, step S4 further includes: extracting a row vector of a training set sample with a known sarcasm label from the multi-modal optimal granularity matrix as a prototype mode vector of the mode M , wherein v is a prototype, and m is the number of row vectors in the prototype mode vector; extracting a row vector of a test set sample from the multi-modal optimal granularity matrix as a test mode vector of the mode M , wherein u is a test, and n is the number of row vectors in the test mode vector; constructing a sarcasm order parameter according to the prototype mode vector and the test mode vector .

[0028] The calculation model formula of the sarcasm order parameter is: , wherein d is the number of samples in the test set, is an order parameter, is a sarcasm order parameter of the mode M, and T is a text.

[0029] In this embodiment, for each mode, first, all samples in the training set that are explicitly labeled as "sarcasm" are traversed, and their row vectors in the corresponding modal optimal granularity matrix are sequentially taken out and recorded as a prototype mode vector set; then, for each sample in the test set to be identified, its row vector in the same modal optimal granularity matrix is also extracted and recorded as a test mode vector set. Since the matrix has been screened by dependence, the dimensions of the prototype and test vectors remain highly consistent, avoiding the dimension disaster caused by redundant features. After obtaining the above two groups of vectors, the given similarity function is directly used to construct a sarcasm order parameter: for the a-th sample in the test set, the cosine similarity between it and all prototype mode vectors is normalized to form the sarcasm order parameter of the mode.

[0030] Specifically, as shown in Figure 3 , the method extracts the corresponding row vector of the training set sample that meets the known sarcasm label from each modal optimal granularity matrix as the prototype mode vector under the mode. The prototype mode vector represents the typical mode of the known sarcasm label. The prototype mode vector represents a typical example feature set of a certain category, which is the centralized representation of the category. As shown in Figure 3 , the process of co-evolution is shown. The prototype mode vector includes a text prototype mode vector , image prototype mode vector , audio prototype mode vector , and video prototype mode vector . The test mode vector includes text test mode vector , image test mode vector , audio test mode vector , and video test mode vector .

[0031] Further, the calculation model of the text irony sequence parameter is: The calculation model of the image irony sequence parameter is: The calculation model of the audio irony sequence parameter is: The calculation model of the video irony sequence parameter is: . Wherein, T is text, P is image, A is audio, and V is video.

[0032] S5, the irony sequence parameter is cooperatively identified and preprocessed by the cooperative neural network model of the irony sequence parameter, to generate an irony recognition result.

[0033] Specifically, step S5 further includes: constructing a cooperative neural network model based on the cooperative principle, and reconstructing the sequence parameter by the cooperative neural network model of the irony sequence parameter; The reconstructed sequence parameter is fused and sorted, and the prototype mode vector corresponding to the highest sequence parameter is selected; When it is judged that the to-be-identified sample belongs to the prototype mode vector corresponding to the highest sequence parameter, the irony annotation result of the to-be-identified sample is obtained, and the irony recognition result is generated.

[0034] The formula of the cooperative neural network model under the mode M is: , wherein, is the attention parameter, which is used to control the speed of the to-be-tested mode change, B is the lateral inhibition coefficient, C is the self-inhibition coefficient, is the mode index.

[0035] In addition, in the cooperative neural network model, the parameter combination determines the recognition performance of the cooperative mode recognition together; is the self-excitation term, which represents the feedback excitation effect of the mode on itself; is the self-inhibition term, which reflects the inhibition of the mode to its excessive growth; is the lateral inhibition term, which reflects the mutual inhibition effect between modes, which is equivalent to a penalty term.

[0036] In this embodiment, after the sarcasm order parameters are recalculated, the order parameters are immediately sent to the cooperative neural network model built according to the principle of synergetics. Based on the initial state, the evolution is performed in the discrete time step. When the network is iterated, each mode runs in parallel, does not need additional training, and can complete the reconstruction of the order parameter through the dynamic equation. After a small number of time steps, the internal competition tends to be stable, and the reconstructed order parameter fully reflects the cross-modal consensus. Subsequently, the four-mode reconstruction results are weighted and fused, and the weight is determined by the normalized dependence of each mode; the fusion vector is arranged in descending order of numerical value, and the prototype mode vector corresponding to the highest one is regarded as the final decision basis. When and only when the highest order parameter exceeds the preset threshold, it is determined that the to-be-identified sample belongs to the sarcasm class, otherwise it belongs to the non-sarcasm class, and the corresponding sarcasm labeling result is output.

[0037] Specifically, the cooperative neural network model under the text mode is: The cooperative neural network model under the image mode is: The cooperative neural network model under the audio mode is: The cooperative neural network model under the video mode is: .

[0038] In the cooperative neural network model, , , and are self-excitation terms, representing the feedback excitation effect of the mode on itself; , , and are self-inhibition terms, which reflect the inhibition of excessive growth of the mode; , , and are side inhibition terms, which reflect the mutual inhibition effect between modes, which is equivalent to a penalty term. In the formula, B=C=1.2, and takes values of 0.18, 0.36, 0.54, and 0.72, respectively.

[0039] Based on the order parameters of each mode, a weight fusion multi-modal is introduced, and the weight coefficients of each mode are calculated by dependence (corresponding to the weight coefficients of the text, image, audio, and video modes, respectively). Then, the calculation formula of the weight coefficient of the text mode is: The calculation formula of the weight coefficient of the image mode is: The calculation formula of the weight coefficient of the audio mode is: The calculation formula of the weight coefficient of the video mode is: .

[0040] Based on the weight coefficients of each modality, the sequence parameters are fused to calculate a multi-modal fusion sequence parameter The calculation formula of the multi-modal fusion sequence parameter is: . Wherein, represents a text irony sequence parameter, represents an image irony sequence parameter, represents an audio irony sequence parameter, represents a video irony sequence parameter. According to the multi-modal fusion sequence parameter , the irony labeling mode of the to-be-recognized sample is obtained, and an irony recognition result and a final irony label are obtained.

[0041] Of course, the preset number of the optimal granularity of the four modal data can also be different, which can be set according to actual needs. If the preset number of the optimal granularity of each modality data is different, the optimal granularity of each modality needs to be vectorized and mapped to the same dimension when performing cross-modal fusion. Specifically, first, the features in the optimal granularity of each modality are vectorized to obtain feature vectors of each modality; second, in order to ensure subsequent operations, all modality feature vectors can be mapped to the same dimension by using a full connection layer or the like; third, each mapped feature vector is multiplied by its corresponding weight coefficient; and finally, all weighted modality feature vectors are spliced.

[0042] To sum up, the method integrates multi-modal data, combines hierarchical granularity analysis method, and uses collaborative evolution to realize all-around recognition of irony samples, uses the complementarity and synergy of multi-modal data, comprehensively analyzes multi-modal data of different granularities, and gives a more accurate irony recognition method. Simply put, the multi-modal collaborative evolution irony recognition method based on granular computing realizes accurate modeling and recognition of irony expression by extracting multi-modal data and performing multi-granularity hierarchical analysis, and solves the technical problems of insufficient accuracy and insufficient generalization ability in the prior art.

[0043] Referring to Figure 4 , the second embodiment of the present application provides a multi-modal collaborative evolution irony recognition device based on granular computing, which comprises: A data processing unit 101 is configured to extract sample data of different modalities from a preset sample source library, and construct a multi-modal irony data set based on the sample data; A multi-granularity unit 102 is configured to perform hierarchical granularity analysis and division processing on the multi-modal irony data set, and construct a multi-modal multi-granularity knowledge representation; An optimal granularity unit 103 is configured to calculate the dependency of each modality according to the multi-modal multi-granularity knowledge representation, select a plurality of optimal granularity levels of each modality, and obtain a multi-modal optimal granularity matrix; The sequence parameter unit 104 is configured to extract a prototype mode vector and a test mode vector from the multi-modal optimal granularity matrix, and construct a sarcastic sequence parameter according to the prototype mode vector and the test mode vector. The cooperative recognition unit 105 is configured to perform cooperative recognition preprocessing on the sequence parameter by using a cooperative neural network model of the sarcastic sequence parameter, to generate a sarcastic recognition result.

[0044] The above describes the preferred embodiments of the present application. It should be noted that, for those skilled in the art, without departing from the principles of the present application, a number of improvements and refinements can be made, which are also considered within the scope of protection of the present application.

Claims

1. A multimodal cooperative evolutionary irony recognition method based on granular computing, characterized in that, include: Extract sample data of different modalities from a pre-defined sample source library, and construct a multimodal satirical dataset based on the sample data; A hierarchical granularity analysis and partitioning process is performed on the multimodal irony dataset to construct a multimodal, multigranular knowledge representation; The dependency of each mode is calculated based on the multimodal multigranularity knowledge representation, and the optimal granularity levels of each mode are selected to obtain the multimodal optimal granularity matrix. Extract the prototype mode vector and test mode vector from the multimodal optimal granularity matrix, and construct the irony order parameter based on the prototype mode vector and test mode vector; A collaborative neural network model for irony order parameters is used to perform collaborative recognition preprocessing on the order parameters to generate irony recognition results.

2. The multimodal cooperative evolution irony recognition method based on granular computing according to claim 1, characterized in that, Sample data of different modalities are extracted from a pre-defined sample source library, and a multimodal irony dataset is constructed based on the sample data, specifically as follows: Multimodal sample data is obtained by extracting sample data of different modalities from a pre-defined sample source library. The modalities include text, images, audio, and video. Multimodal sample data is preprocessed, and the preprocessed data is then annotated with irony labels. The annotation results are divided into two categories: irony and non-irony. The preprocessing includes word segmentation, stop word removal, and standardization for text data; size normalization, illumination adjustment, face detection and alignment for image data; noise reduction, volume standardization, and frame segmentation for audio data; and frame rate unification, resolution adjustment, and keyframe extraction for video data. The labeled multimodal sample data are aligned according to the timestamps to construct a multimodal satirical dataset.

3. The multimodal cooperative evolution irony recognition method based on granular computing according to claim 1, characterized in that, A hierarchical granularity analysis and partitioning process is performed on the multimodal irony dataset to construct a multimodal, multi-granularity knowledge representation, specifically: Features are extracted from the multimodal irony dataset to obtain the feature space of each modality, and different feature subsets are obtained from each modality feature space through feature clustering technology to construct knowledge representations of different granularities; The multimodal satirical dataset is partitioned based on different feature subsets to obtain multi-granularity feature subspaces for each modality. The multi-granularity feature subspaces for each modality are then aligned to construct a multimodal multi-granularity knowledge representation.

4. The multimodal cooperative evolution irony recognition method based on granular computing according to claim 1, characterized in that, The dependencies of each mode are calculated based on multimodal multigranularity knowledge representation. The optimal granularity levels for each mode are then selected, resulting in the multimodal optimal granularity matrix, as follows: The list of granularity levels constructed from the modality M in the multimodal multigranularity knowledge representation is as follows: Where I is the number of granularities in the granularity level list of mode M. The granularity is for the i-th mode M; Based on the decision set D and the granularity of the i-th mode M The dependency of mode M is calculated using the following formula: ; The n granularities with the highest dependence in mode M are selected, and the optimal granularity matrix of the multimodal model is obtained based on the optimal n granularity levels of each mode, where n is the preset number of granularities.

5. The multimodal cooperative evolution irony recognition method based on granular computing according to claim 1, characterized in that, Extract the prototype mode vector and test mode vector from the multimodal optimal granularity matrix, and construct the irony order parameter based on the prototype mode vector and test mode vector, specifically: Extract the row vectors of the training set samples with known ironic labels from the multimodal optimal granularity matrix as the prototype pattern vectors of modality M. , where v is the prototype and m is the number of row vectors in the prototype pattern vector; The row vectors of the test set samples are extracted from the multimodal optimal granularity matrix and used as the test mode vectors of mode M. Where u is the test and n is the number of row vectors in the test pattern vector; Based on prototype pattern vector and test mode vector Construct an ironic order parameter that reflects the similarity between the prototype pattern vector and the test pattern vector.

6. The multimodal cooperative evolution irony recognition method based on granular computing according to claim 5, characterized in that, The formula for calculating the irony order parameter is as follows: Where d is the number of samples in the test set. As an order parameter, Let T be the ironic order parameter of modality M, and T be the text.

7. The multimodal cooperative evolution irony recognition method based on granular computing according to claim 6, characterized in that, A collaborative neural network model for ironic order parameters is used to perform collaborative recognition preprocessing on the order parameters to generate ironic recognition results, specifically: A collaborative neural network model is constructed based on the principle of synergetics, and the order parameter is reconstructed using a collaborative neural network model with ironic order parameters. The reconstructed order parameters are fused and sorted to select the prototype pattern vector corresponding to the highest order parameter in the sequence. When it is determined that the sample to be identified belongs to the prototype pattern vector corresponding to the highest order parameter in the sequence, the irony labeling result of the sample to be identified is obtained, and the irony recognition result is generated.

8. The multimodal cooperative evolution irony recognition method based on granular computing according to claim 7, characterized in that, The formula for the cooperative neural network model under mode M is: ,in, Note the parameters: B is the lateral inhibition coefficient, and C is the self-inhibition coefficient. This is a pattern index.

9. A multimodal cooperative evolution irony recognition device based on granular computing, characterized in that, include: The data processing unit is used to extract sample data of different modalities from a preset sample source library and construct a multimodal irony dataset based on the sample data; Multi-granularity units are used to perform hierarchical granularity analysis and partitioning of multimodal irony datasets to construct multimodal multi-granularity knowledge representations; The optimal granularity unit is used to calculate the dependency of each mode based on the multimodal multigranularity knowledge representation, select the optimal multiple granularity levels for each mode, and obtain the multimodal optimal granularity matrix. The order parameter unit is used to extract the prototype mode vector and test mode vector from the multimodal optimal granularity matrix, and construct the ironic order parameter based on the prototype mode vector and test mode vector; The collaborative recognition unit is used to perform collaborative recognition preprocessing on the order parameter using a collaborative neural network model of the irony order parameter, and generate irony recognition results.

Images