Cross-modal multi-granularity humor identification method and device based on composite distinguishing measure

Abstract

The invention provides a cross-modal multi-granularity humor recognition method and device based on composite distinguishing measure, and relates to the technical field of data mining. The method comprises the following steps: acquiring and preprocessing multi-modal data, and establishing a multi-modal data set; performing multi-granularity sampling on the multi-modal data in each modal to obtain a cross-modal multi-granularity sample space; selecting an optimal granularity combination through coordination; and carrying out identification and classification on humor contents by adopting cross-modal fusion of distinguishing measurement and variable coordination measurement. According to the method, the accuracy and robustness of humor recognition are improved through fusion of multi-modal feature collaborative representation and multi-granularity knowledge mining, a new thought is provided for the field of humor recognition, and the method has important theoretical and application values.

Description

Technical Field

[0001] The present invention relates to the field of data mining and artificial intelligence technology, and in particular to a cross-modal multi-granularity humor recognition method and device based on composite discrimination measurement. Background Art

[0002] With the rapid development of artificial intelligence technology, humor recognition has attracted widespread attention in fields such as human-computer interaction and content recommendation. Humor recognition aims to automatically determine whether content has humorous attributes by analyzing multimodal data such as text, images, audio, and video. However, since humorous expressions involve complex contextual understanding, cultural cognition, and deep logical relationships, its automatic recognition faces many challenges. Humorous content often relies on specific situational and contextual information, and exhibits diversity and innovation in different cultural backgrounds, which further increases the difficulty of technical implementation. Although advances in deep learning technology have enabled computers to demonstrate near-human understanding capabilities in certain specific tasks, existing methods still have obvious limitations when dealing with humorous content with high cognitive complexity.

[0003] The main problems of current humor recognition technology are reflected in the following aspects. First, existing studies generally adopt single-modal analysis methods, such as focusing only on text or audio features, which makes it difficult to fully capture the rich information in multimodal data. This limitation prevents the system from fully utilizing humor elements in non-text forms such as vision and hearing, thus affecting the recognition effect. Second, in the process of knowledge representation and feature extraction, existing methods lack effective coordination evaluation criteria and cannot accurately select the optimal feature combination. Third, the fusion strategy of cross-modal features is relatively simple, and an effective measure has not been established to evaluate the importance and complementarity of different modal features, resulting in information loss during the fusion process. In addition, when processing multi-granularity information, it is difficult to achieve the optimal representation of data features due to the lack of quantitative indicators of system complexity and category discrimination ability. The existence of these problems seriously limits the accuracy and robustness of humor recognition technology in practical applications.

[0004] In view of this, this application is hereby filed. Summary of the Invention

[0005] The present invention aims to provide a cross-modal and multi-granular humor recognition method and device based on composite discriminant measurement to address the problems existing in existing humor recognition technologies, such as insufficient utilization of single modality information, insufficient multi-granularity knowledge mining, inability to accurately evaluate and select optimal feature combinations, and simplification of cross-modal fusion strategies. This method can achieve a comprehensive characterization of humorous content and significantly improve the accuracy and robustness of humor recognition.

[0006] In order to solve the above technical problems, the present invention is implemented through the following technical solutions: A cross-modal and multi-granular humor recognition method based on composite discriminative measures includes: S1, acquire multimodal data and preprocess them to establish a multimodal dataset; S2, based on the multimodal dataset, performing multi-granularity sampling in each modality to obtain a cross-modal multi-granularity sample space; S3, based on the division of each modal data in the multimodal dataset with respect to the preset decision set and granularity, selecting a preset number of optimal granularities that satisfy coordination from the multi-granularity sample space of each modality to form an optimal granularity combination; S4, based on the optimal granularity combination, using the discrimination measure and the variable coordination measure to perform cross-modal fusion, calculate a composite discrimination measure value to identify humorous content, and obtain a humor recognition result.

[0007] Preferably, the multimodal data includes text data, and / or image data, and / or audio data, and / or video data.

[0008] Preferably, the preprocessing includes: word segmentation, and / or stop word removal, and / or standardization of text data; size normalization, and / or color space conversion of image data; noise reduction, and / or frame segmentation of audio data; frame extraction, and / or action extraction of video data.

[0009] Preferably, the method further includes humor annotation and alignment of the preprocessed multimodal data.

[0010] Preferably, the process of acquiring the cross-modal multi-granularity sample space is specifically as follows: Extracting features of each modality data based on the multimodal dataset; Obtaining multiple feature subsets based on the extracted features; Different feature subsets constitute different granularities. Based on different feature subsets, the data set is divided to obtain multi-granularity sample spaces of each modality. The multi-granularity sample spaces corresponding to each modal data are combined to obtain a cross-modal multi-granularity sample space.

[0011] Preferably, the process of selecting the optimal granularity combination specifically includes: Get the preset decision set ; according to For the multimodal dataset All samples in are divided into ; According to the cross-modal multi-granularity sample space, the multimodal dataset All samples in are divided into , Indicates the level of granularity; ; Indicates the total number of granularities. The larger the number, the finer the granularity division. Indicates the indivual the granularity of the modality; According to the granularity level from coarse to fine, find the multi-granularity sample space corresponding to each mode that satisfies of granularities, the optimal granularity of each mode is obtained, and the optimal granularity combination is formed.

[0012] Among them, when satisfied When , it indicates the mode In particle size On and in decision sets Coordinate with others.

[0013] Preferably, the process of using the discrimination measure and the variable coordination measure to perform cross-modal fusion and calculate the composite discrimination measure value is specifically as follows: Using variable coordination measures to describe the relationship between granularity and decision making, the modality In particle size Variable coordination measure under The calculation formula is: ; The variable compatibility metric is used to calculate the weight coefficient of each mode, so the mode The calculation formula of the weight coefficient is: ; in, Represents various modes; Indicates one of the modes; Indicates the preset number of optimal granularity; Indicates modality In particle size variable coordination measures under ; According to the division and division Calculate the different granularities of each mode Conditional entropy under ; Then, the modal In particle size Conditional entropy under The calculation formula is: ; ; ; in, is an equivalence class probability; is an equivalence class Decision-making The conditional probability of Indicates the cardinality of a set.

[0014] According to the conditional entropy of each mode Calculate the different granularities of each mode The discrimination measure under ; Then, the modal In particle size The discrimination measure under The calculation formula is: ; ; in, is an equivalence class probability; is modal In particle size Conditional entropy under ; Indicates the cardinality of a set.

[0015] Based on the weight coefficient of each modality, cross-modal fusion calculates the composite discrimination measure value , then the composite discrimination measure value The calculation formula is: ; in, Indicates modality In particle size The discrimination measure below.

[0016] Preferably, the process of identifying humorous content is as follows: Setting humor detection thresholds , if the composite discrimination measure value , it is judged as humorous content, otherwise it is non-humorous content.

[0017] The present invention also provides a cross-modal multi-granularity humor recognition device based on a composite discrimination measure, comprising: An acquisition unit, used to acquire and preprocess multimodal data to establish a multimodal dataset; A multi-granularity sampling unit, configured to perform multi-granularity sampling in each modality based on the multi-modal dataset to obtain a cross-modal multi-granularity sample space; an optimal granularity unit, configured to select a preset number of optimal granularities that satisfy coordination from the multi-granularity sample space of each modality based on the division of each modal data in the multimodal data set relative to the preset decision set and granularity, to form an optimal granularity combination; The recognition unit is used to perform cross-modal fusion based on the optimal granularity combination using a discrimination measure and a variable coordination measure to calculate a composite discrimination measure value to recognize humorous content and obtain a humor recognition result.

[0018] The present invention also provides a cross-modal multi-granularity humor recognition device based on a composite discriminant measure, comprising a processor and a memory, wherein the memory stores a computer program that can be executed by the processor to implement the cross-modal multi-granularity humor recognition method based on a composite discriminant measure as described above.

[0019] The present invention also provides a computer-readable storage medium having computer-readable instructions stored thereon. When the computer-readable instructions are executed by a processor of a device where the computer-readable storage medium is located, the computer-readable instructions implement the above-mentioned cross-modal multi-granularity humor recognition method based on a composite discriminant measure.

[0020] In summary, compared with the prior art, the present invention has the following beneficial effects: First, the present invention uses multimodal data collaborative representation to integrate multimodal information such as text, images, audio and video, fully explores the multidimensional characteristics of humorous content, and overcomes the problem of insufficient utilization of single modal information.

[0021] Second, the present invention extracts features from different granularity levels through a multi-granularity knowledge mining strategy, avoiding the information loss problem that may be caused by fixed-granularity feature extraction and improving the comprehensiveness and flexibility of feature representation.

[0022] Third, the present invention selects the optimal granularity through coordination, combines the discrimination measure and the variable coordination measure to perform the optimal granularity combination for cross-modal fusion, accurately selects the optimal feature combination, effectively integrates the complementarity and synergy of each modality data, and enhances the effect of feature fusion.

[0023] Fourthly, the present invention performs humor recognition and classification by calculating a composite discrimination measure value, thereby improving the accuracy and robustness of humor recognition.

[0024] In summary, the method of the present invention not only solves the limitations of existing technologies in cross-modal multi-granularity sampling and fusion, but also significantly improves the performance of humor recognition, providing new research ideas and technical paths in the field of humor recognition, and has important theoretical value and practical application prospects. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following briefly introduces the drawings required for use in the embodiments. It should be understood that the following drawings only illustrate certain embodiments of the present invention and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other relevant drawings can be obtained based on these drawings without paying any creative work.

[0026] Figure 1 A schematic diagram of a cross-modal and multi-granular humor recognition method based on a composite discriminant measure provided in Example 1.

[0027] Figure 2 This is a schematic diagram of the multi-granularity sampling process provided in Example 1.

[0028] Figure 3 Schematic diagram of cross-modal multi-granularity sampling and optimal granularity selection provided in Example 1.

[0029] Figure 4 Schematic diagram of cross-modal fusion calculation of composite discrimination measurement values ​​provided in Example 1.

[0030] Figure 5 This is a schematic diagram of a cross-modal and multi-granular humor recognition device based on a composite discrimination measure provided in Example 2.

[0031] The present invention is further described in detail below with reference to the accompanying drawings and specific embodiments. DETAILED DESCRIPTION

[0032] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention. Therefore, the following detailed description of the embodiments of the present invention provided in the drawings is not intended to limit the scope of the invention for which protection is sought, but merely represents selected embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0033] Example 1 Embodiment 1 of the present invention provides a cross-modal, multi-granular humor recognition method based on a composite discriminant measure. The method can be implemented by a cross-modal, multi-granular humor recognition device based on a composite discriminant measure (hereinafter referred to as a humor recognition device), and in particular, executed by one or more processors within the humor recognition device.

[0034] In this embodiment, the humor recognition device may be an electronic device equipped with a processor, the processor carrying a computer program for the cross-modal multi-granularity humor recognition method based on a composite discriminant measure, and the computer program can be executed, such as a computer, a smart phone, a smart tablet, a workstation, etc., without limitation herein.

[0035] like Figure 1 As shown, a cross-modal multi-granularity humor recognition method based on composite discrimination measurement includes steps S1 to S4.

[0036] S1, obtain multimodal data and preprocess it to establish a multimodal dataset.

[0037] Specifically, step S1 includes steps S11 to S14.

[0038] S11 , collecting TED talks as multimodal data, where the multimodal data includes text data, and / or image data, and / or audio data, and / or video data, etc. Extracting data in multiple modalities, such as text, image, audio, and video, from the multimodal data.

[0039] S12, pre-processing such as word segmentation, stop word removal, and standardization of text data; pre-processing such as size normalization and color space conversion of image data; pre-processing such as noise reduction and frame segmentation of audio data; pre-processing such as frame extraction and motion extraction of video data.

[0040] Specifically, for text data, we first use natural language processing tools such as NLTK to perform word segmentation, dividing the continuous text content into word sequences; then remove stop words such as "the", "a", and "an" that do not contribute much to semantics; then perform word form restoration to unify different forms of words into their original forms; finally, perform standardization to unify uppercase and lowercase letters and remove special characters.

[0041] For image data, all images are first uniformly scaled to a standard size such as a preset size (such as 299×299); then the RGB image is converted to a color space such as HSV or Lab to obtain better feature representation; then image enhancement processing is performed, including adjusting brightness, contrast, etc.; finally, normalization is performed to scale the pixel values ​​to the range [0,1].

[0042] For audio data, a noise reduction algorithm is first used to remove background noise and improve signal quality. The audio signal is then divided into frames, typically using a 25ms frame length and a 10ms frame shift. Acoustic features such as MFCC are then extracted for subsequent analysis. Finally, volume normalization is performed to ensure consistent loudness across different audio clips.

[0043] For video data, we first extract key frames at fixed intervals (such as 1 frame per second); then use tools such as OpenPose to extract human posture features in the video; then extract optical flow features to capture motion information; finally, perform time sampling on the frame sequence to keep all video clips of a uniform length for easy subsequent processing.

[0044] S13, annotating the preprocessed multimodal data, including whether there is humor or not.

[0045] Specifically, the collected original samples are manually labeled to determine whether the sample contains humorous expressions (0 for yes / 1 for no).

[0046] S14: Align the data of different modalities according to time sequence or semantics, align the multimodal data, and establish a multimodal dataset.

[0047] Preprocessing can remove noise from multimodal data (such as typos in text, blurred areas in images, and background noise in audio), ensuring the consistency and validity of the data and providing reliable input for subsequent granular analysis and model training.

[0048] S2: Based on the multimodal dataset, multi-granularity sampling is performed in each modality to obtain a cross-modal multi-granularity sample space.

[0049] Specifically, step S2 includes steps S21 to S26.

[0050] S21, extracting text features based on the text data; wherein the text features include part of speech, and / or grammatical structure, and / or semantic role, and / or emotion, and / or phonology, and so on.

[0051] S22, extracting image features based on the image data; wherein the image features include color, and / or topological relationship, and / or posture, and / or action, and / or atmosphere, etc.

[0052] S23, extracting audio features based on the audio data; wherein the audio features include pitch, and / or timbre, and / or stress, and / or intonation, and / or tone, etc.

[0053] S24, extracting video features based on the video data; wherein the video features include action changes, and / or scene transitions, and / or activities, and / or plot developments, and / or transition effects, etc.

[0054] Extract features from "coarse-grained" to "fine-grained" for different modalities (text, image, audio, video). For example: Text modality: coarse-grained (paragraph level, topic level), medium-grained (phrase level, sentence level), fine-grained (word level, character level); Image modality: coarse-grained (scene level, semantic level), medium-grained (region level, object level), and fine-grained (pixel level, edge features); Audio modality: coarse-grained (sentence level, sentiment level), medium-grained (phoneme level, syllable level), and fine-grained (spectral frame level); Video modality: coarse-grained (sequence level, storyline level), medium-grained (clip level, scene level), and fine-grained (frame level, action detail level).

[0055] S25. Under each modal data, multiple feature subsets are obtained based on the full set of extracted features. Different feature subsets constitute different granularities, i.e., multi-granularity sampling, thereby obtaining a text multi-granularity sample space, a picture multi-granularity sample space, an audio multi-granularity sample space, and a video multi-granularity sample space respectively.

[0056] Preferably, single-granularity sampling refers to dividing the data by a set of specific feature subsets. In order to fully capture the sample information, multi-granularity sampling is introduced. Multi-granularity sampling refers to obtaining multiple feature subsets based on the extracted features, and dividing the data set differently based on different feature subsets. Figure 2 As shown, the process of multi-granularity sampling is demonstrated.

[0057] For text data, different samples can be generated according to the word level, phrase level and sentence level respectively; for image data, features at different levels can be extracted according to pixel blocks, local areas and global structures; for audio data, hierarchical sampling can be performed according to the frame level, fragment level and overall waveform.

[0058] For example, for the full feature set consisting of N features extracted from the text modality, the feature subset can be generated by traversing all possible feature combinations, that is, generating Non-empty feature subsets. More specifically, for example, the full set of features extracted from the text modality ({part of speech, grammatical structure, semantic role}) can be combined into multiple feature subsets, such as a single-feature granularity of {part of speech}, {grammatical structure}, {semantic role}; a two-feature granularity of {part of speech, grammatical structure}, {part of speech, semantic role}, {grammatical structure, semantic role}; and a three-feature granularity of {part of speech, grammatical structure, semantic role}. Each feature subset is a granularity, and together they constitute the multi-granularity sample space for that modality.

[0059] S26, combining the multi-granularity sample spaces corresponding to the modal data to obtain a cross-modal multi-granularity sample space.

[0060] In this step, these features at different levels are combined into a cross-modal multi-granularity sample space, which contains feature sets from different modalities and different granularities for subsequent analysis and processing.

[0061] In other embodiments, those skilled in the art may further extract other features without being limited to the above description, and the present invention does not make any specific limitation thereto.

[0062] S3, according to the division of each modal data in the multimodal data set relative to the preset decision set and granularity, select the optimal granularity of the preset number that satisfies the coordination from the multi-granularity sample space of each modality to form an optimal granularity combination.

[0063] Specifically, such as Figure 2 and Figure 3 As shown, step S3 includes steps S31 to S33.

[0064] S31, given a decision set ,according to The sample set All samples in the multimodal dataset are divided into ; wherein the sample set A collection of collected TED speech samples.

[0065] Specifically, the “decision set " refers to the classification basis or target label set used to guide the fusion of multi-granularity knowledge. It contains the humor annotation information of each sample in the corpus, that is, the humor label (humorous or not humorous). The decision set , N represents the set The total number of samples in .

[0066] S32, assume that the granularity of each modal data is: text data: , image data: , audio data: , video data: , , Indicates the total number of granularities (features). A larger number indicates a finer granularity.

[0067] according to The sample sets All samples in are divided into .

[0068] S33, according to the granularity level from coarse to fine, find the multi-granularity sample space corresponding to each mode that meets the coordination granularities, the optimal granularity of each mode is obtained, and the optimal granularity combination is formed.

[0069] In this embodiment, coordination refers to the harmony, consistency, or complementarity exhibited by samples or features at different granularity levels under certain evaluation criteria. In the process of finding the optimal granularity combination, coordination is used as an important criterion for evaluating the effectiveness of samples or features at different granularity levels. By comparing the coordination at different granularity levels, it is possible to determine which granularity level has the most significant impact on overall performance.

[0070] The evaluation criteria for coordination in this embodiment are: according to the granularity level from coarse to fine, find the multi-granularity sample space corresponding to each mode that satisfies of granularity, and obtain the optimal granularity of each mode. That is: According to the division and division choose of The coarsest granularity is used to obtain the optimal granularity of text mode.

[0071] According to the division and division choose of The coarsest granularity is used to obtain the optimal granularity of the image modality.

[0072] According to the division and division choose of The coarsest granularity is used to obtain the optimal granularity of the audio mode.

[0073] According to the division and division choose of The coarsest granularity is used to obtain the optimal granularity of the video modality.

[0074] Among them, when satisfied When , it indicates the mode In particle size On and in decision sets Coordinate with others.

[0075] The optimal granularity of each modality constitutes the optimal granularity combination.

[0076] S4, based on the optimal granularity combination, using the discrimination measure and the variable coordination measure to perform cross-modal fusion, calculate a composite discrimination measure value to identify humorous content, and obtain a humor recognition result.

[0077] Specifically, such as Figure 4 As shown, step S4 includes steps S41 to S46.

[0078] S41, uses a variable consistency measure to describe the relationship between granularity and decision making, so as to flexibly describe the consistency of granularity in the dataset.

[0079] In this embodiment, the variable coordination metric is to evaluate the ability of a system or individual to dynamically adjust the degree of coordination and harmony between its components under different conditions, speeds, rhythms or environments. Its core is to quantify the level of coordination in this dynamic adaptation process. Variable coordination refers to the ability of a system or individual to flexibly adjust the degree of coordination between its internal components when faced with different conditions, speeds, rhythms or environmental changes to achieve optimal overall performance or performance. This coordination is not only reflected in a static equilibrium state, but also emphasizes adaptability and flexibility in dynamic changes.

[0080] Specifically, the text mode is granular Variable coordination measure under The calculation formula is: .

[0081] Image modality in granularity Variable coordination measure under The calculation formula is: .

[0082] Audio modality at granularity Variable coordination measure under The calculation formula is: .

[0083] Video modality in granularity Variable coordination measure under The calculation formula is: .

[0084] in, Represents the mode The next granularity; Respectively represent the particle size The following division.

[0085] S42, using variable coordination metrics to calculate the weight coefficients of each mode (corresponding to the weight coefficients of text, picture, audio, and video modalities respectively).

[0086] Specifically, assuming that there are four modalities of data, namely text, image, audio and video, the preset number of optimal granularity is a unified specific value, that is, the preset number of optimal granularity of the four modalities of data is the same, then the text modality weight coefficient is The calculation formula is: .

[0087] Image modality weight coefficient The calculation formula is: .

[0088] Weight coefficient of audio mode The calculation formula is: .

[0089] Video modality weight coefficient The calculation formula is: .

[0090] in, Indicates the preset number of optimal granularity. Of course, the preset numbers of optimal granularity of the four modal data can also be set to different numbers, which can be set according to actual needs.

[0091] If the preset number of optimal granularities for each modality is different, cross-modal fusion requires first vectorizing and mapping the optimal granularities of each modality to the same dimension. Specifically, the following steps are required: First, the features in the optimal granularity of each mode are vectorized to obtain the feature vector of each mode; Secondly, to ensure subsequent operations, the feature vectors of all modalities can be mapped to the same dimension using methods such as fully connected layers; Then, each mapped feature vector is multiplied by its corresponding weight coefficient; Finally, all weighted modal eigenvectors are concatenated.

[0092] This process ensures that the complementarity and synergy between different modalities are fully utilized, while retaining important contextual information and inter-modal correlation features.

[0093] S43, according to the division and division Calculate the different granularities of each mode Conditional entropy under ; Specifically, text conditional entropy The calculation formula is: ; ; .

[0094] Image conditional entropy The calculation formula is: .

[0095] Audio Conditional Entropy The calculation formula is: .

[0096] Video Conditional Entropy The calculation formula is: .

[0097] in, is an equivalence class probability; is an equivalence class Decision-making The conditional probability of Indicates the cardinality of a set.

[0098] S44, according to the conditional entropy of each mode Calculate the different granularities of each mode The discrimination measure under .

[0099] Specifically, the discriminative measure is a concept in information theory that reflects the uncertainty and discriminative ability of data. Then, the text discriminative measure The calculation formula is: ; .

[0100] Image discrimination measure The calculation formula is: .

[0101] Audio discrimination measure The calculation formula is: .

[0102] Video discrimination measure The calculation formula is: .

[0103] in, is an equivalence class probability; Indicates the cardinality of a set.

[0104] S45, based on the weight coefficient of each modality, cross-modal fusion calculates the composite discrimination measure value .

[0105] Composite discrimination measure The calculation formula is: ; in, Indicates the preset number of optimal granularity.

[0106] S46, identifying the humorous content to obtain a humorous recognition result.

[0107] Setting humor detection thresholds , if the composite discrimination measure value , it is judged as humorous content, otherwise it is non-humorous content.

[0108] The embodiments of the present invention establish a comprehensive dataset containing multimodal information such as text, images, audio, and video through multimodal data collection and preprocessing. Text features such as part of speech, grammar, emotion, and phonology, image features such as color, posture, and movement, audio features such as pitch and intonation, and video features such as scene and plot are extracted from different modal data to construct a cross-modal multi-granularity sample space. A coordination-based optimal granularity selection method is used to select the optimal granularity for each modality. A discriminant measure and a variable coordination metric are used for cross-modal fusion. A composite discriminant measure value is calculated to identify and classify humorous content.

[0109] The specific application scenarios of the present invention include, but are not limited to, automatic detection of humorous content on social media platforms, recommendation of humorous teaching resources in online education, and generation of humorous dialogues in intelligent customer service systems. In social media platforms, the present invention can quickly identify content with humorous attributes and classify and label it by analyzing the text, pictures, and voice content posted by users, thereby improving the user experience. In the field of online education, the present invention can assist teachers in screening humorous materials suitable for classroom use and enhance the fun of teaching. In intelligent customer service systems, the present invention can help robots understand users' humorous intentions and generate more natural and humane reply content.

[0110] In summary, compared with the prior art, the present invention has the following beneficial effects: First, through the collaborative representation of multimodal data, it integrates multimodal information such as text, images, audio, and video, fully exploring the multidimensional characteristics of humorous content and overcoming the problem of insufficient utilization of single-modal information. For example, in a TED talk sample, the text modality provides humorous clues at the linguistic level, the image modality supplements the expression and scene information, the audio modality captures the tone and intonation, and the video modality records the action and plot development. Together, these information constitute a comprehensive description of the humorous content.

[0111] Second, through a multi-granularity knowledge mining strategy, features are extracted from different granularity levels, avoiding the information loss that may occur when extracting features at a fixed granularity and improving the comprehensiveness and flexibility of feature representation. For example, in text modalities, coarse-grained features focus on the overall semantic structure, while fine-grained features focus on local vocabulary collocation. The combination of the two can more accurately portray the semantic connotation of humorous expressions.

[0112] Third, the present invention uses coordination to select the optimal granularity, combining a discriminative measure with a variable coordination metric to achieve the optimal granularity combination for cross-modal fusion. This accurately selects the optimal feature combination, effectively integrating the complementarity and synergy of each modal data and enhancing the effectiveness of feature fusion. For example, during the optimal granularity selection process, coordination is used to filter out the feature subset that best represents humorous content, thereby improving the accuracy of feature expression. During cross-modal fusion, the importance of each modality is weighted using a variable coordination metric, avoiding the problem of uneven weight distribution between modalities.

[0113] Fourth, the present invention improves the accuracy and robustness of humor recognition by calculating composite discrimination measures for humor identification and classification. For example, by setting thresholds and combining composite fusion, it effectively captures the various modal features of humorous content, thereby improving classification performance.

[0114] In summary, the method of the present invention not only solves the limitations of existing technologies in cross-modal multi-granularity sampling and fusion, but also significantly improves the performance of humor recognition, providing new research ideas and technical paths in the field of humor recognition, and has important theoretical value and practical application prospects.

[0115] Example 2 like Figure 5 As shown, the second embodiment of the present invention further provides a cross-modal multi-granularity humor recognition device based on a composite discrimination measure, comprising: An acquisition unit, used to acquire and preprocess multimodal data to establish a multimodal dataset; A multi-granularity sampling unit, configured to perform multi-granularity sampling in each modality based on the multi-modal dataset to obtain a cross-modal multi-granularity sample space; an optimal granularity unit, configured to select a preset number of optimal granularities that satisfy coordination from the multi-granularity sample space of each modality based on the division of each modal data in the multimodal data set relative to the preset decision set and granularity, to form an optimal granularity combination; The recognition unit is used to perform cross-modal fusion based on the optimal granularity combination using a discrimination measure and a variable coordination measure to calculate a composite discrimination measure value to recognize humorous content and obtain a humor recognition result.

[0116] Example 3 A third embodiment of the present invention further provides a cross-modal multi-granular humor recognition device based on a composite discriminant measure, comprising a memory and a processor. The memory stores a computer program, and the computer program can be executed by the processor to implement the cross-modal multi-granular humor recognition method based on a composite discriminant measure as described above.

[0117] Example 4 A fourth embodiment of the present invention further provides a computer-readable storage medium having computer-readable instructions stored thereon. When the computer-readable instructions are executed by a processor of a device where the computer-readable storage medium is located, the cross-modal and multi-granular humor recognition method based on a composite discriminative measure is implemented.

[0118] In the several embodiments provided in the embodiments of the present invention, it should be understood that the disclosed apparatus and method can also be implemented in other ways. The apparatus and method embodiments described above are merely illustrative. For example, the flowcharts in the accompanying drawings show apparatus and methods according to multiple embodiments of the present invention. In this regard, each box in the flowchart or block diagram can represent a module, a program segment or a portion of code, and the module, program segment or a portion of code contains one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in an order different from that marked in the accompanying drawings. For example, two consecutive boxes can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flowchart, and the combination of boxes in the block diagram and / or flowchart, can be implemented using a dedicated hardware-based system that performs the specified function or action, or can be implemented using a combination of dedicated hardware and computer instructions.

[0119] In addition, the functional modules in the various embodiments of the present invention may be integrated together to form an independent part, or each module may exist independently, or two or more modules may be integrated to form an independent part.

[0120] The terms used in the embodiments of the present invention are only for the purpose of describing specific embodiments and are not intended to limit the present invention. The singular forms "a", "an", "the" and "the" used in the embodiments of the present invention and the appended claims are also intended to include plural forms unless the context clearly indicates otherwise.

[0121] It should be understood that the term "and / or" as used herein is merely a description of the relationship between associated objects, indicating that three possible relationships exist. For example, "A and / or B" can represent: A exists alone, A and B exist simultaneously, or B exists alone. Furthermore, the character " / " in this document generally indicates that the associated objects are in an "or" relationship.

[0122] The word "if," as used herein, may be interpreted as "at the time of" or "when" or "in response to determining" or "in response to detecting," depending on the context. Similarly, the phrases "if it is determined" or "if (stated condition or event) is detected" may be interpreted as "when it is determined" or "in response to the determination" or "when detecting (stated condition or event)" or "in response to detecting (stated condition or event)," depending on the context.

[0123] The "first" and "second" mentioned in the embodiments are merely used to distinguish similar objects and do not represent a specific ordering of the objects. It is understood that the specific order or precedence of "first" and "second" can be interchanged where appropriate. It should be understood that the objects distinguished by "first" and "second" can be interchanged where appropriate, so that the embodiments described herein can be implemented in an order other than that illustrated or described herein.

[0124] The foregoing description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Those skilled in the art will readily appreciate that various modifications and variations of the present invention are possible. Any modifications, equivalent substitutions, or improvements made within the spirit and principles of the present invention are intended to be within the scope of protection of the present invention.

Claims

1. A cross-modal and multi-granular humor recognition method based on composite discriminant measure, characterized by: include: S1, acquire multimodal data and preprocess them to establish a multimodal dataset; S2, based on the multimodal dataset, performing multi-granularity sampling in each modality to obtain a cross-modal multi-granularity sample space; S3, based on the division of each modal data in the multimodal dataset with respect to the preset decision set and granularity, selecting a preset number of optimal granularities that satisfy coordination from the multi-granularity sample space of each modality to form an optimal granularity combination; S4, based on the optimal granularity combination, using the discrimination measure and the variable coordination measure to perform cross-modal fusion, calculate a composite discrimination measure value to identify humorous content, and obtain a humor recognition result.

2. A cross-modal multi-granular humor recognition method based on composite discrimination measure according to claim 1, characterized in that ,The multimodal data includes text data, and / or image data, and / or audio data, and / or video data.

3. A cross-modal multi-granular humor recognition method based on composite discrimination measure according to claim 2, characterized in that The preprocessing includes: word segmentation, stop word removal, and / or standardization of text data; size normalization and / or color space conversion of image data; noise reduction and / or frame segmentation of audio data; frame extraction and / or action extraction of video data.

4. A cross-modal multi-granular humor recognition method based on composite discriminant measure according to claim 1, characterized in that ,It also includes humor annotation and alignment of pre-processed multimodal data.

5. A cross-modal multi-granular humor recognition method based on composite discriminant measure according to claim 1, characterized in that ,The process of acquiring the cross-modal multi-granularity sample space is specifically as follows: Extracting features of each modality data based on the multimodal dataset; Obtaining multiple feature subsets based on the extracted features; Different feature subsets constitute different granularities. Based on different feature subsets, the data set is divided to obtain multi-granularity sample spaces of each modality. The multi-granularity sample spaces corresponding to each modal data are combined to obtain a cross-modal multi-granularity sample space.

6. A cross-modal multi-granular humor recognition method based on composite discriminant measure according to claim 1, characterized in that ,The selection process of the optimal granularity combination specifically includes: Get the preset decision set ; according to For the multimodal dataset All samples in are divided into ; According to the cross-modal multi-granularity sample space, the multimodal dataset All samples in are divided into , Indicates the level of granularity; ; Indicates the total number of granularities. The larger the number, the finer the granularity division. Indicates the indivual the granularity of the modality; According to the granularity level from coarse to fine, find the multi-granularity sample space corresponding to each mode that satisfies of granularities, obtaining the optimal granularity of each mode, and forming the optimal granularity combination; Among them, when satisfied When , it indicates the mode In particle size On and in decision sets Coordinate with others.

7. The cross-modal multi-granular humor recognition method based on composite discriminant measure according to claim 6, characterized in that: The process of using the discrimination measure and variable coordination measure to perform cross-modal fusion to calculate the composite discrimination measure value is as follows: Use variable coordination metrics to describe the relationship between the granularity of each modality and the decision; then, the modality In particle size Variable coordination measure under The calculation formula is: ； The variable compatibility metric is used to calculate the weight coefficient of each mode. The weight coefficient The calculation formula is: ； in, Represents various modes; Indicates one of the modes; Indicates the preset number of optimal granularity; Indicates modality In particle size variable coordination measures under ; According to the division and division Calculate the different granularities of each mode Conditional entropy under ; Then, the modal In particle size Conditional entropy under The calculation formula is: ； ； ； in, is an equivalence class probability; is an equivalence class Decision-making The conditional probability of Indicates the cardinality of a set; According to the conditional entropy of each mode Calculate the different granularities of each mode The discrimination measure under , then the modal In particle size The discrimination measure under The calculation formula is: ； ； in, is an equivalence class probability; is modal In particle size Conditional entropy under ; Indicates the cardinality of a set; Based on the weight coefficient of each modality, cross-modal fusion calculates the composite discrimination measure value , composite discrimination measure value The calculation formula is: ； in, Indicates modality In particle size The discrimination measure below.

8. A cross-modal multi-granular humor recognition method based on composite discriminant measure according to claim 1, characterized in that ,The process of identifying humor content is as follows: Setting humor detection thresholds , if the calculated composite discrimination measure value , it is judged as humorous content, otherwise it is non-humorous content.

9. A cross-modal multi-granular humor recognition device based on a composite discriminant measure, for implementing a cross-modal multi-granular humor recognition method based on a composite discriminant measure according to any one of claims 1 to 8, characterized in that: include: An acquisition unit, used to acquire and preprocess multimodal data to establish a multimodal dataset; A multi-granularity sampling unit, configured to perform multi-granularity sampling in each modality based on the multi-modal dataset to obtain a cross-modal multi-granularity sample space; an optimal granularity unit, configured to select a preset number of optimal granularities that satisfy coordination from the multi-granularity sample space of each modality based on the division of each modal data in the multimodal data set relative to the preset decision set and granularity, to form an optimal granularity combination; The recognition unit is used to perform cross-modal fusion based on the optimal granularity combination using a discrimination measure and a variable coordination measure to calculate a composite discrimination measure value to recognize humorous content and obtain a humor recognition result.

Images