A multi-modal open intention recognition method and system based on granulocyte characterization

Abstract

The application discloses a multi-modal open intention recognition method and system based on granular ball representation, and belongs to the technical field of artificial intelligence and multi-modal intention understanding, which comprises the following steps: performing feature extraction and mode fusion on multi-modal input data; performing structural modeling on the feature representation of each mode and the fusion mode by using a self-adaptive granular ball clustering method, and generating a multi-granularity granular ball set; taking the granular ball centroid as a multi-granularity anchor point, and aligning the granular balls with the same label in different modes; introducing a weighting mechanism based on purity and sample size in the fusion mode; generating boundary-constrained pseudo-distribution out-of-distribution samples in the fusion mode space; constructing a self-adaptive decision boundary based on the fusion mode granular balls obtained through training, and performing known class classification and unknown class detection. By introducing the multi-granularity anchor point and the structure-perception granular ball representation mode, the application realizes the joint recognition of known classes and unknown classes in a multi-modal scene, and significantly improves the accuracy and robustness of intention recognition.

Description

Technical Field

[0001] This invention relates to the fields of artificial intelligence and multimodal intent understanding, and in particular to a method and system for multimodal open intent recognition based on particle-sphere representation. Background Technology

[0002] The goal of the multimodal open intent recognition task is to utilize information from multiple modalities, such as text, video, and audio, to accurately classify in-distribution (ID) intents while effectively detecting out-of-distribution (OOD) intents. This capability is of great significance for building robust and secure human-computer interaction systems.

[0003] In existing technologies, cross-modal alignment is often a core component of multimodal representation learning. Common methods are mostly based on contrastive learning, which narrows modal differences by bringing samples with the same label or temporal synchronization relationship closer together and pushing other samples further apart. However, this rigid instance-level alignment is often too strict and can easily compress modality-specific semantic information, leading to decreased representational ability and insufficient generalization. Different modalities actually encode complementary information of the same instance, but they are not exactly the same. Therefore, forced alignment may result in information loss and semantic ambiguity.

[0004] To relax constraints, some methods attempt to introduce cluster alignment in multimodal scenarios. However, most existing clustering methods rely on a preset number of clusters (such as k-means), and the clustering results are relatively coarse, making it difficult to accurately characterize the real data structure. This deficiency directly affects the effectiveness of cross-modal alignment, limiting the model's performance in complex open environments. Therefore, how to achieve flexible and efficient cross-modal alignment while maintaining modal characteristics remains a pressing technical challenge. Furthermore, existing technologies also have shortcomings in detecting out-of-world (OOD) intent in open-world scenarios. On the one hand, models lack a modeling mechanism for the distribution of real open spaces; on the other hand, decision boundaries mostly rely on fixed thresholds, making it difficult to adapt to different spatial structures, resulting in unsatisfactory detection rates for unknown intents.

[0005] Therefore, there is an urgent need for a new multimodal open intent recognition method that can achieve flexible alignment while maintaining modal characteristics, and combine adaptive structural modeling and boundary awareness mechanisms to improve recognition and detection capabilities in open worlds. Summary of the Invention

[0006] The purpose of this invention is to overcome the problems of poor cross-modal alignment generalization, loss of modal characteristics, and insufficient OOD intent detection in the prior art. It provides a multimodal open intent recognition method and system based on granular sphere representation. By introducing multi-granularity anchor points and structure-aware granular sphere representation, it realizes the joint recognition of known and unknown categories in multimodal scenarios, which significantly improves the accuracy and robustness of intent recognition.

[0007] The objective of this invention is achieved through the following technical solution:

[0008] Firstly, a multimodal open intent recognition method based on particle-sphere representation is provided, including the following steps:

[0009] S1. Perform feature extraction and modality fusion on multimodal input data to obtain high-level semantic representations of text, video, audio, and fused modalities;

[0010] S2. Particle structure modeling: The feature representations of each mode and the fusion mode are structurally modeled using an adaptive particle clustering method to generate a multi-granularity particle set;

[0011] S3. Structure-aware particle-level contrastive learning: Using the centroid of the particle as a multi-granularity anchor point, aligning particles with the same label in different modalities; introducing a weighting mechanism based on purity and sample size in the fusion modality;

[0012] S4. Pseudo-out-of-distribution sample generation and representation learning: Generate pseudo-out-of-distribution samples with boundary constraints in the fusion modality space;

[0013] S5. Inference phase: Based on the fusion modal particles obtained from training, construct an adaptive decision boundary and perform known class classification and unknown class detection.

[0014] In some embodiments, the feature extraction and modality fusion of the multimodal input data includes:

[0015] Text modality uses BERT to extract features, video modality uses Swing Transformer to extract features, and audio modality uses WavLM to extract features.

[0016] By using a weighted fusion strategy, the representations of each modality are merged into a unified representation, resulting in the feature representation of the fused modality.

[0017] In some embodiments, the adaptive particle clustering method includes:

[0018] The sample set is recursively split and aggregated to generate multiple spheres, each characterized by its centroid, radius, purity, size, and label attributes.

[0019] The splitting condition is that the particle purity is below the threshold and the number of samples is above the threshold, while the aggregation condition is that the particle purity is above the threshold or the number of samples is below the threshold.

[0020] In some embodiments, aligning particles with the same label in different modalities includes:

[0021] Use the following total loss function:

[0022]

[0023] in, This represents the total loss of multimodal alignment. This represents the alignment contrast loss between the text and video modalities. This represents the alignment contrast loss between the text and speech modalities.

[0024] In some embodiments, intra-class comparison targets are designed in the fusion modality of step S3.

[0025] In some embodiments, generating pseudo-out-of-distribution samples with boundary constraints in the fusion modality space includes:

[0026] Candidate pseudo-distribution samples are generated by extrapolating the centroids of different types of granules; and a radius filtering mechanism is used to remove samples that fall within the known granule class range.

[0027] In some embodiments, generating pseudo-out-of-distribution samples with boundary constraints in the fusion modality space further includes:

[0028] Introducing OOD separation loss increases the distance between samples outside the pseudo-distribution and the centroids of particles within the distribution.

[0029] In some embodiments, the final overall optimization objective function is:

[0030]

[0031] in, This represents the cross-modal particle-sphere alignment loss; This represents the structure-aware contrastive loss of the fusion modality; This represents the ID-OOD separation loss for pseudo-OOD samples; Indicates standard classification loss; This represents the weight hyperparameter of the loss term.

[0032] In some embodiments, step S5 specifically includes:

[0033] The spheroid clustering results of the fusion modality are used as the initial decision boundary for the in-distribution categories, and a scaling factor is introduced to expand the spheroid radius to form the final category determination region.

[0034] Secondly, a multimodal open intent recognition system based on particle-sphere representation is provided, including:

[0035] The feature extraction and fusion module is used to extract features and fuse modalities from multimodal input data to obtain high-level semantic representations of text, video, audio, and fused modalities.

[0036] The particle modeling module is used to perform particle structure modeling. It uses an adaptive particle clustering method to perform structural modeling on the feature representations of each mode and the fusion mode, generating a multi-granularity particle set.

[0037] The contrastive learning module is used for structure-aware particle-level contrastive learning, using the particle centroid as a multi-granularity anchor point to align particles with the same label in different modalities; a weighting mechanism based on purity and sample size is introduced in the fusion modality.

[0038] The pseudo-out-of-distribution sample generation module is used to generate pseudo-out-of-distribution samples and learn their representations, generating boundary-constrained pseudo-out-of-distribution samples in the fusion modality space.

[0039] The inference module is used to perform the inference phase. It constructs an adaptive decision boundary based on the fusion modal particles obtained from training and performs known class classification and unknown class detection.

[0040] It should be further noted that the technical features corresponding to the above embodiments can be combined or substituted with each other to form new technical solutions without conflict.

[0041] Compared with existing technologies, this invention, by introducing multi-granularity anchor points and a structure-aware granular-sphere representation method, achieves joint recognition of known and unknown categories in multimodal scenarios, significantly improving the accuracy and robustness of intent recognition. Specific beneficial effects are as follows:

[0042] 1. This invention employs an adaptive granular clustering method to model the feature space of multimodal samples. Through recursive splitting and aggregation operations, a series of granular spheres are generated without requiring a preset number of clusters. Each sphere is characterized by attributes such as centroid, radius, purity, and size. This modeling process can generate multi-granular semantic structures, effectively approximating the true intra-class distribution features. Compared to traditional static clustering methods, the granular sphere structure of this invention possesses adaptability, interpretability, and robustness, simultaneously reflecting macroscopic class distribution and local fine-grained structure, providing structured basic units for subsequent cross-modal alignment and open-space modeling.

[0043] 2. This invention proposes a structure-aware contrastive learning method based on particle structure modeling. This method uses the particle centroid as a multi-granularity anchor point. During cross-modal alignment, it only pairs particles with the same label, avoiding the loss of modal characteristics caused by instance-level forced alignment. By introducing a weighting mechanism based on purity and sample size, high-purity particles with large sample sizes receive greater weight during optimization, thereby reducing interference from low-quality particles. In fusion modalities, this invention further designs intra-class contrast targets, making particles of the same class more compact and maintaining sufficient distinction between particles of different classes. This mechanism effectively improves the consistency between modalities and the discriminative ability of the fusion space, while preserving the unique features of each modality, taking into account multimodal semantic sharing and structural differences, and significantly improving robustness in open scenarios.

[0044] 3. To enhance the recognition capability of open spaces, this invention proposes a boundary-constrained pseudo-OOD sample generation strategy. In the fusion modality space, centroids of different classes of particles are first selected, and candidate pseudo-OOD samples are generated through extrapolation synthesis. Then, a radius filtering mechanism is used to remove invalid samples falling within the radius range of known particle classes, retaining only high-quality pseudo-OOD samples located outside the boundary. Furthermore, this invention introduces an OOD separation loss function to forcibly increase the gap between pseudo-OOD samples and the centroids of ID particles, thereby achieving effective differentiation between ID and OOD. This strategy does not rely on additional unknown class data, can simulate the distribution characteristics of real open spaces, significantly improves the model's ability to detect unknown intentions in the absence of prior information, and has practicality and scalability.

[0045] 4. In the inference phase, this invention constructs an adaptive decision boundary based on the fusion modality particles obtained during training. Each category is represented by multiple particles, and the category boundary is defined by the centroid and radius. To improve detection flexibility, this invention introduces a radius scaling factor to uniformly expand or shrink the radius of each particle. Specifically, when a test sample falls within the scaling boundary of a certain category particle, it is classified as belonging to that category; if it falls outside the boundaries of all category particles, it is classified as an unknown class. By adjusting the scaling factor, the classification accuracy and the detection rate of unknown classes can be balanced according to the actual application. Compared with the traditional fixed threshold method, this scheme can dynamically adapt to the data distribution of different scenarios, realize the integrated processing of known class classification and unknown class detection, and significantly improve the robustness and reliability of the system in open environments. Attached Figure Description

[0046] Figure 1 This is a flowchart of a multimodal open intent recognition method based on particle-sphere representation according to the present invention. Detailed Implementation

[0047] The technical solution of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. The components of the embodiments of this application described and shown in the accompanying drawings can generally be arranged and designed in various different configurations. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0048] It should be noted that the defects in the solutions in the prior art are all the results of the inventors' practice and careful research. Therefore, the discovery process of the above problems and the solutions proposed by the embodiments of this application in the following text should be the inventors' contributions to this application in the process of invention and creation, and should not be understood as technical content known to those skilled in the art.

[0049] This invention first extracts representations of single-modal and fused modalities, and models multi-granular structural information using an adaptive particle clustering method. Next, it performs structure-aware contrastive learning, enhancing modal consistency through particle alignment with consistent inter-modal labels, while simultaneously optimizing the fused modal representation using an attribute-guided weighting mechanism. Then, a boundary-constrained pseudo-OOD sample generation module is designed to synthesize pseudo-unknown samples outside the known class decision boundary, and an OOD separation loss is introduced to enhance open set discrimination capability. During the inference phase, the decision boundary is constructed using particle radius scaling learned during training, achieving a joint task of known class classification and unknown class detection.

[0050] In one exemplary embodiment, reference is made to Figure 1 This paper provides a multimodal open intent recognition method based on particle-sphere representation, including the following steps:

[0051] S1. Perform feature extraction and modality fusion on multimodal input data to obtain high-level semantic representations of text, video, audio, and fused modalities;

[0052] S2. Particle structure modeling: The feature representations of each mode and the fusion mode are structurally modeled using an adaptive particle clustering method to generate a multi-granularity particle set;

[0053] S3. Structure-aware particle-level contrastive learning: Using the centroid of the particle as a multi-granularity anchor point, aligning particles with the same label in different modalities; introducing a weighting mechanism based on purity and sample size in the fusion modality;

[0054] S4. Pseudo-out-of-distribution sample generation and representation learning: Generate pseudo-out-of-distribution samples with boundary constraints in the fusion modality space;

[0055] S5. Inference phase: Based on the fusion modal particles obtained from training, construct an adaptive decision boundary and perform known class classification and unknown class detection.

[0056] Before describing the specific implementation process, a formal definition of the problem is given, based on a multimodal intent recognition dataset. Each intent sample It includes three modes, among which, Representing text, Indicates video, This represents audio. The in-distribution (known) intent sample dataset is defined as follows:

[0057]

[0058] Among them, sample labels This indicates that the sample belongs to One of the known intent categories.

[0059] The Out-of-Distribution (OOD) intent sample dataset is defined as follows:

[0060]

[0061] The samples came from unseen categories, and each sample was assigned a uniform label. This is represented as an unknown class. Only used during the training and validation phases. ,and Available only during the testing phase. The goal is to build a model that can accurately classify ID intent samples and identify OOD intent samples in testing.

[0062] For example, granular structure modeling specifically includes:

[0063] 1. Feature extraction and modality fusion

[0064] We first extract low-level features from text, video, and audio modalities, and then perform modality fusion. Specifically, for the text modality, we use BERT to extract text embeddings, resulting in:

[0065]

[0066] Video modalities are used to extract visual features using the Swing Transformer:

[0067]

[0068] The audio modality is then extracted using WavLM to embed the audio data.

[0069]

[0070] in, Indicates the length of each modal sequence. This represents the feature dimension of each modality.

[0071] Next, the extracted features from the three modalities are input into the Transformer encoder to extract high-level semantic representations:

[0072] For text modalities, the [CLS] representation of the last layer is taken as the sentence-level representation:

[0073]

[0074] in , represents the sentence-level feature vector of the text modality, and [CLS] represents the output state of the corresponding classification token ([CLS] token) in the last layer of the Transformer, which serves as the aggregate representation of the entire input sequence.

[0075] For video and audio modalities, the classic Transformer, average pooling, and linear transformation are applied sequentially:

[0076]

[0077]

[0078] in, , It is a linear mapping matrix. It is a unified fusion feature dimension.

[0079] Subsequently, a weighted modality fusion strategy was adopted to fuse the representations of the three modalities into a single overall representation. The calculation method is as follows:

[0080]

[0081] in Representing modes The weights are obtained through adaptive learning by a lightweight neural network.

[0082] 2. Adaptive Particle Clustering

[0083] Particle-spherical clustering can adaptively capture fine-grained structures with different granularities, forming spherical representations of varying sizes, without requiring a pre-specified number of clusters, and can approximate the true distribution well. Because of its good performance in preserving the structural properties of the sample distribution, it is applied to the feature representations of each modality and fused modality to guide subsequent representation learning.

[0084] 2.1. Definition and properties of granules

[0085] Granular-balls are adaptive clustering units that can characterize the true distribution of data. Let the sample set... ,in For the feature vector of the sample, This represents the category label of the sample. By clustering this sample set, a series of spheres can be obtained, represented as a set. Each of the granules Depend on It consists of 10 samples and has the following attributes:

[0086] Particle size The number of samples contained in a sphere; the centroid of the sphere. Mean of all sample features within the sphere; sphere radius : The average Euclidean distance from all samples within the sphere to the centroid; sphere label The label representing the category with the highest percentage of granules; granule purity. : The contents of the pellet belong to the pellet label The sample proportion.

[0087] 2.2. Particle clustering process

[0088] Given a mode The sample is represented as:

[0089]

[0090] First, all samples are clustered into a coarse-grained sphere. Then, an adaptive clustering process is used to recursively divide it into finer-grained spheres. When the sphere purity of a sphere... And sample size When this happens, a partitioning operation is triggered. During the partitioning process, if a particle contains samples from multiple categories, its label set is... Then it will be classified as A new pellet. For each category A sample belonging to that category is randomly selected as a pseudo-center. Subsequently, all samples in this sphere are reassigned to the new sphere corresponding to the nearest pseudo-center based on their Euclidean distance from the pseudo-center.

[0091] When each sphere satisfies: 1) its sphere purity , or 2) number of samples At this point, the partitioning process terminates. Ultimately, each mode (and its fused mode) will result in a set of spheres.

[0092]

[0093] Each sphere It has five attributes:

[0094]

[0095] These represent the tag set, purity, size, center point, and radius, respectively.

[0096] For example, structure-aware granular-level contrastive learning specifically includes:

[0097] To avoid information loss caused by overly strict instance (sample) level alignment and to better preserve modal characteristics, this invention performs structure-aware contrastive learning at the particle level. This strategy comprises two parts: 1) Multimodal particle-sphere contrastive learning: aligning particles with the same label across different modalities; 2) Fusion modal contrastive learning: enhancing the separability between categories. To reduce the interference of low-quality samples on contrastive learning, a weighting mechanism is introduced, prioritizing high-purity, large-sample-volume positive-negative sample pairs.

[0098] 1. Cross-modal alignment

[0099] This invention uses the center point of the particle in the text modality as the anchor point to align the video and audio modalities to the text modality. We first calculate the loss for aligning the video modality to the text modality:

[0100] Let the set of particles for the text modality be: Each sphere contains: a center point ,Label ,purity ,size .

[0101] Let the set of particles for the video mode be: Each sphere contains: a center point ,Label ,purity ,size .

[0102] Given the center point of a text modality By comparing these samples with the particles in the video modality, positive and negative sample sets are formed, defined as follows:

[0103]

[0104] in, Represents the positive sample set. Represents the negative sample set. This represents the label of the j-th particle in the video modality. This represents the label of the i-th particle in the text modality.

[0105] For each pair of pellets The weighting coefficients are defined according to the following formula:

[0106]

[0107] in, This represents the purity of the i-th particle in the text modality. This represents the purity of the j-th particle in the video modality. This represents the size of the i-th particle in the text modality. This represents the size of the j-th particle in the video modality.

[0108] Subsequently, the alignment contrast loss between the text and video modal is defined as follows:

[0109]

[0110] in, Represents cosine similarity. It is the temperature coefficient. The set of particles representing a text modality contains the number of particles, i.e., the number of text modality center points. This represents the center point of the video modal particle sphere. Similarly, the text-audio alignment loss can be defined using the same method. Ultimately, the total loss for multimodal alignment is:

[0111]

[0112] 2. Fusion Modal Comparison Learning

[0113] To enhance the inter-class compactness and intra-class separability of the fused modality representation, a structure-aware contrast target is further defined on the set of fused modality spheres. Let the set of fused modality spheres be:

[0114]

[0115] The center point of each sphere is , tag as For each sphere Its positive and negative sample sets are defined as follows:

[0116]

[0117] in, The set of positive samples representing the fusion modality. The negative sample set represents the fusion modality. This represents the label of the j-th particle in the fusion mode. The label represents the i-th particle in the fusion mode.

[0118] The weighting formula is consistent with that in cross-modal alignment:

[0119]

[0120] The contrast loss of the fused modes is shown below:

[0121]

[0122] in, Indicates the number of particles in the fusion mode. This represents the center of the i-th particle in the fusion mode. This represents the center of the j-th particle in the fusion mode.

[0123] For example, pseudo-out-of-distribution sample generation and representation learning specifically include:

[0124] Generating pseudo-OOD (pseudo-unknown class) samples and incorporating them into training is an effective method to improve OOD detection performance. This invention proposes a pseudo-OOD sample generation strategy based on boundary constraints, applied to particle-sphere representation of fusion modalities, utilizing particle-sphere properties to generate high-quality pseudo-OOD representations. The specific process is as follows:

[0125] 1) Initial Sample Generation: First, initial pseudo-OOD samples are generated based on the centroid of the fused modal particles. 2) Boundary Filtering: Pseudo-samples falling within the ID (Known Class) region are filtered out to ensure they are distributed outside the ID decision boundary.

[0126] Specifically, for each centroid (From fusion modality, tagged as) ), find its nearest from different categories (i.e. The center of mass of ) .from Towards Directional extrapolation generates pseudo-OOD samples:

[0127]

[0128] in, Indicates the center of the fusion mode The pseudo-OOD samples generated by extrapolation It is a random number sampled from a uniform distribution, introducing diversity; each pair of centroids Can generate A pseudo-OOD sample, in the experiment Next, samples falling within the ID decision boundary are filtered out using the following radius condition, retaining valid pseudo-samples:

[0129]

[0130] in, This represents the center of the k-th sphere in the fusion mode. This represents the radius of the k-th particle in the fusion mode, meaning the minimum distance between the pseudo-sample and the centers of all particle spheres must be greater than the corresponding particle sphere radius. Furthermore, an OOD separation loss is defined. This explicitly pushes the centroid of ID samples away from the preserved pseudo-OOD samples, thereby enhancing the separation between ID and OOD representations. The definition is as follows:

[0131]

[0132] in, Indicates the number of pseudo-OOD samples retained; Indicates the number of particles in the fusion mode; Indicates the first A pseudo-OOD representation; Indicates the first The centroid of each fused mode particle. Furthermore, this invention applies a standard cross-entropy supervision loss to the representation of the fused modes, defined as follows:

[0133]

[0134] in, This represents all IDs and pseudo-OOD samples; Indicates the first Each sample corresponds to a category one-hot tags; Indicates inclusion ID category and the The total set of categories for classes (used for pseudo-OOD); This indicates a cosine-based classification head. express The samples in.

[0135] The final overall optimization objective consists of the following four losses:

[0136]

[0137] in, This represents the cross-modal particle-sphere alignment loss; This represents the structure-aware contrast loss of the fusion modality; This represents the ID-OOD separation loss for pseudo-OOD samples; Indicates standard classification loss; This represents the weight hyperparameter of the loss term.

[0138] Backpropagation is performed using the loss function described above to calculate the gradient and update the trainable parameters of the model, thereby gradually improving the model's performance in open intent classification.

[0139] Model updates specifically include:

[0140] 1) Validation set evaluation

[0141] After each round of training, the model's evaluation metric score on the validation set is calculated to assess the model's current performance.

[0142] 2) Update the optimal validation score and the optimal model.

[0143] If the current evaluation score is higher than the historical best score, then the score and the corresponding model parameters are set as the new optimal result.

[0144] If the current evaluation score is lower than or equal to the historical best score, the optimal model will not be updated.

[0145] 3) Continue training

[0146] Subsequent training iterations are based on the parameters of the current best model, which continuously improves the model's adaptability in complex noisy environments.

[0147] Training stopping and boundary saving specifically include:

[0148] If the validation score fails to improve after 10 consecutive iterations, an early stopping strategy is triggered, halting representation learning and saving the optimal model parameters. Simultaneously, the centroids and radii of each known class represented by the high-quality particles are recorded as the final class boundaries.

[0149] For example, the inference phase uses adaptive decision boundaries, specifically including:

[0150] The spheroid clustering results from the fusion modality are used as the initial decision boundary for ID categories, and a radius scaling factor Δ is introduced to expand the spheroid radius to form the final category determination region. For each ID category... Its decision boundary consists of multiple labels. granules As defined, when there are test samples When entering:

[0151] 1) Calculate its Euclidean distance from the centroids of all granules:

[0152]

[0153] 2) If the distance between this sample and the centroid of all granules is greater than the corresponding radius:

[0154]

[0155] The sample is then classified as OOD, where, The particle radius represents the fusion mode. Let j represent any sphere of the fusion mode.

[0156] 3) Otherwise, assign it to the category of the nearest ball.

[0157] In another exemplary embodiment, a multimodal open intent recognition system based on granular sphere representation is provided, comprising:

[0158] The feature extraction and fusion module is used to extract features and fuse modalities from multimodal input data to obtain high-level semantic representations of text, video, audio, and fused modalities.

[0159] The particle modeling module is used to perform particle structure modeling. It uses an adaptive particle clustering method to perform structural modeling on the feature representations of each mode and the fusion mode, generating a multi-granularity particle set.

[0160] The contrastive learning module is used for structure-aware particle-level contrastive learning, using the particle centroid as a multi-granularity anchor point to align particles with the same label in different modalities; a weighting mechanism based on purity and sample size is introduced in the fusion modality.

[0161] The pseudo-out-of-distribution sample generation module is used to generate pseudo-out-of-distribution samples and learn their representations, generating boundary-constrained pseudo-out-of-distribution samples in the fusion modality space.

[0162] The inference module is used to perform the inference phase. It constructs an adaptive decision boundary based on the fusion modal particles obtained from training and performs known class classification and unknown class detection.

[0163] The above detailed embodiments are a description of the present invention. It should not be considered that the specific embodiments of the present invention are limited to these descriptions. For those skilled in the art, several simple deductions and substitutions can be made without departing from the concept of the present invention, and all of these should be considered to fall within the protection scope of the present invention.

Claims

1. A multi-modal open intention recognition method based on granulocyte characterization, characterized in that, Comprise the following steps: S1. Feature extraction and modality fusion on multi-modal input data to obtain high-level semantic representations of text, video, audio and fusion modalities; S2. Granular ball structure modeling: structure modeling of feature representations of each modality and fusion modalities by an adaptive granular ball clustering method to generate a multi-granularity granular ball set; S3. Structure-aware granular ball level contrast learning: aligning granular balls with the same label in different modalities using granular ball centroids as multi-granularity anchors; introducing a weighting mechanism based on purity and sample size in the fusion modality; S4. Pseudo-distributed out-of-distribution sample generation and representation learning: generating boundary-constrained pseudo-distributed out-of-distribution samples in the fusion modality space; S5. Inference stage: based on the fusion modality granular ball constructed by training, an adaptive decision boundary is constructed, and known class classification and unknown class detection are performed.

2. The multi-modal open intent recognition method based on granulocyte characterization according to claim 1, characterized in that, The feature extraction and modality fusion on multi-modal input data comprises: The text modality uses BERT to extract features, the video modality uses Swin Transformer to extract features, and the audio modality uses WavLM to extract features; The representations of each modality are combined into a unified representation by a weighted fusion strategy to obtain the feature representation of the fusion modality. 3.The multi-modal open intention recognition method based on granulocyte characterization according to claim 1, characterized in that, The adaptive granular ball clustering method comprises: Recursively split and aggregate the sample set to generate multiple granular balls, each granular ball is characterized by centroid, radius, purity, size and label attribute; The splitting condition is that the purity of the granular ball is lower than the threshold and the number of samples is higher than the threshold, and the aggregation condition is that the purity of the granular ball is higher than the threshold or the number of samples is lower than the threshold.

4. The multi-modal open intent recognition method based on granulocyte characterization according to claim 1, characterized in that, The aligning granular balls with the same label in different modalities comprises: The following total loss function is used: wherein, represents a total loss of multi-modal alignment, represents an alignment contrast loss between the text and the video modality, represents an alignment contrast loss between the text and the speech modality.

5. The multi-modal open intent recognition method based on granulocyte characterization according to claim 1, characterized in that, An intra-class contrast objective is designed in the fusion modality of step S3.

6. The multi-modal open intent recognition method based on granulocyte characterization according to claim 1, characterized in that, The generation of boundary-constrained pseudo-distributed out-of-distribution samples in the fusion modality space comprises: Candidate pseudo-distributed out-of-distribution samples are generated by extrapolating different class granular ball centroids; and a radius filtering mechanism is used to remove samples falling within the range of known class granular balls.

7. The multi-modal open intent recognition method based on granulocyte characterization according to claim 6, characterized in that, The generation of boundary-constrained pseudo-distributed out-of-distribution samples in the fusion modality space further comprises: An OOD separation loss is introduced to improve the distance between the pseudo-distributed out-of-distribution samples and the centroids of the in-distribution granular balls.

8. The multi-modal open intent recognition method based on granular ball representation according to claim 1, wherein The final overall optimization objective function is wherein, represents the cross-modal granule alignment loss; represents the structure-aware contrastive loss that fuses modalities; represents the ID-OOD separation loss for pseudo OOD samples; represents the standard classification loss; represents the weight hyperparameters for the loss terms.

9. The multi-modal open intent recognition method based on granulocyte characterization according to claim 1, characterized in that, Step S5 specifically comprises: The granular ball clustering results of the fusion modality are used as the initial decision boundary of the in-distribution classes, and a scaling factor is introduced to expand the granular ball radius to form the final class determination region.

10. A multi-modal open intent recognition system based on granulocyte characterization, characterized by, Comprise: A feature extraction and fusion module for feature extraction and modality fusion on multi-modal input data to obtain high-level semantic representations of text, video, audio and fusion modalities; A granular ball modeling module for granular ball structure modeling by an adaptive granular ball clustering method on feature representations of each modality and fusion modalities to generate a multi-granularity granular ball set; A contrast learning module for structure-aware granular ball level contrast learning using granular ball centroids as multi-granularity anchors to align granular balls with the same label in different modalities; introducing a weighting mechanism based on purity and sample size in the fusion modality; The pseudo-out-of-distribution sample generation module is configured to perform pseudo-out-of-distribution sample generation and representation learning, and generate boundary-constrained pseudo-out-of-distribution samples in the fusion modality space. The inference module is configured to perform an inference stage, construct an adaptive decision boundary based on the fusion modality granule constructed through training, and perform known class classification and unknown class detection.

Images