Multi-modal open intention recognition method and system based on pellet characterization

Abstract

The invention discloses a multi-modal open intention recognition method and system based on pellet characterization, and belongs to the technical field of artificial intelligence and multi-modal intention understanding, and the method comprises the steps: carrying out the feature extraction and modal fusion of multi-modal input data; carrying out structural modeling on the feature representation of each mode and the fusion mode through an adaptive particle and ball clustering method, and generating a multi-granularity particle and ball set; the mass centers of the pellets serve as multi-granularity anchor points, and the pellets with the same labels in different modalities are aligned; introducing a weighting mechanism based on purity and sample scale into the fusion mode; generating a boundary-constrained pseudo-distribution outer sample in the fusion modal space; and constructing a self-adaptive decision boundary based on the fusion modal particle ball obtained by training, and carrying out known class classification and unknown class detection. According to the invention, by introducing the multi-granularity anchor point and the structure perception particle-ball representation mode, the joint recognition of the known category and the unknown category in the multi-modal scene is realized, and the accuracy and robustness of intention recognition are remarkably improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of artificial intelligence and multi-modal intent understanding, and in particular to a multi-modal open intent recognition method and system based on granular ball representation. BACKGROUND

[0002] The goal of the multi-modal open intent recognition task is to utilize text, video, audio and other multi-modal information to accurately classify in-distribution (ID) intents while effectively detecting out-of-distribution (OOD) intents. This capability is of great significance for building robust and safe human-computer interaction systems.

[0003] In the prior art, cross-modal alignment is usually the core step of multi-modal representation learning. Common methods are mostly based on contrast learning, which pulls closer samples with the same label or time synchronization relationship in different modalities, and pushes away other samples to reduce the modal difference. However, this rigid alignment at the instance level is often too strict, easily compressing the unique semantic information of each modality, resulting in decreased representation ability and insufficient generalization. Different modalities actually encode complementary information of the same instance, but they are not exactly the same, so forced alignment may cause information loss and semantic ambiguity.

[0004] To relax the constraints, some methods attempt to introduce clustering alignment in the multi-modal scenario. However, existing clustering methods mostly rely on preset cluster numbers (such as k-means), and the clustering results are relatively rough, making it difficult to accurately depict the real data structure. This deficiency will directly affect the effect of cross-modal alignment, limiting the performance of the model in complex open environments. Therefore, how to achieve flexible and efficient cross-modal alignment while maintaining modality characteristics is still a technical problem to be solved. In addition, the OOD intent detection capability of the existing technology in the open world scenario also has deficiencies. On the one hand, the model lacks a modeling mechanism for the real open space distribution; on the other hand, the decision boundary mostly relies on a fixed threshold, making it difficult to adapt to the spatial structure of different classes, resulting in an unsatisfactory unknown intent detection rate.

[0005] Therefore, there is an urgent need for a new multi-modal open intent recognition method that can achieve flexible alignment while maintaining modality characteristics, and combine adaptive structure modeling and boundary perception mechanisms to improve recognition and detection capabilities in the open world. SUMMARY

[0006] The purpose of the present application is to overcome the problems of poor cross-modal alignment generalization, loss of modal characteristics and insufficient OOD intent detection in the prior art, and provide a multi-modal open intent recognition method and system based on granular ball representation, which realizes the joint recognition of known classes and unknown classes in a multi-modal scene by introducing multi-granularity anchor points and structure-aware granular ball representation, and significantly improves the accuracy and robustness of intent recognition.

[0007] The purpose of the present application is achieved by the following technical solutions: In a first aspect, a multi-modal open intent recognition method based on granular ball representation is provided, comprising the following steps: S1. Feature extraction and modal fusion are performed on the multi-modal input data to obtain high-level semantic representations of text, video, audio and fused modalities; S2. Granular ball structure modeling: the feature representations of each modality and the fused modality are modeled by an adaptive granular ball clustering method to generate a multi-granularity granular ball set; S3. Structure-aware granular ball level contrast learning: using granular ball centroids as multi-granularity anchor points, aligning granular balls with the same label in different modalities; introducing a weighting mechanism based on purity and sample size in the fused modality; S4. Pseudo-distributed out-of-distribution sample generation and representation learning: generating boundary-constrained pseudo-distributed out-of-distribution samples in the fused modality space; S5. Inference stage: based on the trained fused modality granular ball, an adaptive decision boundary is constructed, and known class classification and unknown class detection are performed.

[0008] In some embodiments, the feature extraction and modal fusion of the multi-modal input data comprises: The text modality uses BERT to extract features, the video modality uses Swin Transformer to extract features, and the audio modality uses WavLM to extract features; The feature representations of each modality are combined into a unified representation by a weighted fusion strategy to obtain the feature representation of the fused modality.

[0009] In some embodiments, the adaptive granular ball clustering method comprises: The sample set is recursively split and aggregated to generate a plurality of granular balls, each granular ball being characterized by a centroid, a radius, a purity, a size and a label attribute; The splitting condition is that the purity of the granular ball is lower than a threshold and the number of samples is higher than a threshold, and the aggregation condition is that the purity of the granular ball is higher than a threshold or the number of samples is lower than a threshold.

[0010] In some embodiments, the aligning of granular balls with the same label in different modalities comprises: The following total loss function is used: wherein, denotes the total loss of multi-modal alignment, denotes the alignment contrast loss between the text and the video modal, denotes the alignment contrast loss between the text and the speech modal.

[0011] In some embodiments, the in-class contrast objective is designed in the fused modal in step S3.

[0012] In some embodiments, the generating of the boundary-constrained pseudo out-of-distribution samples in the fused modal space comprises: Generating candidate pseudo out-of-distribution samples by extrapolating different class granule centroids; and using a radius filtering mechanism to remove samples falling within the range of known class granules.

[0013] In some embodiments, the generating of the boundary-constrained pseudo out-of-distribution samples in the fused modal space further comprises: Introducing an OOD separation loss to improve the distance between the pseudo out-of-distribution samples and the in-distribution granule centroids.

[0014] In some embodiments, the final overall optimization objective function is wherein, denotes the granule alignment loss across modalities; denotes the structure-aware contrast loss in the fused modal; denotes the ID-OOD separation loss of the pseudo OOD samples; denotes the standard classification loss; denotes the weight hyperparameters of the loss terms.

[0015] In some embodiments, step S5 specifically comprises: Using the granule clustering results of the fused modal as the initial decision boundary of the in-distribution classes, and introducing a scaling factor to expand the granule radius to form the final class decision region.

[0016] In a second aspect, a multi-modal open intent recognition system based on granule representation is provided, comprising: a feature extraction and fusion module for performing feature extraction and modal fusion on multi-modal input data to obtain high-level semantic representations of text, video, audio and fused modal; a granule modeling module for performing granule structure modeling, modeling the feature representations of each modal and the fused modal through an adaptive granule clustering method to generate a set of multi-granularity granules; The contrast learning module is used for structure-aware grain ball level contrast learning, takes grain ball centroids as multi-granularity anchor points, and aligns grain balls with the same label in different modalities; and a weighting mechanism based on purity and sample size is introduced in the fusion modalities; The pseudo-distribution out-of-sample generation module is used for pseudo-distribution out-of-sample generation and representation learning, and boundary-constrained pseudo-distribution out-of-sample is generated in the fusion modality space; The inference module is used for performing an inference stage, constructing an adaptive decision boundary based on the fusion modality grain balls obtained through training, and performing known class classification and unknown class detection.

[0017] It should be further explained that the technical features of the above-mentioned embodiments can be combined or replaced with each other to form new technical solutions without conflict.

[0018] Compared with the prior art, the present application realizes the joint recognition of known classes and unknown classes in a multi-modal scene by introducing multi-granularity anchor points and structure-aware grain ball representation methods, and significantly improves the accuracy and robustness of intent recognition. The specific beneficial effects are as follows: 1. The present application uses an adaptive grain ball clustering method to model the feature space of multi-modal samples. Through recursive splitting and aggregation operations, a series of grain balls are generated without pre-setting the number of clusters, and each grain ball is characterized by centroid, radius, purity, size and other attributes. The modeling process can generate a multi-granularity semantic structure, effectively approximating the real intra-class distribution characteristics. Compared with traditional static clustering methods, the grain ball structure of the present application has adaptability, interpretability and robustness, and can reflect both macro-class distribution and local fine-grained structure, providing a structured basic unit for subsequent cross-modal alignment and open space modeling.

[0019] 2. On the basis of the grain ball structure modeling, the present application proposes a structure-aware contrast learning method. This method takes the grain ball centroid as a multi-granularity anchor point, and only pairs grain balls with the same label during cross-modal alignment, avoiding the loss of modal characteristics caused by instance-level forced alignment. By introducing a weighting mechanism based on purity and sample size, high-purity and large-sample-size grain balls are given greater weight in the optimization process, thereby reducing the interference caused by low-quality grain balls. In the fusion modalities, the present application further designs an intra-class contrast target to make same-class grain balls more compact and different-class grain balls sufficiently distinguishable. This mechanism effectively improves the consistency between modalities and the discriminability of the fusion space, while preserving the unique features of each modality, balancing multi-modal semantic sharing and structural differences, and significantly improving the robustness in open scenarios.

[0020] 3. To enhance the recognition ability of open space, the application proposes a boundary-constrained pseudo OOD sample generation strategy. In the fusion modal space, first select the particle ball center of different categories, generate candidate pseudo OOD samples by extrapolation synthesis method; Then use the radius filtering mechanism to remove invalid samples within the radius range of known class particle ball, and only keep high-quality pseudo OOD samples outside the boundary. Further, the application introduces the OOD separation loss function, which forces to improve the interval between pseudo OOD samples and ID particle ball center, so as to realize the effective differentiation of ID and OOD. This strategy does not need to rely on additional unknown class data, can simulate the distribution characteristics of real open space, significantly improve the unknown intent detection ability of the model in the absence of prior information, and has practicability and expansibility.

[0021] 4. In the reasoning phase, the application constructs an adaptive decision boundary based on the fusion modal particle ball obtained by training. Each category is represented by multiple particle balls, and the boundary of the category is defined by the center and the radius. To improve the detection flexibility, the application introduces a radius scaling factor to uniformly expand or contract the radius of each particle ball. Specifically, when the test sample falls within the scaled boundary of a certain class particle ball, it is determined to be of that class; if it falls outside the boundary of all class particle balls, it is determined to be of unknown class. By adjusting the scaling factor, the classification accuracy and unknown class detection rate can be balanced according to the actual application. Compared with the traditional fixed threshold method, this scheme can dynamically adapt to the data distribution of different scenarios, realize the integrated processing of known class classification and unknown class detection, and significantly improve the robustness and reliability of the system in an open environment. BRIEF DESCRIPTION OF DRAWINGS

[0022] Figure 1 The flowchart of the application is a multi-modal open intent recognition method based on particle ball representation. DETAILED DESCRIPTION

[0023] The technical solutions of the application will be described clearly and completely below in conjunction with the drawings. Obviously, the described embodiments are part of the embodiments of the application, not all embodiments. The components of the embodiments of the application described and shown in the drawings can be arranged and designed in various configurations. Based on the embodiments in the application, all other embodiments obtained by those skilled in the art without creative labor are within the scope of protection of the application.

[0024] It should be noted that the defects of the above prior art schemes are the results obtained by the inventors after practice and careful study, therefore, the discovery process of the above problems and the solutions proposed by the embodiments of the application to the above problems should be the contribution of the inventors to the application in the process of invention and creation, and should not be understood as technical content known to those skilled in the art.

[0025] The present application firstly extracts the representations of single modalities and fused modalities, and models the multi-granularity structure information through an adaptive granular ball clustering method. Then, structure-aware contrastive learning is performed to enhance modality consistency through inter-modal label-consistent granular ball alignment, while combining attribute-guided weighting mechanisms to optimize the representations of fused modalities. Then, a boundary-constrained pseudo OOD sample generation module is designed to synthesize pseudo unknown samples outside the known class decision boundary, and an OOD separation loss is introduced to enhance the open set discrimination ability. In the inference stage, the learned granular ball radius scaling is used to construct the decision boundary, realizing the joint task of known class classification and unknown class detection.

[0026] In an exemplary embodiment, referring to Figure 1 , a multi-modal open intent recognition method based on granular ball representation is provided, comprising the following steps: S1. Feature extraction and modality fusion are performed on the multi-modal input data to obtain high-level semantic representations of text, video, audio and fused modalities; S2. Granular ball structure modeling: the feature representations of each modality and fused modalities are modeled through an adaptive granular ball clustering method to generate a multi-granularity granular ball set; S3. Structure-aware granular ball level contrastive learning: taking the granular ball centroid as a multi-granularity anchor point, aligning the granular balls with the same label in different modalities; introducing a weighting mechanism based on purity and sample size in the fused modality; S4. Pseudo out-of-distribution sample generation and representation learning: generating boundary-constrained pseudo out-of-distribution samples in the fused modality space; S5. Inference stage: based on the fused modality granular balls learned during training, an adaptive decision boundary is constructed, and known class classification and unknown class detection are performed.

[0027] Before describing the specific implementation process, the problem is formalized, and a multi-modal intent recognition dataset is given, each intent sample contains three modalities, where represents text, represents video, represents audio. The in-distribution (known) (In-Distribution, ID) intent sample dataset is defined as: where the sample label indicates that the sample belongs to one of the known intent categories.

[0028] The out-of-distribution (unknown) (Out-of-Distribution, OOD) intent sample dataset is defined as: where the samples come from unseen classes, each sample is assigned a uniform label , denoted as unknown class. The training and validation stage only use , while are only available in the testing stage. The goal is to build a model that can accurately classify ID intent samples and identify OOD intent samples in testing.

[0029] Exemplarily, the granular structure modeling specifically includes: 1. Feature extraction and modal fusion We first extract low-level features from text, video and audio modalities and perform modal fusion. Among them, the text modality uses BERT to extract text embedding, obtaining: The video modality uses Swin Transformer to extract visual features: The audio modality extracts audio embedding through WavLM: where denotes the sequence length of each modality, denotes the feature dimension of each modality.

[0030] Then, the extracted features of the three modalities are input into the Transformer encoder to extract high-level semantic representation: For the text modality, the last layer [CLS] representation is taken as the sentence-level representation: where , denotes the sentence-level feature vector of the text modality, [CLS] denotes the output state of the corresponding classification mark ([CLS] token) in the last layer of the Transformer, as the aggregated representation of the entire input sequence.

[0031] For video and audio modalities, classical Transformer, average pooling and linear transformation are used in turn: where, , is a linear mapping matrix, is a uniform fusion feature dimension.

[0032] Subsequently, a weighted modal fusion strategy is adopted to fuse the representations of the three modalities into a whole representation , the calculation method is as follows: wherein represents the weight of the modal , which is obtained by adaptive learning of the lightweight neural network.

[0033] 2. Adaptive granular ball clustering The granular ball clustering can adaptively capture fine-grained structures with different granularities, form ball representations of different sizes, do not need to specify the number of clusters in advance, and can better approximate the real distribution. Since it performs well in maintaining the structure properties of sample distribution, it is applied to the feature representation of each modal and the fusion modal to guide the subsequent representation learning.

[0034] 2.1. Definition and properties of granular ball Granular ball is an adaptive clustering unit that can depict the real distribution of data. Let the sample set be , wherein is the feature vector of the sample, is the class label of the sample. By clustering the sample set, a series of granular balls can be obtained, represented as a set , wherein each granular ball is composed of samples and has the following properties: Granular ball size : the number of samples contained in the granular ball; granular ball centroid : the mean of all sample features in the granular ball; granular ball radius : the average Euclidean distance of all samples in the granular ball to the centroid; granular ball label : the label with the highest proportion in the granular ball; granular ball purity : the proportion of samples in the granular ball that belong to the granular ball label .

[0035] 2.2. Granular ball clustering process Let the sample representation of a given modal be: First, all samples are clustered into one coarse-grained granular ball, and then an adaptive clustering process is used to recursively divide it into finer granular balls. When the granular ball purity and the number of samples , the division operation is triggered. In the division process, if a granular ball contains samples of multiple classes, its label set is , then it will be divided into new granular balls. For each class , a sample is randomly selected from the samples belonging to the category as a pseudo-center. Then, all the samples in the particle ball are re-assigned to the new particle ball corresponding to the nearest pseudo-center according to the Euclidean distance with the pseudo-center.

[0036] When each particle ball satisfies: 1) its particle ball purity , or 2) the number of samples , the division process is terminated. Finally, each modality (and its fusion modality) will get a particle ball set: where each particle ball has five attributes: represent the label set, purity, size, center point, and radius, respectively.

[0037] Exemplarily, the structure-aware particle-level contrastive learning specifically includes: To avoid information loss caused by too strict instance (sample) level alignment and better preserve modality characteristics, the present application performs structure-aware contrastive learning at the particle level. This strategy includes two parts: 1) multi-modal particle contrastive learning: aligning particle balls with the same label in different modalities; 2) fusion modality contrastive learning: enhancing the separability between classes. In order to reduce the interference of low-quality samples on contrastive learning, a weighting mechanism is introduced, which gives priority to high-purity, large-sample positive and negative sample pairs.

[0038] 1. Cross-modal alignment The present application takes the particle center point in the text modality as an anchor point to align the video modality and the audio modality to the text modality. We first calculate the loss of aligning the video modality to the text modality: Let the particle ball set of the text modality be: Each particle ball contains: center point , label , purity , size .

[0039] Let the particle ball set of the video modality be: Each particle ball contains: center point , label , purity , size .

[0040] Given a text modality center point , compare it with the particle ball in the video modality to form a positive sample set and a negative sample set, respectively, defined as: where, denotes the positive sample set, denotes the negative sample set, denotes the label of the j-th granule in the video modality, denotes the label of the i-th granule in the text modality, for each pair of granules , its weighted coefficient is defined as: where, denotes the purity of the i-th granule in the text modality, denotes the purity of the j-th granule in the video modality, denotes the size of the i-th granule in the text modality, denotes the size of the j-th granule in the video modality.

[0041] Subsequently, the alignment contrast loss between the text and video modalities is defined as follows: where, denotes the cosine similarity, is the temperature coefficient, denotes the number of granules in the granule set of the text modality, i.e., the number of center points of the text modality denotes the center points of the video modality granules. Similarly, the text-audio alignment loss can be defined using the same method . Finally, the total loss of multi-modal alignment is: 2. Fusion modality contrast learning To enhance the inter-class compactness and intra-class separability of the fusion modality representation, a structure-aware contrastive objective is further defined on the fusion modality granule set. Let the granule set of the fusion modality be: Each granule center point is , and the label is . For each granule , its positive and negative sample sets are defined as: where, denotes the positive sample set of the fusion modality, denotes the negative sample set of the fusion modality, denotes the label of the j-th granule in the fusion modality, denotes the label of the i-th granule in the fusion modality, The weighting formula is consistent with that in cross-modal alignment: The contrast loss of the fusion modal is as follows: wherein, represents the number of granules in the fusion modal, represents the center of the i-th granule in the fusion modal, represents the center of the j-th granule in the fusion modal.

[0042] Exemplarily, the pseudo-distributed out-of-distribution sample generation and characterization learning specifically includes: Generating pseudo OOD (pseudo unknown class) samples and incorporating them into training is an effective method to improve OOD detection performance. The present application proposes a pseudo OOD sample generation strategy based on boundary constraints, which is applied to the granule characterization of the fusion modal, and generates high-quality pseudo OOD representations using the properties of granules. The specific process is as follows: 1) Initial sample generation: first, generate initial pseudo OOD samples based on the centroids of the fusion modal granules. 2) Boundary filtering: filter out pseudo samples falling within the ID (known class) region to ensure that they are distributed outside the ID decision boundary.

[0043] Specifically, for each centroid (from the fusion modal, labeled ), find its nearest centroid from a different class (i.e. ). Extrapolate from to to generate pseudo OOD samples: wherein, represents the pseudo OOD sample generated by extrapolating the center of the fusion modal , is a random number sampled from a uniform distribution, introducing diversity; each pair of centroids can generate pseudo OOD samples, and in the experiment. Then, filter out samples falling within the ID decision boundary using the following radius condition to retain effective pseudo samples: wherein, represents the center of the k-th granule in the fusion modal, represents the radius of the k-th granule in the fusion modal, i.e. the minimum distance between the pseudo sample and all granule centers must be greater than the corresponding granule radius. Further, an OOD separation loss to explicitly push the ID sample centroids away from the reserved pseudo OOD samples, thus enhancing the separation degree between ID and OOD representations. The definition is as follows: wherein, represents the number of reserved pseudo OOD samples; represents the number of granules in the fusion modal; represents the th pseudo OOD representation; represents the th centroid of the fusion modal granule. At the same time, the present application also imposes a standard cross-entropy supervision loss on the representation of the fusion modal, and the definition is as follows: wherein, represents all ID and pseudo OOD samples; represents the one-hot label of the th sample corresponding to the class ; represents the total class set containing ID classes and the th class (for pseudo OOD); represents the cosine-based classification head, represents the th sample in

[0044] The final overall optimization objective consists of the following four losses: wherein, represents the granule alignment loss across modalities; represents the structure-aware contrastive loss of the fusion modal; represents the ID-OOD separation loss of the pseudo OOD samples; represents the standard classification loss; represents the weight hyperparameter of the loss term.

[0045] The loss function described above is used for back propagation, gradient calculation and updating of the trainable parameters of the model, so as to gradually improve the performance of the model in open intent classification.

[0046] The model updating specifically includes: 1) validation set evaluation After completing each round of training, the evaluation index score of the model on the validation set is calculated to evaluate the current performance of the model.

[0047] 2) update the optimal validation score and the optimal model If the current evaluation score is higher than the historical optimal score, the score and the corresponding model parameters are set as the new optimal result.

[0048] If the current evaluation score is lower than or equal to the historical optimal score, the optimal model is not updated.

[0049] 3) Continue training The subsequent training iterations are based on the parameters of the current optimal model, so that the adaptability of the model in the complex noise environment is continuously improved.

[0050] The training stop and boundary preservation specifically includes: If the verification score does not improve for 10 consecutive times, the early stopping strategy is triggered, the representation learning is stopped, and the optimal model parameters are saved. At the same time, the centroids of each known class represented by high-quality granules and their radii are recorded as the final class boundaries.

[0051] Exemplarily, the inference stage uses an adaptive decision boundary, specifically including: The granule clustering results of the fused modal are used as the initial decision boundary of the ID class, and a radius scaling factor Δ is introduced to expand the granule radius to form the final class determination region. For each ID class , its decision boundary is defined by multiple granules with labels , when a test sample is input: 1) Calculate the Euclidean distance between it and all granule centroids: 2) If the distance between the sample and all granule centroids is greater than the corresponding radius: Then the sample is judged as OOD, where represents the radius of the granule of the fused modal, represents any granule j of the fused modal.

[0052] 3) Otherwise, it is assigned to the class of the granule with the closest distance.

[0053] In another exemplary embodiment, a multi-modal open intent recognition system based on granule representation is provided, comprising: a feature extraction and fusion module for extracting and fusing features of multi-modal input data to obtain high-level semantic representations of text, video, audio and fused modal; a granule modeling module for granule structure modeling, which models the structure of each modal and fused modal feature representation through an adaptive granule clustering method to generate a set of multi-granularity granules; ​The contrast learning module is used for structure-aware granule-level contrast learning, taking the granule centroid as a multi-granularity anchor point, and aligning granules with the same label in different modalities; and a weighting mechanism based on purity and sample size is introduced in the fusion modalities; The pseudo-distribution out-of-sample generation module is used for pseudo-distribution out-of-sample generation and representation learning, and boundary-constrained pseudo-distribution out-of-sample is generated in the fusion modality space. The inference module is used for performing an inference stage, constructing an adaptive decision boundary based on the fusion modality granules obtained through training, and performing known class classification and unknown class detection.

[0054] The above detailed description is a detailed description of the present application, and cannot be considered as limiting the specific embodiments of the present application to these descriptions. For ordinary skilled persons in the technical field to which the present application belongs, a number of simple deductions and substitutions can be made without departing from the concept of the present application, and all of them should be considered as falling within the protection scope of the present application.

Claims

1. A multimodal open intent recognition method based on particle-sphere representation, characterized in that, Includes the following steps: S1. Perform feature extraction and modality fusion on multimodal input data to obtain high-level semantic representations of text, video, audio, and fused modalities; S2. Particle structure modeling: The feature representations of each mode and the fusion mode are structurally modeled using an adaptive particle clustering method to generate a multi-granularity particle set; S3. Structure-aware particle-level contrastive learning: Using the centroid of the particle as a multi-granularity anchor point, aligning particles with the same label in different modalities; introducing a weighting mechanism based on purity and sample size in the fusion modality; S4. Pseudo-out-of-distribution sample generation and representation learning: Generate pseudo-out-of-distribution samples with boundary constraints in the fusion modality space; S5. Inference phase: Based on the fusion modal particles obtained from training, construct an adaptive decision boundary and perform known class classification and unknown class detection.

2. The multimodal open intent recognition method based on particle-sphere representation according to claim 1, characterized in that, The feature extraction and modality fusion of multimodal input data includes: Text modality uses BERT to extract features, video modality uses Swing Transformer to extract features, and audio modality uses WavLM to extract features. By using a weighted fusion strategy, the representations of each modality are merged into a unified representation, resulting in the feature representation of the fused modality.

3. The multimodal open intent recognition method based on particle-sphere representation according to claim 1, characterized in that, The adaptive particle clustering method includes: The sample set is recursively split and aggregated to generate multiple spheres, each characterized by its centroid, radius, purity, size, and label attributes. The splitting condition is that the particle purity is below the threshold and the number of samples is above the threshold, while the aggregation condition is that the particle purity is above the threshold or the number of samples is below the threshold.

4. The multimodal open intent recognition method based on particle-sphere representation according to claim 1, characterized in that, The alignment of particles with the same label in different modalities includes: Use the following total loss function: in, This represents the total loss of multimodal alignment. This represents the alignment contrast loss between the text and video modalities. This represents the alignment contrast loss between the text and speech modalities.

5. The multimodal open intent recognition method based on particle-sphere representation according to claim 1, characterized in that, In step S3, design intra-class comparison targets in the fusion modality.

6. The multimodal open intent recognition method based on particle-sphere representation according to claim 1, characterized in that, The process of generating pseudo-out-of-distribution samples with boundary constraints in the fusion modality space includes: Candidate pseudo-distribution samples are generated by extrapolating the centroids of different types of granules; and a radius filtering mechanism is used to remove samples that fall within the known granule class range.

7. The multimodal open intent recognition method based on particle-sphere representation according to claim 6, characterized in that, The process of generating pseudo-out-of-distribution samples with boundary constraints in the fused modality space further includes: Introducing OOD separation loss increases the distance between samples outside the pseudo-distribution and the centroids of particles within the distribution.

8. The multimodal open intent recognition method based on particle-sphere representation according to claim 1, characterized in that, The final overall optimization objective function is: in, This represents the cross-modal particle-sphere alignment loss; This represents the structure-aware contrast loss of the fusion modality; This represents the ID-OOD separation loss for pseudo-OOD samples; Indicates standard classification loss; This represents the weight hyperparameter of the loss term.

9. The multimodal open intent recognition method based on particle-sphere representation according to claim 1, characterized in that, Step S5 specifically includes: The spheroid clustering results of the fusion modality are used as the initial decision boundary for the in-distribution categories, and a scaling factor is introduced to expand the spheroid radius to form the final category determination region.

10. A multimodal open intent recognition system based on particle-sphere representation, characterized in that, include: The feature extraction and fusion module is used to extract features and fuse modalities from multimodal input data to obtain high-level semantic representations of text, video, audio, and fused modalities. The particle modeling module is used to perform particle structure modeling. It uses an adaptive particle clustering method to perform structural modeling on the feature representations of each mode and the fusion mode, generating a multi-granularity particle set. The contrastive learning module is used for structure-aware particle-level contrastive learning, using the particle centroid as a multi-granularity anchor point to align particles with the same label in different modalities; a weighting mechanism based on purity and sample size is introduced in the fusion modality. The pseudo-out-of-distribution sample generation module is used to generate pseudo-out-of-distribution samples and learn their representations, generating boundary-constrained pseudo-out-of-distribution samples in the fusion modality space. The inference module is used to perform the inference phase. It constructs an adaptive decision boundary based on the fusion modal particles obtained from training and performs known class classification and unknown class detection.

Images