An intelligent interactive intention recognition method and system based on multi-modal data

Abstract

The application discloses a multi-modal data-based intelligent interaction intention recognition method and system, constructs a multi-smart ring body recognition architecture based on a user, and the multi-smart ring body recognition architecture comprises a plurality of concentric rings with different diameters; multi-modal data of the user is collected, the multi-modal data is preprocessed, and the multi-modal data is respectively mapped to the circumferential surfaces of different concentric rings according to different types of the multi-modal data; static features and dynamic features of the multi-modal data located on the concentric rings are respectively calculated, the static features and the dynamic features are fused to obtain fused features; the fused features are input into an intention classification model to recognize the user intention; the multi-modal data can be mapped to different concentric rings to solve the space-time dislocation problem between cross modalities, and the concentric rings can rotate to dynamically adjust the relative positions of the modal data, so that the static features and the dynamic features can be accurately calculated, and the fused features can be used for accurate recognition of the user intention.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of information processing, in particular to a multi-modal data-based intelligent interaction intention recognition method and system. BACKGROUND

[0002] Human-computer interaction requires the user to issue accurate instructions, and the server identifies the instructions to obtain the user's intention, and then issues control instructions to the controlled components or displays information to the user. The traditional intention recognition method only relies on single modal data, and it is difficult to fully capture the complex intention clues in user interaction. With the development of artificial intelligence technology, intelligent interaction systems gradually evolve from single modal to multi-modal fusion. Users may describe, take pictures and explain the situation through text, pictures and voice at the same time. However, the existing method lacks multi-modal coordination capability, resulting in insufficient recognition accuracy. In addition, some multi-modal fusion methods do not fully consider the temporal correlation and dynamic evolution rules between modalities, such as the prosodic changes of speech signals, the temporal distribution of image key frames, and the complementarity of static features such as word vectors and mel frequency spectrum. SUMMARY

[0003] In view of this, the present application provides a multi-modal data-based intelligent interaction intention recognition method and system. Through the established multi-intelligent ring body recognition architecture, the static and dynamic features of multi-modal data can be accurately obtained, so as to accurately recognize the user's intention.

[0004] The technical scheme of the present application is as follows:

[0005] A multi-modal data-based intelligent interaction intention recognition method, comprising the following steps:

[0006] Step S1, constructing a multi-intelligent ring body recognition architecture based on the user, wherein the multi-intelligent ring body recognition architecture comprises a plurality of concentric rings with different diameters;

[0007] Step S2, collecting multi-modal data of the user, pre-processing the multi-modal data, and mapping the multi-modal data of different types to the circumferential surface of different concentric rings respectively;

[0008] Step S3, calculating the static and dynamic features of the multi-modal data located on the concentric rings respectively, and fusing the static and dynamic features to obtain fused features;

[0009] Step S4, inputting the fused features into an intention classification model to recognize the user's intention.

[0010] Preferably, the specific steps of step S1 comprise:

[0011] Step S11, an interaction scene of a user is acquired, the interaction scene is identified according to historical interaction data, and an estimated multi-modal data type is obtained;

[0012] Step S12, the number of concentric ring bodies is determined according to the number of the estimated multi-modal data types, and a priority of the estimated multi-modal data types is determined based on a focus direction of the interaction scene;

[0013] Step S13, a diameter of each concentric ring body is determined based on the priority, the concentric ring bodies are concentrically arranged in each other, and a multi-intelligent ring body recognition architecture is obtained.

[0014] Preferably, the specific steps of the step S1 further include:

[0015] Step S14, a calibration instruction is sent to the user, and calibration multi-modal data when the user executes the calibration instruction is collected;

[0016] Step S15, a rotation angle of the concentric ring body with the vertical diameter as the axis is adjusted based on the calibration multi-modal data.

[0017] Preferably, the specific steps of collecting the multi-modal data of the user in the step S2 are as follows:

[0018] Step S21, multi-modal data when the user interacts is collected through an interaction screen, a wearable terminal, a camera and a sensor, and the multi-modal data includes text, voice and images;

[0019] Step S22, the multi-modal data is denoised, cleaned and normalized;

[0020] Step S23, the multi-modal data of the text, the voice and the images after the preprocessing is respectively subjected to feature extraction, and a word vector, a sound frame containing a mel spectrum sequence and an image key frame are obtained.

[0021] Preferably, the specific steps of respectively mapping the different multi-modal data types to the circumferential surfaces of different concentric ring bodies in the step S2 are as follows:

[0022] Step S24, a vertical radius of the concentric ring body and an intersection point of the concentric ring body are taken as an origin;

[0023] Step S25, the word vector, the sound frame containing the mel spectrum sequence and the image key frame are sequentially arranged in a clockwise direction on the circumferential surface of the concentric ring body according to a time stamp order.

[0024] Preferably, the specific steps of calculating the static features in the step S3 are as follows:

[0025] Step S31, cosine similarity between two concentric ring bodies at the same time stamp is calculated, and a cosine similarity vector is obtained.

[0026] Step S32, self-similarity calculation is performed on the concentric ring corresponding to the modal data with the highest priority to obtain a self-similarity vector, and the cosine similarity vector and the self-similarity vector are spliced into a static feature.

[0027] Preferably, the specific steps of calculating the dynamic feature in step S3 are as follows:

[0028] Step S33, the collected multi-modal data of the user is compared with the calibration multi-modal data, an adjustment angle is obtained based on the difference, and the concentric ring is rotated according to the adjustment angle.

[0029] Step S34, the Pearson correlation coefficient between each two concentric rings at the same timestamp is calculated, and a first Pearson correlation vector is obtained.

[0030] Step S35, the principal component sequence of the multi-modal data on the concentric ring is extracted, the Pearson correlation coefficient of the principal component sequence of each two concentric rings at the same timestamp is calculated, and a second Pearson correlation vector is obtained.

[0031] Step S36, the first Pearson correlation vector and the second Pearson correlation vector are spliced into a dynamic feature.

[0032] Preferably, the specific steps of fusing the static feature and the dynamic feature in step S3 to obtain a fusion feature are as follows:

[0033] Step S37, after the static feature and the dynamic feature are subjected to abnormal value processing, timestamp alignment and feature normalization, they are spliced into a fusion feature point by point in time.

[0034] Preferably, the specific steps of step S4 are as follows:

[0035] Step S41, historical multi-modal data is collected, and after artificial labeling, the historical multi-modal data is divided into a training set and a test set.

[0036] Step S42, an intent classification model is constructed by using a long short-term memory network, the model is trained by using the training set, and the model is tested by using the test set.

[0037] Step S43, the fusion feature is input into the trained intent classification model, an intent category probability distribution is generated, and the category with the highest probability is output as the user intent.

[0038] A system for intelligent interactive intent recognition based on multi-modal data, characterized in that it comprises:

[0039] A construction unit is configured to construct a multi-smart ring recognition architecture based on a user, the multi-smart ring recognition architecture comprising a plurality of concentric rings with different diameters.

[0040] The collection mapping unit is used for collecting multi-modal data of a user, pre-processing the multi-modal data, and mapping the multi-modal data to the circumferential surface of different concentric ring bodies according to different types of the multi-modal data;

[0041] The feature fusion unit is used for calculating static features and dynamic features of the multi-modal data located on the concentric ring bodies respectively, fusing the static features and the dynamic features, and obtaining fused features;

[0042] The intention recognition unit is used for inputting the fused features into an intention classification model to recognize specific intention of the user;

[0043] The collection mapping unit, the construction unit, the feature fusion unit and the intention recognition unit are sequentially connected in data.

[0044] Compared with the prior art, the beneficial effects of the present application are:

[0045] ① In the constructed multi-intelligent ring body recognition architecture, the same number of concentric ring bodies as the types of multi-modal data are contained, different modal data can be mapped to different concentric ring bodies, and the problem of space-time dislocation between cross-modal is solved;

[0046] ② The concentric ring bodies can be dynamically rotated according to the collected multi-modal data, the relative positions between the modal data are adjusted, so that more accurate results can be obtained when calculating the static features and the dynamic features, finally the static features and the dynamic features can be fused, and more accurate user intention can be recognized through the intention classification model, and the user experience is improved. BRIEF DESCRIPTION OF DRAWINGS

[0047] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings needed in the embodiment description will be briefly introduced. Obviously, the drawings in the following description are only preferred embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor.

[0048] Figure 1 A flowchart of a multi-modal data-based intelligent interactive intention recognition method of the present application;

[0049] Figure 2 A flowchart of step S1 of a multi-modal data-based intelligent interactive intention recognition method of the present application;

[0050] Figure 3 A flowchart of step S2 of a multi-modal data-based intelligent interactive intention recognition method of the present application;

[0051] Figure 4A flow chart of step S3 of a multi-modal data based intelligent interactive intention recognition method of the present application;

[0052] Figure 5 A flow chart of step S4 of a multi-modal data based intelligent interactive intention recognition method of the present application;

[0053] Figure 6 A principle diagram of a multi-modal data based intelligent interactive intention recognition system of the present application;

[0054] In the figure, 1, a construction unit; 2, an acquisition mapping unit; 3, a feature fusion unit; 4, an intention recognition unit. DETAILED DESCRIPTION

[0055] In order to better understand the technical content of the present application, a specific embodiment is provided below, and the present application is further described in combination with the accompanying drawings.

[0056] Referring to Figures 1 to 5 The multi-modal data based intelligent interactive intention recognition method provided by the present application comprises the following steps:

[0057] Step S1, constructing a multi-smart ring body recognition architecture based on a user, the multi-smart ring body recognition architecture comprising a plurality of concentric rings with different diameters;

[0058] Step S2, acquiring multi-modal data of the user, pre-processing the multi-modal data, and respectively mapping the multi-modal data to the circumferential surfaces of different concentric rings according to the different types of multi-modal data;

[0059] Step S3, respectively calculating the static features and dynamic features of the multi-modal data located on the concentric rings, fusing the static features and dynamic features, and obtaining fused features;

[0060] Step S4, inputting the fused features into an intention classification model, and recognizing the user intention.

[0061] Compared with the traditional intention recognition method, the intelligent interaction intention recognition method based on multi-modal data of the application adds different modal data, including text, voice and video, etc. Different modal data includes different information and different content for analysis. Through the combination of different modal data, the user's intention can be more accurately recognized. In order to enhance the interaction between different modal data, the application constructs a multi-intelligent ring body recognition architecture. The multi-intelligent ring body recognition architecture is composed of a plurality of concentric rings, and the diameters of each concentric ring are different. One concentric ring corresponds to one modal data. In the process of interacting with the user, the multi-modal data of the user is collected, and then different types of modal data are mapped to the corresponding concentric rings. The concentric rings are used as the carrier of modal data, and the data processing is converted to the three-dimensional level, which can solve the problem of space-time dislocation between cross-modal. The diameter difference of different concentric rings can dynamically adjust the relative position between each modal data when rotating, so as to accurately obtain the mutual relationship between different modal data, and provide reliable data support for user intention evaluation.

[0062] After obtaining the multi-intelligent ring body recognition architecture containing multi-modal data, the static features and dynamic features of the multi-modal data located on the concentric rings are calculated respectively. The static features are quantified by the instantaneous matching degree between modalities, which provides direct spatial alignment basis for intention recognition. The dynamic features can obtain new features based on the change of user behavior, and combine with the static features. The dynamic features not only retain the modal matching details at a single time point, but also contain the interaction logic across time steps, providing more rich decision information for the intention classification model. The fused features can be input into the pre-constructed intention classification model to identify the accurate user intention, so that the artificial intelligence can make response instructions and operations based on the user intention, and improve the user experience. The application can be applied in various fields, such as car navigation, fault detection, medical diagnosis and smart home, etc.

[0063] Preferably, the specific steps of the step S1 include:

[0064] Step S11, obtaining the interaction scene of the user, identifying the interaction scene according to the historical interaction data, and obtaining the estimated multi-modal data types;

[0065] Step S12, determining the number of concentric rings according to the number of estimated multi-modal data types, and determining the priority of the estimated multi-modal data types based on the emphasis direction of the interaction scene;

[0066] Step S13, determining the diameter of each concentric ring based on the priority, and concentrically nesting the concentric rings to obtain the multi-intelligent ring body recognition architecture.

[0067] The multi-modal data is various, for example, including visual modal, auditory modal, text modal, touch and somatosensory modal and environment modal, etc. Different modal data needs to be carried by corresponding concentric rings. Before constructing a multi-intelligent ring body recognition model, the number of concentric rings needs to be determined. After obtaining the interactive scene of the user, the commonly used multi-modal data types can be determined from the historical interactive data of the interactive scene. For example, in the car navigation interactive scene, the multi-modal data can include text modal, voice modal and visual modal. The estimated multi-modal data types can be quickly obtained according to the historical interactive data. Then, the number of concentric rings can be determined according to the estimated multi-modal data types. The concentric rings are not directly overlapped, and the diameters of different concentric rings need to be different. The diameter of the concentric ring is determined based on the priority after the priority of the multi-modal data is judged. The priority is determined according to the different emphases in the interactive scene. For example, in the car navigation, the priority of voice is greater than that of gesture, that is, the priority of auditory modal is greater than that of visual modal. Therefore, the diameter of the concentric ring corresponding to the auditory modal is greater than that of the concentric ring corresponding to the visual modal. After the priority of each modal data is determined, the diameters of the concentric rings can be determined in turn. Then, the circular points are set, and the concentric rings with different diameters are nested with the circular points as the centers, so as to obtain a multi-intelligent ring body recognition architecture.

[0068] Preferably, the specific steps of the step S1 further include:

[0069] Step S14, sending a calibration instruction to the user, and collecting multi-modal data for calibration when the user executes the calibration instruction;

[0070] Step S15, adjusting the rotation angle of the concentric ring with the vertical diameter as the axis based on the multi-modal data for calibration.

[0071] In the constructed multi-intelligent ring body recognition architecture, the plurality of concentric rings are not arranged in parallel, but can rotate with the vertical diameter as the axis. After the initial multi-intelligent ring body recognition architecture is constructed, a calibration instruction can be sent to the user. The user will perform some actions according to the calibration instruction. Then, the multi-modal data for calibration under the calibration instruction is collected. The rotation angle of each concentric ring is adjusted in real time according to the multi-modal data for calibration, so that a multi-dimensional spherical space is formed between the concentric rings, and the strict alignment of the cross-modal data in the horizontal dimension is ensured.

[0072] Preferably, the specific steps of the step S2 of collecting the multi-modal data of the user and pre-processing the multi-modal data are:

[0073] Step S21, collecting multi-modal data of user interaction through the interactive screen, wearable terminal, camera and sensor, the multi-modal data including text, voice and image;

[0074] Step S22, denoising, data cleaning and normalization processing of the multi-modal data;

[0075] Step S23, feature extraction of the preprocessed multi-modal data of text, voice and image, obtaining word vector, sound frame containing mel spectrum sequence and image key frame.

[0076] When the user performs human-computer interaction, various modal data will be generated, for example, text modal is input through the interactive screen, haptic modal is collected through the wearable terminal, image and voice modal are collected through the camera and microphone, etc. For various modal data, it can be roughly divided into three categories, namely text, voice and image, for example, the user input text can be divided into text, the collected voice data and image data can be divided into voice and image, and the haptic and environmental data collected by the sensor or wearable device can be converted into a trend according to the change of time, and then the trend can generate a visualization chart and be used as image data. Regardless of which kind of multi-modal data, after being collected, it needs to be preprocessed, the preprocessing of the present application includes denoising, identifying and eliminating abnormal data points, data cleaning, and then normalizing the multi-modal data range to a specific interval.

[0077] After the preprocessing is completed, the modal data needs to be processed by feature extraction, and the features extracted from different types of modal data are different. The text can obtain word vector when performing feature extraction, the voice can extract sound frame containing mel spectrum sequence, and the image can extract image key frame, including color histogram and texture features, etc.

[0078] Preferably, the specific steps of mapping the different types of multi-modal data to the circumferential surface of different concentric ring bodies in step S2 are as follows:

[0079] Step S24, taking the intersection of the vertical radius of the concentric ring body and the concentric ring body as the origin;

[0080] Step S25, arranging the word vector, sound frame containing mel spectrum sequence and image key frame in the circumferential surface of the concentric ring body in time stamp order in turn and clockwise at intervals.

[0081] After the features of the multi-modal data are extracted, the features need to be mapped to the circumferential surface of the concentric ring body. In order to ensure the time consistency between the modal data, a starting point needs to be determined. The present application takes the connection between the vertical diameter of each concentric ring body and the top surface of the concentric ring body as the origin, and then maps the extracted word vector, the sound frame containing the mel spectrum sequence and the image key frame interval to the circumferential surface of the concentric ring body. The mapping condition is that the mapping is arranged in a clockwise direction according to the timestamp order, so as to facilitate the accurate correlation calculation of different modal data on the concentric ring body at the same timestamp.

[0082] Preferably, the specific steps of calculating the static feature in the step S3 are:

[0083] Step S31, calculate the cosine similarity between two concentric ring bodies at the same timestamp, and obtain a cosine similarity vector;

[0084] Step S32, calculate the self-similarity of the concentric ring body corresponding to the highest priority modal data, obtain a self-similarity vector, and splice the cosine similarity vector and the self-similarity vector into a static feature.

[0085] The calculation of the static feature and the dynamic feature is through the calculation of the similarity of different modal data. The static feature adopts the method of cosine similarity calculation. The cosine similarity vector can be obtained by calculating the cosine similarity between two concentric ring bodies at the same timestamp. For example, the cosine similarity between the word vector and the mel spectrum, or the similarity between the mel spectrum and the image key frame can be calculated. The direction similarity between vectors can be measured by calculating the cosine similarity. The highest priority modal data is also important for the recognition of the intention, so the self-similarity of the highest priority modal data is also calculated. The self-similarity vector and the cosine similarity vector are spliced into a complete static feature.

[0086] Preferably, the specific steps of calculating the dynamic feature in the step S3 are:

[0087] Step S33, compare the collected multi-modal data of the user with the calibration multi-modal data, obtain an adjustment angle based on the difference, and rotate the concentric ring body according to the adjustment angle;

[0088] Step S34, calculate the Pearson correlation coefficient between two concentric ring bodies at the same timestamp, and obtain a first Pearson correlation vector;

[0089] Step S35, extract the principal component sequence of the multi-modal data on the concentric ring body, calculate the Pearson correlation coefficient of the principal component sequence of two concentric ring bodies at the same timestamp, and obtain a second Pearson correlation vector;

[0090] Step S36, splice the first Pearson correlation vector and the second Pearson correlation vector into a dynamic feature.

[0091] The calculation of the static feature does not need to adjust the angle of the concentric ring body, but directly maps the multi-modal to the concentric ring body for calculation after adjusting the angle of the concentric ring body with the multi-modal data for calibration. After the calculation of the static feature is completed, the dynamic feature needs to be calculated. The dynamic feature of the application refers to the calculation result of the Pearson correlation between different modal data of the concentric ring body after the angle adjustment according to the actual multi-modal data of the user. When the user interacts subsequently, the habits and environment during the calibration will not be exactly the same, so it is necessary to compare the differences between the currently collected multi-modal data of the user and the multi-modal data for calibration, rotate the concentric ring body based on the differences, dynamically adjust the state of the concentric ring body of the multi-intelligent ring body recognition architecture, and then calculate the Pearson correlation coefficient between each two concentric ring bodies at the same timestamp to obtain a first Pearson correlation vector. In addition, the principal component sequence of the modal data on each concentric ring body is extracted, and then the Pearson correlation coefficient between each two principal component sequences is calculated to obtain a second Pearson correlation vector. Finally, the first Pearson correlation vector and the second Pearson correlation vector are spliced together to obtain the dynamic feature. The second Pearson correlation vector measures the synergy of the multi-modal data in the global trend, such as the common fluctuations of semantics, voice, and vision in the dialogue.

[0092] Preferably, in the step S3, the specific steps of fusing the static feature and the dynamic feature to obtain the fused feature are as follows:

[0093] Step S37, after the static feature and the dynamic feature are subjected to abnormal value processing, timestamp alignment, and feature normalization, the static feature and the dynamic feature are spliced into the fused feature at each time point.

[0094] After obtaining the static feature and the dynamic feature, feature fusion can be performed. In order to ensure the accuracy of the fusion, the static feature and the dynamic feature need to be preprocessed, including abnormal value processing, timestamp alignment, and feature normalization. Then, the static feature and the dynamic feature are spliced according to the passage of time to obtain the fused feature. The fused feature can be used for intent recognition.

[0095] Preferably, the specific steps of the step S4 are as follows:

[0096] Step S41, collect historical multi-modal data, and divide the historical multi-modal data into a training set and a test set according to a 7:3 ratio after artificial labeling;

[0097] Step S42, use a long short-term memory network to build an intent classification model, train the model through the training set, and test the model through the test set;

[0098] Step S43, input the fused feature into the trained intent classification model to generate an intent category probability distribution, and output the category with the highest probability as the user intent.

[0099] The intent classification model of the present application is based on a long short-term memory network (LSTM) framework, trained by collected historical multi-modal data, and tested by a test set after training is completed. When the test accuracy meets the requirements, the training is stopped. At this time, the fusion features can be input into the long short-term memory network, and the long short-term memory network can generate probability distribution of multiple intent categories after processing. Finally, the class with the highest probability is taken as the user intent and output. Artificial intelligence or data processing systems can execute corresponding commands or operations based on the user intent to meet user needs and improve user experience.

[0100] Referring to Figure 6 The system for intelligent interactive intent recognition based on multi-modal data, characterized in that it comprises:

[0101] The construction unit 1 is used to construct a multi-smart ring body recognition architecture based on users, and the multi-smart ring body recognition architecture comprises a plurality of concentric rings with different diameters.

[0102] The acquisition and mapping unit 2 is used to acquire multi-modal data of users, pre-process the multi-modal data, and map the multi-modal data to the circumferential surfaces of different concentric rings according to different types of multi-modal data.

[0103] The feature fusion unit 3 is used to calculate the static features and dynamic features of the multi-modal data located on the concentric rings respectively, fuse the static features and dynamic features, and obtain fusion features.

[0104] The intent recognition unit 4 is used to input the fusion features into an intent classification model to identify the specific intent of the user.

[0105] The acquisition and mapping unit 2, the construction unit 1, the feature fusion unit 3, and the intent recognition unit 4 are sequentially connected.

[0106] The construction unit 1 can construct a multi-intelligent ring body identification architecture containing several concentric rings of different diameters, each concentric ring corresponding to a modal data, the number of concentric rings can be determined according to different current interaction scenes, after the construction of the multi-intelligent ring body identification architecture is completed, the user can perform human-computer interaction, and the multi-modal data of the user is collected through the collection mapping unit 2, then according to the different types of multi-modal data, the multi-modal data is respectively mapped to the corresponding concentric ring, each concentric ring represents a modal data, the relative position and distance between the concentric rings represent the correlation between different modal data, then the feature fusion unit 3 can calculate the static features and dynamic features between the concentric rings, and splice into fusion features, the fusion features can be input into the intention classification model constructed by the intention recognition unit 4, and the user intention is obtained by the intention classification model, by constructing a three-dimensional multi-intelligent ring body identification architecture, the multi-modal data is converted into a correlation calculation problem in a three-dimensional solid, and accurate multi-modal data correlation parameters are obtained by calculating the static features and dynamic features, finally, the fusion features are recognized by the intention classification model, and the user intention can be accurately obtained, so that operation commands or related information can be executed based on the user intention, the needs of the user are met, and the experience of the user is improved.

[0107] The above only describes the preferred embodiments of the present application and is not intended to limit the present application. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included in the protection scope of the present application.

Claims

1. A method for intelligent interaction intent recognition based on multi-modal data, characterized in that, The method comprises the following steps: Step S1, constructing a multi-smart ring body recognition architecture based on a user, the multi-smart ring body recognition architecture comprising a plurality of concentric rings with different diameters; Step S2, collecting multi-modal data of the user, pre-processing the multi-modal data, and mapping the multi-modal data of different types to the circumferential surfaces of different concentric rings respectively; Step S3, calculating static features and dynamic features of the multi-modal data located on the concentric rings respectively, fusing the static features and the dynamic features to obtain fused features; Step S4, inputting the fused features into an intention classification model to identify the user's intention; The specific steps of step S1 comprise: Step S11, obtaining an interactive scene of the user, identifying the interactive scene according to historical interactive data, and obtaining estimated multi-modal data types; Step S12, determining the number of concentric rings according to the number of estimated multi-modal data types, and determining the priority of the estimated multi-modal data types based on the focus direction of the interactive scene; Step S13, determining the diameter of each concentric ring based on the priority, concentrically arranging the concentric rings, and obtaining the multi-smart ring body recognition architecture; The specific steps of calculating the static features in step S3 are as follows: Step S31, calculating the cosine similarity between two concentric rings at the same timestamp, and obtaining a cosine similarity vector; Step S32, calculating the self-similarity of the concentric ring corresponding to the modal data with the highest priority to obtain a self-similarity vector, and splicing the cosine similarity vector and the self-similarity vector into static features. 2.The intelligent interactive intent recognition method based on multi-modal data according to claim 1, characterized in that, The specific steps of step S1 further comprise: Step S14, sending a calibration instruction to the user, and collecting calibration multi-modal data of the user when executing the calibration instruction; Step S15, adjusting the rotation angle of the concentric ring with the vertical diameter as the axis based on the calibration multi-modal data. 3.The intelligent interactive intent recognition method based on multi-modal data according to claim 1, characterized in that, The specific steps of collecting multi-modal data of the user and pre-processing the multi-modal data in step S2 are as follows: Step S21, collecting multi-modal data of the user during interaction through an interactive screen, a wearable terminal, a camera, and a sensor, wherein the multi-modal data comprises text, voice, and images; Step S22, denoising, data cleaning, and normalization processing the multi-modal data; Step S23, extracting features from the pre-processed text, voice, and image multi-modal data respectively to obtain word vectors, sound frames containing mel spectrum sequences, and image key frames. 4.The intelligent interactive intent recognition method based on multi-modal data according to claim 3, characterized in that, The specific steps of mapping the multi-modal data of different types to the circumferential surfaces of different concentric rings in step S2 are as follows: Step S24, taking the intersection of the vertical radius of the concentric ring and the concentric ring as the origin; Step S25, arranging the word vectors, sound frames containing mel spectrum sequences, and image key frames in a clockwise and interval manner on the circumferential surface of the concentric ring in time stamp order. 5.The intelligent interactive intent recognition method based on multi-modal data according to claim 2, characterized in that, The specific steps of calculating the dynamic features in step S3 are as follows: Step S33, comparing the collected multi-modal data of the user with the calibration multi-modal data, obtaining an adjustment angle based on the difference, and rotating the concentric ring according to the adjustment angle; Step S34, calculate the Pearson correlation coefficient between each two concentric rings at the same timestamp, and obtain a first Pearson correlation vector; Step S35, extract the principal component sequence of the multi-modal data on the concentric rings, calculate the Pearson correlation coefficient of the principal component sequence of each two concentric rings at the same timestamp, and obtain a second Pearson correlation vector; Step S36, splice the first Pearson correlation vector and the second Pearson correlation vector into a dynamic feature. 6.The intelligent interactive intent recognition method based on multi-modal data according to claim 1, characterized in that, In the step S3, the specific steps of fusing the static feature and the dynamic feature to obtain the fusion feature are: Step S37, after the static feature and the dynamic feature are processed for abnormal values, time stamp alignment and feature normalization, splice them into the fusion feature at each time point. 7.The intelligent interactive intent recognition method based on multi-modal data according to claim 1, characterized in that, The specific steps of the step S4 are: Step S41, collect historical multi-modal data, and divide them into a training set and a test set after artificial labeling; Step S42, build an intent classification model using a long short-term memory network, train it through the training set, and test it through the test set; Step S43, input the fusion feature into the trained intent classification model to generate an intent category probability distribution, and output the class with the highest probability as the user intent.

8. A system for applying the intelligent interactive intent recognition method based on multi-modal data according to any one of claims 1-7, characterized in that, Comprise: A building unit is configured to build a multi-smart ring body recognition architecture based on a user, and the multi-smart ring body recognition architecture comprises a plurality of concentric rings with different diameters; A collection and mapping unit is configured to collect multi-modal data of the user, pre-process the multi-modal data, and map the multi-modal data to the circumferential surface of different concentric rings according to different types of multi-modal data; A feature fusion unit is configured to calculate the static feature and the dynamic feature of the multi-modal data located on the concentric rings respectively, fuse the static feature and the dynamic feature, and obtain a fusion feature; An intent recognition unit is configured to input the fusion feature into an intent classification model to identify the specific intent of the user; The collection and mapping unit, the building unit, the feature fusion unit, and the intent recognition unit are sequentially connected in data.

Images