Method of multi-modal cloud fusion service platform

Abstract

The invention discloses a method for a multi-modal cloud fusion service platform, and relates to the technical field of cloud service platforms, and the method comprises the following steps: S1, connecting a data source of a user through an API interface, collecting data of different modals, and obtaining a user fusion service demand and an application scene; and S2, preprocessing the collected data, establishing a cloud storage database, providing a multi-modal cloud storage service, establishing a data index and the like. Different modal data are collected in multiple modes, data sources are expanded, data richness and diversity are improved, data quality is improved through data preprocessing, a reliable data basis is provided for follow-up analysis, data are conveniently and rapidly retrieved and queried in multiple data indexing modes, data storage and management efficiency is improved, and the method is suitable for large-scale popularization and application. And user portraits and clustering analysis are established, so that user demands can be better known, personalized multi-modal services are provided for different user groups, and user satisfaction is improved.

Description

Technical Field

[0001] The present invention relates to the technical field of cloud service platforms, and in particular to a method for a multi-modal cloud fusion service platform. Background Art

[0002] With the rapid development of information technology, various sensors, devices and network platforms continue to generate large amounts of multimodal data, including text, images, audio, video, etc. Social media platforms are full of rich text, audio and video content; IoT devices continuously collect various environmental data and device status data. The development of cloud computing technology provides powerful computing and storage resources for multimodal cloud fusion service platforms. Cloud computing platforms can provide flexible computing capabilities and dynamically allocate computing resources according to user needs to meet the high computing requirements of multimodal data processing and analysis.

[0003] At present, traditional multimodal data processing methods are usually based on single-modal processing and lack effective multimodal fusion methods. This makes it impossible to fully utilize the correlation and complementarity between data of different modalities, affecting the effect and accuracy of data processing. In addition, they can usually only provide a single service and cannot meet the diverse needs of users. It is difficult to provide personalized services based on user needs and preferences. Summary of the invention

[0004] The purpose of the present invention is to provide a method for a multimodal cloud fusion service platform to solve the problem that the traditional multimodal data processing method proposed in the above background technology is usually based on single-modal processing and lacks an effective multimodal fusion method, which makes it impossible to give full play to the correlation and complementarity between data of different modalities, affecting the effect and accuracy of data processing, and usually can only provide a single service, unable to meet the diverse needs of users, and it is difficult to provide personalized services according to user needs and preferences.

[0005] To achieve the above object, the present invention provides the following technical solution: a method for a multimodal cloud fusion service platform, comprising the following steps:

[0006] S1. Use the API interface to connect to the user's data source to collect data of different modalities, and obtain the user's fusion service requirements and application scenarios. The API interface and specific data collection technology ensure the efficiency and accuracy of data collection;

[0007] S2. Preprocess the collected data, establish a cloud storage database to provide multimodal cloud storage services and establish data indexes. The data indexes include inverted indexes, spatial indexes and hybrid indexes. The API interface and specific data collection technology ensure the efficiency and accuracy of data collection.

[0008] S3. Extract features from data of different modalities and represent the extracted features in a unified manner. The unified representation can better reflect the comprehensive information of multimodal data and facilitate subsequent fusion and analysis.

[0009] S4. The extracted data features are fused based on the polynomial kernel function method, and the deep neural network model is used to automatically learn the association and fusion mode between different modalities, and the fusion results are evaluated and optimized. The feature extraction method for different modal data can fully explore the characteristics and value of each modal data. The fusion method based on the polynomial kernel function and the deep neural network model can effectively integrate different modal data, give play to the complementarity of multimodal data, and improve the accuracy of analysis and classification;

[0010] S5. Provide multimodal services according to user needs and application scenarios, and deploy the multimodal cloud fusion service platform to the cloud;

[0011] S6. Provide a user interaction interface for users to upload multimodal data, query service results and manage personal information;

[0012] S7. Regularly maintain and update the platform, monitor the platform's operating status, and collect user feedback. Regular maintenance and updates and monitoring of the platform's operating status ensure the stability and performance of the platform. Collecting user feedback helps to continuously improve the platform's functions and services and enhance the platform's adaptability and competitiveness.

[0013] Preferably, in step S1, the API interface uses JSON or XML as a data exchange format to provide interaction between different systems. In the multimodal data collection, for image data, API collection and web crawler technology are used to collect relevant images from a specific image website, and a deep learning-based target detection algorithm is used to automatically identify and classify image content; for audio data, direct collection or speech recognition technology is used to convert audio into text.

[0014] Preferably, in step S1, obtaining user fusion service requirements and application scenarios includes the following steps:

[0015] S11. Obtain user behavior data, preference data and demand data, integrate user data collected from different channels and data sources, establish a unified user data warehouse and perform data cleaning, and perform feature extraction and analysis on the cleaned data;

[0016] S12. Build a user portrait model and use the K-Means clustering method to group users. For each user data point in the data set, calculate its distance to each cluster center. According to the principle of the closest distance, assign the data point to the corresponding cluster and divide the users into different groups.

[0017] S13. Provide personalized services for different groups, including personalized recommendations, customized functional services, and exclusive content creation and services.

[0018] Preferably, in step S2, the data preprocessing includes text data cleaning and format conversion processing, image data enhancement processing and data missing value processing, wherein the missing value processing uses the K nearest neighbor algorithm to fill the missing values ​​and specifically includes the following steps:

[0019] S21, for two sample points A=(a1, a2,…, a n ) and B=(b1,b2,…,b n ), the Euclidean distance calculation formula is:

[0020]

[0021] S22. Calculate the distance between the filled sample X and other samples, select the k samples with the closest distance as the nearest neighbors, and set the corresponding feature values ​​of these k nearest neighbor samples to be v1, v2, …, v n ;

[0022] S23. Different weights are assigned according to the sample distance. The closer the distance, the greater the weight. Suppose the distance between sample X and its nearest neighbor sample i is , the weight is defined as, and the filling value is the weighted average:

[0023]

[0024] Preferably, in step S3, the feature extraction includes the following steps:

[0025] S31, text data: Use the RNN recurrent neural network model to model the text sequence and extract semantic features;

[0026] S32, Image data: Use convolutional neural networks to highlight important areas in the image for feature extraction;

[0027] S33, audio data: use Mel frequency cepstral coefficients combined with convolutional neural network to extract audio features. The specific process is as follows:

[0028] ① Audio preprocessing: Split the continuous audio signal into several short audio frames, each of which is 20-40 milliseconds long. There is a certain overlap between frames to ensure the continuity of the audio signal. Apply a window function to each audio frame for windowing processing, pre-emphasize the audio signal, and enhance the high-frequency part of the audio signal through a first-order high-pass filter to compensate for the attenuation of the speech signal in the high-frequency band and highlight the high-frequency resonance peak of the audio signal;

[0029] ② Perform fast Fourier transform on the windowed audio frame to convert the time domain signal into the frequency domain signal to obtain the spectrum. The spectrum is converted from the linear frequency scale to the Mel frequency scale through the Mel filter bank. The Mel filter bank is a group of triangular bandpass filters uniformly distributed on the Mel frequency scale. The conversion relationship between the Mel frequency and the actual frequency is:

[0030]

[0031] Where f is the actual frequency. The logarithmic energy of each Mel filter output is taken to obtain the logarithmic Mel spectrum. The logarithmic Mel spectrum is subjected to discrete cosine transform to obtain the Mel frequency cepstrum coefficients. The first 13-20 coefficients are taken as the feature representation of the audio.

[0032] ③ Use the calculated MFCC features as the input of the convolutional neural network to extract the audio features and convert them into feature vectors of fixed length.

[0033] Preferably, in step S4, the method based on the polynomial kernel function to fuse the extracted data features comprises the following steps:

[0034] S41. For each modal data, calculate its kernel matrix. The data of the two modalities are set as X = (x1, x2, ..., x n ) and Y=(y1,y2,…,y m );

[0035] Among them, the kernel matrix K of X X ; K X (i,j)=K(x i ,x j ), the kernel matrix K of Y Y ; K Y (i,j)=K(y i ,y j );

[0036] S42. Weighted fusion of kernel matrices of different modes:

[0037] K f =αK X +(1-α)K Y ;

[0038] Among them, K f is the fused kernel matrix, α is the weight parameter;

[0039] S43. Based on the fused kernel matrix, use support vector machine (SVM) to perform classification in the kernel space.

[0040] Preferably, in step S4, the process of evaluating and optimizing the fusion results is as follows: according to the specific application scenario, define specific evaluation indicators, and then divide the data set into k equal parts, use k-1 parts for each training, and use the remaining part for verification, repeat k times, record the evaluation results of each fold and calculate the average value, count various evaluation indicators, analyze the performance of the model in different categories or different modes, identify the advantages and disadvantages of the model, analyze the confusion matrix, find the categories of model confusion and identify the direction that needs improvement.

[0041] Preferably, in step S5, the platform is deployed to the cloud using containerized deployment technology, and a continuous integration and continuous deployment process is implemented to ensure that updates and improvements to the platform are released online simultaneously.

[0042] Preferably, in step S6, the user uploading multimodal data provides a visual data upload progress bar, so that the user can understand the status and progress of the upload, and perform real-time preview and verification of the uploaded data to ensure that the quality and format of the data meet the requirements; in the query service results, a collaborative filtering algorithm is used to provide the user with query results that meet their needs based on the user's historical queries and preferences.

[0043] Compared with the prior art, the present invention has the following beneficial effects:

[0044] 1. In the present invention, different modal data are collected through multiple methods, which expands the data source, improves the richness and diversity of the data, improves the data quality through data preprocessing, and provides a reliable data basis for subsequent analysis. Multiple data indexing methods facilitate rapid retrieval and query of data, improve the efficiency of data storage and management, establish user portraits and cluster analysis, can better understand user needs, provide personalized multimodal services for different user groups, and improve user satisfaction.

[0045] 2. In the present invention, the fusion method based on polynomial kernel function and deep neural network model can effectively integrate different modal data, give play to the complementarity of multimodal data, improve the accuracy of analysis and classification, evaluate and optimize the fusion results, and identify the advantages and disadvantages of the model and the direction of improvement by defining evaluation indicators, cross-validation, analyzing evaluation results and confusion matrix.

[0046] 3. In the present invention, multimodal services are provided according to user needs and application scenarios, the platform is deployed to the cloud using containerized deployment technology, and continuous integration and continuous deployment processes are implemented to ensure synchronous updates. By providing a user interaction interface, including a visual upload progress bar, real-time preview and verification of uploaded data, a collaborative filtering algorithm is used to provide users with query results that meet their needs and personal information management functions. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] Figure 1A flowchart of a method for a multi-modal cloud fusion service platform of the present invention DETAILED DESCRIPTION

[0048] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the implementation regulations described are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0049] See also Figure 1 The present invention provides a technical solution: a method for a multimodal cloud fusion service platform, comprising the following steps:

[0050] Step 1: Use the API interface to connect to the user's data source to collect data of different modalities, and obtain the user's fusion service requirements and application scenarios. The API interface and specific data collection technology ensure the efficiency and accuracy of data collection. At the same time, in order to improve the security of the interface, authentication mechanisms such as API keys and OAuth can be used to prevent unauthorized access, thereby effectively protecting the security of user data;

[0051] The API interface uses JSON or XML as the data exchange format to provide interaction between different systems. In multimodal data collection, API collection and web crawler technology are used to collect relevant images from specific image websites for image data, and the target detection algorithm based on deep learning is used to automatically identify and classify image content; for audio data, direct collection or speech recognition technology is used to convert audio into text;

[0052] The acquisition of user integration service requirements and application scenarios includes the following steps:

[0053] 11) Obtain user behavior data, preference data and demand data, integrate user data collected from different channels and data sources, establish a unified user data warehouse and perform data cleaning, and perform feature extraction and analysis on the cleaned data;

[0054] 12) Build a user portrait model and use the K-Means clustering method to group users. For each user data point in the data set, calculate its distance to each cluster center. According to the principle of the closest distance, assign the data point to the corresponding cluster and divide the users into different groups.

[0055] 13) Providing personalized services for different groups, including personalized recommendations, customized functional services, and exclusive content creation and services, to better understand user needs, provide personalized multimodal services for different user groups, and improve user satisfaction;

[0056] Step 2: Preprocess the collected data, establish a cloud storage database to provide multimodal cloud storage services and establish data indexes. The data indexes include inverted indexes, spatial indexes, and hybrid indexes. The multi-source data indexing method facilitates rapid retrieval and query of data, thereby improving the efficiency of data storage and management.

[0057] Establish a cloud storage database and provide multimodal cloud storage services. The cloud storage database has the advantages of high reliability and high scalability, and can meet the storage needs of multimodal data. At the same time, establish data indexes, including inverted indexes, spatial indexes, and hybrid indexes. Inverted indexes are suitable for text data and can quickly retrieve text content based on keywords. Spatial indexes are suitable for data with spatial characteristics such as image data and can be retrieved based on spatial location. Hybrid indexes combine the advantages of inverted indexes and spatial indexes and are suitable for retrieval of multimodal data. Through a variety of data indexing methods, data can be quickly retrieved and queried, improving the efficiency of data storage and management;

[0058] Among them, data preprocessing includes text data cleaning and format conversion processing, image data enhancement processing and data missing value processing. Missing value processing uses the K nearest neighbor algorithm to fill missing values, which can effectively fill missing values ​​in the data and improve data integrity. Specifically, it includes the following steps:

[0059] 21) For two sample points A=(a1,a2,…,a n ) and B=(b1,b2,…,b n ), the Euclidean distance calculation formula is:

[0060]

[0061] 22) Calculate the distance between the filled sample X and other samples, select the k samples with the closest distance as the nearest neighbors, and set the corresponding feature values ​​of these k nearest neighbor samples to be v1, v2, …, v n ;

[0062] 23) Different weights are assigned according to the sample distance. The closer the distance is, the greater the weight is. Suppose the distance between sample X and its nearest neighbor sample i is, the weight is defined as, and the filling value is the weighted average:

[0063]

[0064] Step 3: Extract features from data of different modalities and represent the extracted features in a unified manner. The unified representation can better reflect the comprehensive information of multimodal data and facilitate subsequent fusion and analysis. Feature extraction includes the following steps:

[0065] 31) Text data: Use the RNN recurrent neural network model to model the text sequence and extract semantic features;

[0066] 32) Image data: Use convolutional neural networks to highlight important areas in the image for feature extraction;

[0067] 33) Audio data: Use Mel frequency cepstral coefficients combined with convolutional neural networks to extract audio features. The specific process is as follows:

[0068] ① Audio preprocessing: Split the continuous audio signal into several short audio frames, each of which is 20-40 milliseconds long. There is a certain overlap between frames to ensure the continuity of the audio signal. Apply a window function to each audio frame for windowing processing, pre-emphasize the audio signal, and enhance the high-frequency part of the audio signal through a first-order high-pass filter to compensate for the attenuation of the speech signal in the high-frequency band and highlight the high-frequency resonance peak of the audio signal;

[0069] ② Perform fast Fourier transform on the windowed audio frame to convert the time domain signal into the frequency domain signal to obtain the spectrum. The spectrum is converted from the linear frequency scale to the Mel frequency scale through the Mel filter bank. The Mel filter bank is a group of triangular bandpass filters uniformly distributed on the Mel frequency scale. The conversion relationship between the Mel frequency and the actual frequency is:

[0070]

[0071] Where f is the actual frequency. The logarithmic energy of each Mel filter output is taken to obtain the logarithmic Mel spectrum. The logarithmic Mel spectrum is subjected to discrete cosine transform to obtain the Mel frequency cepstrum coefficients. The first 13-20 coefficients are taken as the feature representation of the audio.

[0072] ③ Use the calculated MFCC features as the input of the convolutional neural network, extract the audio features and convert them into feature vectors of fixed length;

[0073] Step 4: The extracted data features are fused based on the polynomial kernel function method, and the deep neural network model is used to automatically learn the association and fusion mode between different modalities, and the fusion results are evaluated and optimized. The feature extraction method for different modal data can fully explore the characteristics and value of each modal data. The fusion method based on the polynomial kernel function and the deep neural network model can effectively integrate different modal data, give play to the complementarity of multimodal data, and improve the accuracy of analysis and classification;

[0074] The method based on the polynomial kernel function to fuse the extracted data features includes the following steps:

[0075] 41) For each modal data, calculate its kernel matrix. The data of the two modalities are set as X = (x1, x2, ..., x n ) and Y=(y1,y2,…,y m );

[0076] Among them, the kernel matrix K of X X ; K X (i,j)=K(x i ,x j ), the kernel matrix K of Y Y ; K Y (i,j)=K(y i ,y j ); By calculating the kernel matrix, data of different modalities can be mapped into a high-dimensional space, so that data of different modalities have better separability in the high-dimensional space;

[0077] 42) Perform weighted fusion of kernel matrices of different modalities:

[0078] K f =αK X +(1-α)K Y ;

[0079] Among them, K f is the kernel matrix after fusion, and α is the weight parameter. By adjusting the weight parameter, the contribution of different modalities in the fusion can be controlled, which can give full play to the advantages of different modalities and improve the accuracy of the fusion result.

[0080] 43) Based on the fused kernel matrix, support vector machine (SVM) is used to perform classification in the kernel space, which can make full use of the fusion features of multimodal data and improve the accuracy of classification;

[0081] Specifically, the process of evaluating and optimizing the fusion results is as follows: define specific evaluation indicators according to the specific application scenario, then divide the data set into k equal parts, use k-1 parts for each training, and use the remaining part for verification, repeat k times, record the evaluation results of each fold and calculate the average value, count various evaluation indicators, analyze the performance of the model in different categories or different modes, identify the advantages and disadvantages of the model, analyze the confusion matrix, find the categories that the model is confused about and identify the direction that needs improvement; through this cross-validation method, more stable and reliable evaluation results can be obtained. Count various evaluation indicators, analyze the performance of the model in different categories or different modes, identify the advantages and disadvantages of the model, and adjust the parameters of the kernel function, optimize the structure of the deep neural network, adjust the weight parameters, etc. to improve the accuracy and stability of the fusion results according to the shortcomings of the model;

[0082] Step 5: Provide multimodal services based on user needs and application scenarios, and deploy the multimodal cloud fusion service platform to the cloud;

[0083] The platform is deployed to the cloud using containerized deployment technology, and continuous integration and continuous deployment processes are implemented to ensure that updates and improvements to the platform are released online simultaneously. Containerized deployment and continuous integration and continuous deployment processes ensure the high availability, scalability and rapid updates of the platform to adapt to changing user needs and technological developments.

[0084] Step 6: Provide a user interaction interface for users to upload multimodal data, query service results and manage personal information;

[0085] When users upload multimodal data, a visual data upload progress bar is provided to let users know the status and progress of the upload, and to preview and verify the uploaded data in real time to ensure that the quality and format of the data meet the requirements. In the query service results, a collaborative filtering algorithm is used to provide users with query results that meet their needs based on their historical queries and preferences. By analyzing the user's historical behavior and the behavior of other users, user groups similar to the current user are found, and then relevant service results are recommended to the current user based on the behavior of these similar users, thereby improving the accuracy and personalization of user query results and meeting user needs.

[0086] Step 7. Regularly maintain and update the platform, monitor the platform's operating status, and collect user feedback. Regular maintenance and updates and monitoring of the platform's operating status ensure the stability and performance of the platform. Collecting user feedback helps to continuously improve the platform's functions and services and enhance the platform's adaptability and competitiveness.

[0087] In this method, different modal data are collected in a variety of ways, which expands the data sources, improves the richness and diversity of data, establishes user portraits and cluster analysis, can better understand user needs, provide personalized multimodal services for different user groups, and improve user satisfaction. Data preprocessing improves data quality and provides a reliable data basis for subsequent analysis. Multiple data indexing methods facilitate rapid retrieval and query of data, improve the efficiency of data storage and management, and the fusion method based on polynomial kernel function and deep neural network model can effectively integrate different modal data, give play to the complementarity of multimodal data, and improve the accuracy of analysis and classification. Containerized deployment and continuous integration and continuous deployment processes ensure the high availability, scalability and rapid update of the platform, adapt to the ever-changing user needs and technological development, provide a good user experience, and facilitate users to upload data, query results and manage personal information.

[0088] Although the present invention has been described in detail with reference to the aforementioned embodiments, it is still possible for those skilled in the art to modify the technical solutions described in the aforementioned embodiments, or to make equivalent substitutions for some of the technical features therein. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A method for a multimodal cloud fusion service platform, characterized in that: The following steps are involved: S1. Use the API interface to connect to the user's data source to collect data of different modalities, and obtain the user's fusion service requirements and application scenarios; S2. Preprocess the collected data, establish a cloud storage database to provide multimodal cloud storage services and establish data indexes, which include inverted indexes, spatial indexes and hybrid indexes; S3, extracting features from data of different modalities and expressing the extracted features in a unified manner; S4, based on the polynomial kernel function method, the extracted data features are fused, the deep neural network model is used to automatically learn the association and fusion mode between different modalities, and the fusion results are evaluated and optimized; S5. Provide multimodal services according to user needs and application scenarios, and deploy the multimodal cloud fusion service platform to the cloud; S6. Provide a user interaction interface for users to upload multimodal data, query service results and manage personal information; S7. Regularly maintain and update the platform, monitor the platform's operating status, and collect user feedback.

2. The method of a multimodal cloud fusion service platform according to claim 1, characterized in that: In step S1, the API interface uses JSON or XML as a data exchange format to provide interaction between different systems; in the multimodal data collection, for image data, API collection and web crawler technology are used to collect relevant images from a specific image website, and a deep learning-based target detection algorithm is used to automatically identify and classify image content; for audio data, direct collection or speech recognition technology is used to convert audio into text.

3. The method of a multimodal cloud fusion service platform according to claim 1, characterized in that: In step S1, obtaining user fusion service requirements and application scenarios includes the following steps: S11. Obtain user behavior data, preference data and demand data, integrate user data collected from different channels and data sources, establish a unified user data warehouse and perform data cleaning, and perform feature extraction and analysis on the cleaned data; S12. Build a user portrait model and use the K-Means clustering method to group users. For each user data point in the data set, calculate its distance to each cluster center. According to the principle of the closest distance, assign the data point to the corresponding cluster and divide the users into different groups. S13. Provide personalized services for different groups, including personalized recommendations, customized functional services, and exclusive content creation and services.

4. The method of a multimodal cloud fusion service platform according to claim 1, characterized in that: In step S2, the data preprocessing includes text data cleaning and format conversion processing, image data enhancement processing and data missing value processing. The missing value processing uses the K nearest neighbor algorithm to fill in the missing values ​​and specifically includes the following steps: S21, for two sample points A=(a1, a2,…, a n ) and B=(b1,b2,…,b n ), the Euclidean distance calculation formula is: S22. Calculate the distance between the filled sample X and other samples, select the k samples with the closest distance as the nearest neighbors, and set the corresponding feature values ​​of these k nearest neighbor samples to be v1, v2, …, v n ; S23. Different weights are assigned according to the sample distance. The closer the distance, the greater the weight. Suppose the distance between sample X and its nearest neighbor sample i is , the weight is defined as, and the filling value is the weighted average:

5. The method of a multimodal cloud fusion service platform according to claim 1, characterized in that: In step S3, the feature extraction includes the following steps: S31, text data: Use the RNN recurrent neural network model to model the text sequence and extract semantic features; S32, Image data: Use convolutional neural networks to highlight important areas in the image for feature extraction; S33, audio data: use Mel frequency cepstral coefficients combined with convolutional neural network to extract audio features. The specific process is as follows: ① Audio preprocessing: Split the continuous audio signal into several short audio frames, each of which is 20-40 milliseconds long. There is a certain overlap between frames to ensure the continuity of the audio signal. Apply a window function to each audio frame for windowing processing, pre-emphasize the audio signal, and enhance the high-frequency part of the audio signal through a first-order high-pass filter to compensate for the attenuation of the speech signal in the high-frequency band and highlight the high-frequency resonance peak of the audio signal; ② Perform fast Fourier transform on the windowed audio frame to convert the time domain signal into the frequency domain signal to obtain the spectrum. The spectrum is converted from the linear frequency scale to the Mel frequency scale through the Mel filter bank. The Mel filter bank is a group of triangular bandpass filters uniformly distributed on the Mel frequency scale. The conversion relationship between the Mel frequency and the actual frequency is: Where f is the actual frequency. The logarithmic energy of each Mel filter output is taken to obtain the logarithmic Mel spectrum. The logarithmic Mel spectrum is subjected to discrete cosine transform to obtain the Mel frequency cepstrum coefficients. The first 13-20 coefficients are taken as the feature representation of the audio. ③ Use the calculated MFCC features as the input of the convolutional neural network to extract the audio features and convert them into feature vectors of fixed length.

6. The method of a multimodal cloud fusion service platform according to claim 1, characterized in that: In step S4, the method based on the polynomial kernel function to fuse the extracted data features includes the following steps: S41. For each modal data, calculate its kernel matrix. The data of the two modalities are set as X = (x1, x2, ..., x n ) and Y=(y1,y2,…,y m ); Among them, the kernel matrix K of X X ; K X (i,j)=K(x i ,x j ), the kernel matrix K of Y Y ; K Y (i,j)=K(y i ,y j ); S42. Weighted fusion of kernel matrices of different modes: K f =αK X +(1-α)K Y ; Among them, K f is the fused kernel matrix, α is the weight parameter; S43. Based on the fused kernel matrix, use support vector machine (SVM) to perform classification in the kernel space.

7. The method of a multimodal cloud fusion service platform according to claim 6, characterized in that: In step S4, the process of evaluating and optimizing the fusion results is as follows: according to the specific application scenario, define specific evaluation indicators, then divide the data set into k equal parts, use k-1 parts for each training, and use the remaining part for verification, repeat k times, record the evaluation results of each fold and calculate the average value, count various evaluation indicators, analyze the performance of the model in different categories or different modes, identify the advantages and disadvantages of the model, analyze the confusion matrix, find the categories of model confusion and identify the direction that needs improvement.

8. The method of a multimodal cloud fusion service platform according to claim 1, characterized in that: In step S5, the platform is deployed to the cloud using containerized deployment technology, and a continuous integration and continuous deployment process is implemented to ensure that updates and improvements to the platform are released online simultaneously.

9. The method of a multimodal cloud fusion service platform according to claim 1, characterized in that: In step S6, the user uploads multimodal data and provides a visual data upload progress bar to let the user know the status and progress of the upload, and preview and verify the uploaded data in real time to ensure that the quality and format of the data meet the requirements; in the query service results, a collaborative filtering algorithm is used to provide the user with query results that meet their needs based on the user's historical queries and preferences.

Images