Multi-modal learning automatic complaint content analysis and classification method and system

Abstract

The invention discloses a multi-modal learning automatic complaint content analysis and classification method and system, and relates to the technical field of semantic analysis, and the method comprises the steps: collecting multi-modal complaint data, and carrying out the preprocessing of the collected multi-modal complaint data; extracting features of the preprocessed multi-modal complaint data, and fusing the features of different modal data; constructing a complaint content classification model by using the fused features, carrying out model training, and evaluating and optimizing the trained model; and deploying the optimized model for real-time classification. According to the multi-modal learning automatic complaint content analysis and classification method provided by the invention, the complaint content is quickly and accurately classified, and a targeted processing strategy is provided according to the main category, the emotional tendency and the specific theme of the complaint. And an emergency response mechanism is automatically triggered to ensure that problems are processed in real time and effective utilization of resources is ensured. Decision rules are continuously optimized from historical data learning, and the intelligent level of a processing strategy and the customer satisfaction degree are improved.

Description

Technical Field

[0001] The present invention relates to the technical field of semantic analysis, and specifically to a multimodal learning automated complaint content analysis and classification method and system. Background Art

[0002] In the current multimodal complaint handling system, the main challenge is how to efficiently analyze and respond to complex complaints that include text, images, audio and contextual data. Traditional processing methods often rely on analyzing these different types of data in isolation, lacking an effective mechanism to integrate and utilize the potential connections between these data, which may cause important connections in the processing process to be overlooked, affecting the accuracy of classification and the timeliness of response.

[0003] In addition, existing automated complaint handling systems usually make decisions using fixed rules that lack flexibility and cannot adjust themselves based on real-time data or historical processing results. Such systems that lack dynamic learning and adaptability are ineffective in handling sudden or complex complaints, cannot provide personalized handling strategies, and have difficulty meeting growing customer service expectations.

[0004] Therefore, there is an urgent need for a multimodal learning automated complaint content analysis and classification method to optimize the analysis process of complaint handling and provide more intelligent and personalized technical solutions, thereby improving processing efficiency and customer satisfaction. Summary of the invention

[0005] In view of the above-mentioned problems, the present invention is proposed.

[0006] Therefore, the technical problems solved by the present invention are: the existing complaint handling methods have the problems of low processing efficiency and insufficient flexibility in response, and the optimization problem of how to comprehensively utilize multimodal data and dynamically optimize the processing strategy.

[0007] In order to solve the above technical problems, the present invention provides the following technical solutions: a multimodal learning automated complaint content analysis and classification method, comprising: Collect multimodal complaint data and pre-process the collected multimodal complaint data; Extract the features of pre-processed multimodal complaint data and fuse the features of data of different modalities; Use the fused features to build a complaint content classification model, perform model training, and evaluate and optimize the trained model; Deploy the optimized model for real-time classification.

[0008] As a preferred solution of the multimodal learning automated complaint content analysis and classification method of the present invention, wherein: the multimodal complaint data includes complaint data and context data; The complaint data includes text data, image data and audio data; The preprocessing includes data cleaning, data standardization, format conversion and data integration.

[0009] As a preferred solution of the multimodal learning automated complaint content analysis and classification method described in the present invention, wherein: the extracting features of the pre-processed multimodal complaint data includes constructing a multi-input feature extraction model, processing text, image, audio and context data through four branches, extracting features of each modality, and forming a preliminary feature representation; Introducing a feature interaction layer, applying the self-attention mechanism to optimize feature relationships, and generating a comprehensive joint feature representation through the feature interaction mechanism; Based on the joint feature representation generated by the feature interaction layer, a context-aware fusion strategy is designed to dynamically adjust the feature fusion weights and optimize the fusion effect.

[0010] As a preferred solution of the multimodal learning automated complaint content analysis and classification method described in the present invention, wherein: the extraction of each modal feature includes encoding the text data using the BERT model, extracting semantic features, and converting them into text feature vectors; Use the pre-trained ResNet model to encode the image data, extract the visual features of the image, and convert them into image feature vectors; Use MFCC technology to extract the spectral features of the audio, use convolutional neural network to encode the extracted audio features, and convert them into audio feature vectors; The context data is converted into numerical features through the embedding layer to generate a context feature vector, which is integrated with features of other modalities within the model.

[0011] As a preferred solution of the multimodal learning automated complaint content analysis and classification method of the present invention, wherein: the generated joint feature representation includes weighted processing of feature vectors extracted from text, image, audio and context data; By calculating the similarity between each feature vector and other feature vectors, the self-attention mechanism automatically adjusts and optimizes the weights and associations between different features; After the self-attention mechanism is used, the feature interaction layer fuses and interactively learns all weighted feature vectors to capture the potential correlation between modalities and combines and optimizes the features of each modality through the feature interaction mechanism. When the feature interaction layer is processed, a joint feature representation is generated; The optimization fusion strategy includes generating a dynamic weight adjustment joint feature representation based on the joint feature representation and according to the context information analyzed in real time; By training the optimized joint feature representation, using the patterns in historical data and the information from real-time feedback, the feature fusion strategy is further adjusted to generate the final optimized joint feature representation.

[0012] As a preferred solution of the multimodal learning automated complaint content analysis and classification method described in the present invention, wherein: the construction of the complaint content classification model includes using the final optimized joint feature representation as input to construct a multi-output deep learning model, the underlying feature extraction layer of the model first shares the input joint feature representation, and performs preliminary feature processing through a multi-layer neural network; The model is divided into three parallel task branches, including a main classification task branch, a sentiment analysis task branch, and a topic identification task branch; The main classification task branch uses the features further processed from the primary feature layer to perform deep learning through a multi-layer fully connected neural network, and finally outputs the main categories of complaints; The sentiment analysis task branch starts from a shared primary feature layer, adds a neural layer to capture emotional features, and the neural layer focuses on understanding the emotional tendency, and finally outputs the emotional tendency of the complaint; The topic identification task branch starts from the shared primary feature layer, adds the text analysis layer to process the text data, identifies and classifies the key topics in the text, and finally outputs the specific topic of the complaint; During the model training phase, a comprehensive loss function is defined to simultaneously consider the task effects of main classification, sentiment analysis, and topic identification, and a multi-task learning strategy is used to optimize the branches. The weights and biases of the model are adjusted through the gradient descent algorithm, and the Adam optimization algorithm is used for dynamic learning rate adjustment; After each training cycle, an independent validation set is used to evaluate the performance of the model, and the hyperparameters are adjusted based on the evaluation results.

[0013] As a preferred solution of the multimodal learning automated complaint content analysis and classification method described in the present invention, the real-time classification includes receiving new complaint data in real time after the model is deployed, preprocessing the data and extracting joint feature representations, inputting them into the model for classification, and the model simultaneously outputs the main categories, sentiment tendencies and specific topics of the complaints based on the input features; The extracted features are input into the trained deep learning model, which simultaneously outputs the main categories, sentiment tendencies, and specific topics of the complaints; The output results are then fed into a dynamic decision engine, which builds a decision model based on a decision tree and generates processing strategies for various scenarios; If the main category is a security issue, the emergency response mechanism is triggered, the automated monitoring and immediate reporting process is started, the system status is monitored in real time, and emergency notifications are automatically sent to relevant departments and management to ensure immediate processing; If the main category is product quality or service feedback, then enter the sentiment analysis process; For negative emotions, emotion recognition technology is introduced to further analyze the urgency and severity of complaints and adjust response priorities and resource allocation; For positive or neutral emotions, we conduct satisfaction prediction analysis to predict the customer's satisfaction with the current processing progress and then move on to the topic identification process; Based on the results of topic identification, for refund requests, trigger the financial process to predict potential refund risks based on customer historical data and purchasing patterns, and adjust inventory and pricing strategies accordingly; For the topic of product suggestions, consolidate customer feedback and generate reports.

[0014] Another object of the present invention is to provide a multimodal learning automated complaint content analysis and classification system, which can solve the problems of insufficient information utilization in existing single data source processing and insufficient flexibility in response in fixed rule decision-making systems by constructing a multimodal learning automated complaint content analysis and classification system.

[0015] To solve the above technical problems, the present invention provides the following technical solutions: a multimodal learning automated complaint content analysis and classification system, comprising: a data acquisition module, a data fusion module, a model training module and a classification prediction module; the data acquisition module is used to collect multimodal complaint data and preprocess the collected multimodal complaint data; the data fusion module is used to extract the features of the preprocessed multimodal complaint data and fuse the features of different modal data; the model training module is used to use the fused features to perform model training, evaluate and optimize the trained model; the classification prediction module is used to deploy the optimized model for real-time classification.

[0016] A computer device includes a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the steps of the multimodal learning automated complaint content analysis and classification method as described above are implemented.

[0017] A computer-readable storage medium stores a computer program, which, when executed by a processor, implements the steps of the multimodal learning automated complaint content analysis and classification method as described above.

[0018] Beneficial effects of the present invention: After receiving new complaint data in real time, the multimodal learning automated complaint content analysis and classification method provided by the present invention can quickly and accurately classify the complaint content, and propose targeted processing strategies based on the main categories, emotional tendencies and specific topics of the complaints. Automatically trigger the emergency response mechanism to ensure that the problem is handled immediately. Adjust the processing priority and resource allocation based on emotional analysis to ensure the effective use of resources. Through the context-aware dynamic decision engine, it is possible to learn from historical data, continuously optimize decision rules, and improve the intelligence level of processing strategies and customer satisfaction. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative work.

[0020] Figure 1 An overall flow chart of a multimodal learning automated complaint content analysis and classification method provided for one embodiment of the present invention.

[0021] Figure 2 An overall structural diagram of a multimodal learning automated complaint content analysis and classification system provided for the second embodiment of the present invention. DETAILED DESCRIPTION

[0022] In order to make the above-mentioned purposes, features and advantages of the present invention more obvious and easy to understand, the specific implementation methods of the present invention are described in detail below in conjunction with the drawings of the specification. Obviously, the described embodiments are part of the embodiments of the present invention, but not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary persons in the art without creative work should fall within the scope of protection of the present invention.

[0023] In the following description, many specific details are set forth to facilitate a full understanding of the present invention, but the present invention may also be implemented in other ways different from those described herein, and those skilled in the art may make similar generalizations without violating the connotation of the present invention. Therefore, the present invention is not limited to the specific embodiments disclosed below.

[0024] Example 1 Reference Figure 1 , is an embodiment of the present invention, and provides a multimodal learning automated complaint content analysis and classification method, comprising: Collect multimodal complaint data and pre-process the collected multimodal complaint data; Extract the features of pre-processed multimodal complaint data and fuse the features of data of different modalities; Use the fused features to build a complaint content classification model, perform model training, and evaluate and optimize the trained model; Deploy the optimized model for real-time classification.

[0025] The multimodal complaint data includes complaint data and context data; The complaint data includes text data, image data and audio data; The preprocessing includes data cleaning, data standardization, format conversion and data integration.

[0026] Contextual data includes device type, error code, timestamp, geolocation information, and user history data.

[0027] Equipment type, error code Collect technical parameters such as equipment model, error code, etc. related to the complaint.

[0028] Timestamp and geolocation information collects the specific time and location of the complaint to provide the environmental context of the incident.

[0029] User historical data includes user’s past complaint records, service usage history, and any relevant user behavior patterns.

[0030] Text cleaning removes HTML tags, URL links, and non-printing characters, and uses regular expressions to clean special symbols and numbers to purify text data and make it more suitable for subsequent NLP processing.

[0031] Image cleaning automatically detects and repairs damaged image files, removes background noise, improves image quality, and prepares for subsequent image feature extraction.

[0032] Audio cleaning filters out background noise and echo, standardizes volume, clarifies audio data, and provides accurate input for audio feature extraction.

[0033] Unified formatting ensures the consistency of all data in time and space, which helps to consider temporal and spatial factors in subsequent analysis.

[0034] The extracting of features of the pre-processed multimodal complaint data includes constructing a multi-input feature extraction model, processing text, image, audio and context data through four branches, extracting features of each modality, and forming a preliminary feature representation; Introducing a feature interaction layer, applying the self-attention mechanism to optimize feature relationships, and generating a comprehensive joint feature representation through the feature interaction mechanism; Based on the joint feature representation generated by the feature interaction layer, a context-aware fusion strategy is designed to dynamically adjust the feature fusion weights and optimize the fusion effect.

[0035] For feature extraction of multimodal data, by building a multi-input feature extraction model and using the self-attention mechanism and feature interaction layer to optimize the correlation and interaction between features of different modalities, the model's ability to understand and process various data types is effectively improved, making the features extracted from text, images, audio, and contextual data more accurate and relevant. The dynamic feature fusion strategy further enhances the ability to respond to real-time data changes, thereby achieving more accurate data analysis and classification in practical applications.

[0036] The extracting of each modality feature includes encoding the text data using a BERT model, extracting semantic features, and converting them into text feature vectors; Use the pre-trained ResNet model to encode the image data, extract the visual features of the image, and convert them into image feature vectors; Use MFCC technology to extract the spectral features of the audio, use convolutional neural network to encode the extracted audio features, and convert them into audio feature vectors; The context data is converted into numerical features through the embedding layer to generate a context feature vector, which is integrated with features of other modalities within the model.

[0037] By applying BERT, ResNet and MFCC technologies, and deeply processing text, image and audio data, the use of these advanced models has significantly improved the ability to extract deep features from complex data. Combined with the self-attention mechanism and feature interaction layer, the extraction efficiency of each modality feature is optimized, and the model's capture of subtle relationships between different data is enhanced, making the final feature representation more comprehensive and refined.

[0038] The generated joint feature representation includes weighting feature vectors extracted from text, image, audio and context data; By calculating the similarity between each feature vector and other feature vectors, the self-attention mechanism automatically adjusts and optimizes the weights and associations between different features, ensuring that the comprehensive feature vector can fully reflect the interaction of multimodal data and enhance the correlation between features; After the self-attention mechanism is used, the feature interaction layer fuses and interactively learns all weighted feature vectors to capture the potential correlation between modalities and combines and optimizes the features of each modality through the feature interaction mechanism. After the feature interaction layer is processed, a joint feature representation is generated to generate a more informative joint feature representation, thereby improving the overall feature expression ability and analysis effect; The optimization fusion strategy includes generating a dynamic weight adjustment joint feature representation based on the joint feature representation and according to the context information analyzed in real time; By training the optimized joint feature representation, using the patterns in historical data and the information from real-time feedback, the feature fusion strategy is further adjusted to generate the final optimized joint feature representation.

[0039] The generated joint feature representation is further processed through the self-attention mechanism, which strengthens the interaction and information fusion between features. By calculating the similarity between feature vectors and optimizing the association between them, this link not only inherits the detailed features of the previous model, but also improves the overall feature expression ability and classification accuracy through the composite operation of the feature interaction layer. The adjustment of dynamic weights enables the model to be flexibly adjusted according to contextual information, enhancing the system's adaptability to different situations.

[0040] The construction of the complaint content classification model includes using the final optimized joint feature representation as input to construct a multi-output deep learning model, wherein the underlying feature extraction layer of the model first shares the input joint feature representation and performs preliminary feature processing through a multi-layer neural network; Primary features refer to basic features extracted directly from the joint feature representation that have not been optimized for any specific analysis task. Primary features provide a general feature basis for further task-specific processing.

[0041] The model is divided into three parallel task branches, each branch is for a specific analysis task, including the main classification task branch, sentiment analysis task branch and topic identification task branch; The main classification task branch uses the features further processed from the primary feature layer to perform deep learning through a multi-layer fully connected neural network, and finally outputs the main categories of complaints; The sentiment analysis task branch starts from a shared primary feature layer and adds a dedicated neural layer to capture emotional features. The neural layer focuses on understanding the emotional tendency and finally outputs the emotional tendency of the complaint. The topic identification task branch starts from the shared primary feature layer, adds the text analysis layer to process the text data, identifies and classifies the key topics in the text, and finally outputs the specific topic of the complaint; During the model training phase, a comprehensive loss function is defined to simultaneously consider the task effects of main classification, sentiment analysis, and topic identification, and a multi-task learning strategy is used to optimize the branches. The weights and biases of the model are adjusted through the gradient descent algorithm, and the Adam optimization algorithm is used for dynamic learning rate adjustment; After each training cycle, an independent validation set is used to evaluate the performance of the model and the hyperparameters are adjusted based on the evaluation results; Adjustments include adjusting the learning rate, configuring the number of neural network layers, etc. to ensure that the model has good generalization capabilities on various tasks.

[0042] After the model training is completed, unseen datasets are used to evaluate the classification accuracy and robustness of the model on actual complaint data to ensure that the model can effectively perform classification tasks in actual environments.

[0043] The multi-output deep learning model uses the final optimized joint feature representation as input, effectively realizing multi-task learning. This model structure allows each task branch such as main classification, sentiment analysis and topic identification to share the primary feature layer and then optimize independently. This architecture greatly improves processing efficiency and accuracy. The use of multi-task learning strategy not only optimizes resource allocation, but also ensures the balanced performance of the model on each task through comprehensive loss function adjustment, significantly improving the classification accuracy and response speed in practical applications.

[0044] The real-time classification includes receiving new complaint data in real time after the model is deployed, preprocessing the data and extracting joint feature representations, inputting the data into the model for classification, and the model simultaneously outputting the main category, sentiment tendency and specific topic of the complaint based on the input features; Major categories include product quality, service feedback, and safety issues.

[0045] Sentimental tendencies include positive, negative, and neutral.

[0046] Specific topics include refund requests, product suggestion reports.

[0047] The extracted features are input into the trained deep learning model, which simultaneously outputs the main categories, sentiment tendencies, and specific topics of the complaints; The output results are then fed into a dynamic decision engine, which builds a decision model based on a decision tree and generates processing strategies for various scenarios; If the main category is a security issue, the emergency response mechanism is triggered, the automated monitoring and immediate reporting process is started, the system status is monitored in real time, and emergency notifications are automatically sent to relevant departments and management to ensure immediate processing; If the main category is product quality or service feedback, then enter the sentiment analysis process; For negative emotions, emotion recognition technology is introduced to further analyze the urgency and severity of complaints and adjust response priorities and resource allocation; For positive or neutral emotions, we conduct satisfaction prediction analysis to predict the customer's satisfaction with the current processing progress and then move on to the topic identification process; Based on the results of topic identification, for refund requests, trigger the financial process to predict potential refund risks based on customer historical data and purchasing patterns, and adjust inventory and pricing strategies accordingly; For the topic of product suggestions, we integrate customer feedback and generate reports for reference by the product development team. At the same time, we initiate innovation incentives to encourage employees to innovate based on customer suggestions.

[0048] The decision engine optimizes the decision path based on the above classification and historical data, and adjusts the strategy in real time to maximize processing efficiency and customer satisfaction.

[0049] By introducing an adaptive learning mechanism, the decision engine continuously learns from the results of each complaint handling and adjusts its decision algorithm to cope with changing customer needs and situational changes.

[0050] Based on the output of the decision engine, the corresponding back-end workflow is automatically triggered, such as customer return visits, refund processing, or product quality inspection.

[0051] The workflow achieves seamless execution of operations by integrating with the enterprise resource planning (ERP) system and customer relationship management (CRM) system, ensuring data consistency and operational efficiency.

[0052] Example 2 Reference Figure 2 , is an embodiment of the present invention, and provides a multimodal learning automated complaint content analysis and classification system, comprising: Data acquisition module, data fusion module, model training module and classification prediction module; The data collection module is used to collect multimodal complaint data and pre-process the collected multimodal complaint data; The data fusion module is used to extract the features of the pre-processed multi-modal complaint data and fuse the features of data of different modalities; The model training module is used to construct a complaint content classification model using the fused features, perform model training, and evaluate and optimize the trained model; The classification prediction module is used to deploy the optimized model for real-time classification.

[0053] Example 3 An embodiment of the present invention is different from the first two embodiments in that: If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions to enable a computer device (which can be a personal computer, server, or network device, etc.) to perform all or part of the steps of the methods described in various embodiments of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc., which can store program codes.

[0054] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as an ordered list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by an instruction execution system, device or apparatus (such as a computer-based system, a system including a processor, or other system that can fetch instructions from an instruction execution system, device or apparatus and execute instructions), or in conjunction with such instruction execution systems, devices or apparatuses. For the purposes of this specification, "computer-readable medium" can be any device that can contain, store, communicate, propagate or transmit a program for use by an instruction execution system, device or apparatus, or in conjunction with such instruction execution systems, devices or apparatuses.

[0055] More specific examples of computer-readable media (a non-exhaustive list) include the following: an electrical connection with one or more wires (electronic device), a portable computer disk case (magnetic device), a random access memory (RAM), a read-only memory (ROM), an erasable and programmable read-only memory (EPROM or flash memory), an optical fiber device, and a portable compact disk read-only memory (CDROM). In addition, the computer-readable medium may even be a paper or other suitable medium on which the program is printed, since the program may be obtained electronically, for example, by optically scanning the paper or other medium, followed by editing, deciphering or, if necessary, processing in another suitable manner, and then stored in a computer memory.

[0056] It should be understood that the various parts of the present invention can be implemented by hardware, software, firmware or a combination thereof. In the above-mentioned embodiments, multiple steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented by hardware, as in another embodiment, it can be implemented by any one of the following technologies known in the art or their combination: a discrete logic circuit having a logic gate circuit for implementing a logic function for a data signal, a dedicated integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.

[0057] Example 4 An embodiment of the present invention provides a multimodal learning automated complaint content analysis and classification method, comprising: Collect multimodal complaint data and pre-process the collected multimodal complaint data; Extract the features of pre-processed multimodal complaint data and fuse the features of data of different modalities; Use the fused features to build a complaint content classification model, perform model training, and evaluate and optimize the trained model; Deploy the optimized model for real-time classification.

[0058] Multimodal complaint data , containing text data , image data , audio data and contextual data .

[0059] For each data modality, preprocessing is performed. The preprocessing of text data is expressed as: , in, Represents the cleaned text data, clean It is a function used to remove noise such as HTML tags, URL links, and non-printing characters. The specific implementation is as follows: , Among them, RemoveHTML removes HTML tags and RemoveNoise removes noise.

[0060] The preprocessing of image data is expressed as: , in, Represents preprocessed image data, preprocess Is a function that includes operations such as image cropping, scaling, and normalization. The specific implementation is as follows: , Among them, Crop crops the image, Resize adjusts the image size, and Normalize normalizes it.

[0061] The preprocessing of audio data is expressed as: , in, represents the denoised audio data, is a function used to filter out background noise and echo. The specific implementation is as follows:

[0062] Among them, Filter filters out background noise.

[0063] The preprocessing of situational data is expressed as: , in, Represents the standardized situation data, normalize It is a function used to convert all timestamps to UTC format, standardize geographic location data to latitude and longitude format, etc. The specific implementation is as follows: , Among them, StandardizeTime unifies the time format, and StandardizeGeo standardizes the geographic location information.

[0064] Features of each data modality are extracted through a multi-input feature extraction model.

[0065] Text feature extraction uses the BERT model, expressed as: , in, represents the extracted text features, It is a function that uses the BERT model to encode text data and extract deep semantic features. The specific implementation is as follows: , Among them, Tokenize segments text data into words, and TransformerEncoder encodes the segmented data.

[0066] Image feature extraction, using the ResNet model, is expressed as: , here represents the extracted image features, is a function that uses the pre-trained ResNet model to encode image data and extract high-level visual features. The specific implementation is as follows: , Among them, ConvLayers represents the convolutional layer in the ResNet model.

[0067] Audio feature extraction, using MFCC and CNN, is expressed as: , in, represents the extracted audio features, It is a function that extracts the spectral features of audio; is a function that uses a convolutional neural network to encode the extracted audio features. The specific implementation is as follows: , Among them, MFCC extracts audio spectrum features, and ConvLayers represents the convolution layer.

[0068] Context feature extraction, expressed as: , in, Represents the extracted context features, Embedding is a function that converts contextual data into numerical features through the embedding layer. The specific implementation is as follows: , Among them, DenseLayer represents the fully connected layer.

[0069] The extracted features are optimized and fused using the self-attention mechanism and feature interaction layer.

[0070] The self-attention mechanism calculates the attention weight, which is expressed as: , in, Representation characteristics and The attention weights between It is a feature and The similarity score between The specific implementation is as follows: , in, and are the weight matrices for queries and keys, respectively, is the scaling factor.

[0071] Weighted fusion features are expressed as: , in, Represents the weighted eigenvector, by all eigenvectors Perform weighted summation to obtain .

[0072] The features of all modalities are fused to generate a joint feature representation, which is expressed as: , here Represents joint feature representation, and concat is a function that concatenates the feature vectors of all modalities.

[0073] A multi-output deep learning model is constructed using joint feature representation, which includes three task branches: main classification, sentiment analysis, and topic identification.

[0074] The main classification task branch is represented as: , in, Represents the output of the main classification task, Softmax is an activation function, Represents a fully connected layer, which is implemented as follows: , in, and are the weights and biases of the fully connected layer respectively.

[0075] The sentiment analysis task branch is represented as: , in, Represents the output of the sentiment analysis task. LSTM is a long short-term memory network used to capture sequence information. The specific implementation is as follows: , in, and are the weights and biases of the fully connected layer respectively.

[0076] The topic identification task branch is expressed as: , in, Represents the output of the topic identification task. Transformer is a model based on the attention mechanism for processing text data. The specific implementation is as follows:

[0077] in, and are the weights and biases of the fully connected layer respectively.

[0078] Define a comprehensive loss function to optimize the main classification, sentiment analysis, and topic identification tasks simultaneously: , in, represents the comprehensive loss function, and Represent the losses of main classification, sentiment analysis and topic identification respectively, and is the weight coefficient, which is implemented as follows: , The weights and biases of the model are optimized by the gradient descent algorithm, and the Adam optimization algorithm is used to dynamically adjust the learning rate, which is expressed as: , in, represents the parameters of the model, represents the learning rate, Represents the gradient of the loss function with respect to the parameters.

[0079] After the model is deployed, new complaint data is received in real time for classification, and a processing strategy is generated based on the classification results.

[0080] Example 5 As an embodiment of the present invention, a multimodal learning automated complaint content analysis and classification method is provided. In order to verify the beneficial effects of the present invention, a scientific demonstration is carried out through simulation experiments.

[0081] Using a multimodal complaint dataset containing text, images, and audio, each sample includes a customer complaint (text description), related images (such as product photos), and audio recordings (customer voice complaints). The dataset will be annotated to include major categories, sentiment tendencies, and specific topics.

[0082] All modal data are standardized, including word segmentation and denoising of text, resizing and standardization of images, and sampling rate unification and background noise reduction of audio.

[0083] The method of the present invention uses an optimized deep learning model, integrates a self-attention mechanism and a multi-task learning strategy, and automatically classifies and responds to complaints.

[0084] Traditional approaches use single-modality data processing and fixed rule-based decision-making systems to handle complaints.

[0085] The dataset was divided into a training set, a validation set, and a test set. The multimodal deep learning model of the present invention and the traditional text classification model were trained using the training set. The two methods were run on the test set, and the time it took for each method to process each complaint was recorded. The accuracy and user satisfaction of each method in complaint classification were evaluated, and the experimental results are shown in Table 1.

[0086] Table 1 Comparison of experimental results , The method of the present invention adopts the comprehensive utilization of multimodal data and the introduction of self-attention mechanism, so that the model can understand and classify complex complaint content at a deeper level. Such improvement greatly improves the classification accuracy. When processing complaints containing complex emotions and multi-dimensional information, the method of the present invention can more accurately capture key information, thereby making more appropriate classification judgments.

[0087] The response time is shortened through automated feature extraction and real-time data processing. The introduction of a dynamic decision engine further accelerates the decision-making process, and reduces the total time from receiving a complaint to responding by analyzing the contextual information of the complaint in real time and responding quickly. This improves processing efficiency and enhances customer satisfaction.

[0088] The improvement of user satisfaction proves the effectiveness of the method of the present invention in practical application. Through accurate classification and rapid response, coupled with personalized processing strategies designed for different complaint scenarios, efficient and high-quality customer service is ensured.

[0089] The method of the present invention first extracts rich features from text, images, audio and context data through a multi-input feature extraction model, uses the self-attention mechanism to significantly improve the recognition of associations between features, and then adjusts the feature weights through a dynamic fusion strategy to adapt to different real-time contexts. This process is more accurate and flexible than traditional static feature processing methods. Thanks to the highly optimized joint feature representation, the constructed multi-output deep learning model can simultaneously handle classification, sentiment analysis and topic identification, significantly improving the efficiency and accuracy of data processing.

[0090] In addition, the introduced dynamic decision-making engine can automatically generate processing strategies based on real-time classification results to achieve rapid response, especially in emergency situations, it can automatically trigger necessary operations, thereby greatly improving the timeliness and accuracy of complaint handling.

[0091] It has achieved full process automation from data collection to decision output, improved operational efficiency, reduced labor costs, and enhanced the ability to handle complex situations, enabling companies to manage and respond to user complaints more effectively, thereby improving customer satisfaction and the company's operational efficiency.

[0092] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solutions of the present invention may be modified or replaced by equivalents without departing from the spirit and scope of the technical solutions of the present invention, which should all be included in the scope of the claims of the present invention.

Claims

1. A multimodal learning automated complaint content analysis and classification method, characterized in that: include: Collect multimodal complaint data and pre-process the collected multimodal complaint data; Extract the features of pre-processed multimodal complaint data and fuse the features of data of different modalities; Use the fused features to build a complaint content classification model, perform model training, and evaluate and optimize the trained model; Deploy the optimized model for real-time classification.

2. The multimodal learning automated complaint content analysis and classification method according to claim 1, characterized in that: The multimodal complaint data includes complaint data and context data; The complaint data includes text data, image data and audio data; The preprocessing includes data cleaning, data standardization, format conversion and data integration.

3. The multimodal learning automated complaint content analysis and classification method according to claim 2, characterized in that: The extracting of features of the pre-processed multimodal complaint data includes constructing a multi-input feature extraction model, processing text, image, audio and context data through four branches, extracting features of each modality, and forming a preliminary feature representation; Introducing a feature interaction layer, applying the self-attention mechanism to optimize feature relationships, and generating a comprehensive joint feature representation through the feature interaction mechanism; Based on the joint feature representation generated by the feature interaction layer, a context-aware fusion strategy is designed to dynamically adjust the feature fusion weights and optimize the fusion effect.

4. The multimodal learning automated complaint content analysis and classification method according to claim 3, characterized in that: The extracting of each modality feature includes encoding the text data using a BERT model, extracting semantic features, and converting them into text feature vectors; Use the pre-trained ResNet model to encode the image data, extract the visual features of the image, and convert them into image feature vectors; Use MFCC technology to extract the spectral features of the audio, use convolutional neural network to encode the extracted audio features, and convert them into audio feature vectors; The context data is converted into numerical features through the embedding layer to generate a context feature vector, which is integrated with features of other modalities within the model.

5. The multimodal learning automated complaint content analysis and classification method according to claim 4, characterized in that: The generated joint feature representation includes weighting feature vectors extracted from text, image, audio and context data; By calculating the similarity between each feature vector and other feature vectors, the self-attention mechanism automatically adjusts and optimizes the weights and associations between different features; After the self-attention mechanism is used, the feature interaction layer fuses and interactively learns all weighted feature vectors to capture the potential correlation between modalities and combines and optimizes the features of each modality through the feature interaction mechanism. When the feature interaction layer is processed, a joint feature representation is generated; The optimization fusion strategy includes generating a dynamic weight adjustment joint feature representation based on the joint feature representation and according to the context information analyzed in real time; By training the optimized joint feature representation, using the patterns in historical data and the information from real-time feedback, the feature fusion strategy is further adjusted to generate the final optimized joint feature representation.

6. The multimodal learning automated complaint content analysis and classification method according to claim 5, characterized in that: The construction of the complaint content classification model includes using the final optimized joint feature representation as input to construct a multi-output deep learning model, wherein the underlying feature extraction layer of the model first shares the input joint feature representation and performs preliminary feature processing through a multi-layer neural network; The model is divided into three parallel task branches, including a main classification task branch, a sentiment analysis task branch, and a topic identification task branch; The main classification task branch uses the features further processed from the primary feature layer to perform deep learning through a multi-layer fully connected neural network, and finally outputs the main categories of complaints; The sentiment analysis task branch starts from a shared primary feature layer, adds a neural layer to capture emotional features, and the neural layer focuses on understanding the emotional tendency, and finally outputs the emotional tendency of the complaint; The topic identification task branch starts from the shared primary feature layer, adds the text analysis layer to process the text data, identifies and classifies the key topics in the text, and finally outputs the specific topic of the complaint; During the model training phase, a comprehensive loss function is defined to simultaneously consider the task effects of main classification, sentiment analysis, and topic identification, and a multi-task learning strategy is used to optimize the branches. The weights and biases of the model are adjusted through the gradient descent algorithm, and the Adam optimization algorithm is used for dynamic learning rate adjustment; After each training cycle, an independent validation set is used to evaluate the performance of the model, and the hyperparameters are adjusted based on the evaluation results.

7. The multimodal learning automated complaint content analysis and classification method according to claim 6, characterized in that: The real-time classification includes receiving new complaint data in real time after the model is deployed, preprocessing the data and extracting joint feature representations, inputting the data into the model for classification, and the model simultaneously outputting the main category, sentiment tendency and specific topic of the complaint based on the input features; The extracted features are input into the trained deep learning model, which simultaneously outputs the main categories, sentiment tendencies, and specific topics of the complaints; The output results are then fed into a dynamic decision engine, which builds a decision model based on a decision tree and generates processing strategies for various scenarios; If the main category is a security issue, the emergency response mechanism is triggered, the automated monitoring and immediate reporting process is started, the system status is monitored in real time, and emergency notifications are automatically sent to relevant departments and management to ensure immediate processing; If the main category is product quality or service feedback, then enter the sentiment analysis process; For negative emotions, emotion recognition technology is introduced to further analyze the urgency and severity of complaints and adjust response priorities and resource allocation; For positive or neutral emotions, we conduct satisfaction prediction analysis to predict the customer's satisfaction with the current processing progress and then move on to the topic identification process; Based on the results of topic identification, for refund requests, trigger the financial process to predict potential refund risks based on customer historical data and purchasing patterns, and adjust inventory and pricing strategies accordingly; For the topic of product suggestions, consolidate customer feedback and generate reports.

8. A system using the multimodal learning automated complaint content analysis and classification method as described in any one of claims 1 to 7, characterized in that: include: Data acquisition module, data fusion module, model training module and classification prediction module; The data collection module is used to collect multimodal complaint data and pre-process the collected multimodal complaint data; The data fusion module is used to extract the features of the pre-processed multi-modal complaint data and fuse the features of data of different modalities; The model training module is used to construct a complaint content classification model using the fused features, perform model training, and evaluate and optimize the trained model; The classification prediction module is used to deploy the optimized model for real-time classification.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, it implements the steps of the multimodal learning automated complaint content analysis and classification method described in any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the multimodal learning automated complaint content analysis and classification method described in any one of claims 1 to 7 are implemented.

Images