A multimodal data closed-loop management method and system based on government affairs

Abstract

The present invention provides a multimodal data closed-loop management method and system based on government affairs, which relates to the field of data processing technology. The method comprises: acquiring multimodal data from multiple information channels; performing modal classification, and extracting features of text information, voice information and image information respectively; performing feature conversion and normalization processing, and generating a cross-modal unified feature vector set; identifying cross-channel complaint information of the same event; matching with preset complaint category labels, and identifying complaint categories; determining responsible departments, and generating task assignment instructions; acquiring the processing status corresponding to the task assignment instructions, constructing task progress records, and generating early warning information when the task times out or the processing progress is abnormal; generating feedback information, and returning the feedback information through a communication channel corresponding to the complainant, and updating the processing status corresponding to the task assignment instructions at the same time. The present invention improves the autonomy and accuracy of multimodal data closed-loop management.

Description

Technical Field

[0001] The present invention relates to the field of data processing technology, and in particular to a multimodal data closed-loop management method and system based on government affairs. Background Art

[0002] In the existing technology, in government affairs, multimodal data management systems usually support the collection, processing and management of multimodal data by integrating multiple single data sources, such as text, images, sounds, etc. These systems use data mining, machine learning and other technologies to process data of different modalities and automate business processes. For example, government departments use automated systems to handle resident registration, complaint management and other businesses, which can efficiently collect and analyze data from different channels such as telephone, email, etc., and generate corresponding feedback.

[0003] However, in practical applications, existing technologies may have technical defects in the closed-loop management of multimodal data. Specifically, in the process of handling citizen complaints, some multimodal data management systems lack effective data integration and real-time feedback mechanisms. When receiving complaints from multiple information channels, such as social media, online complaint platforms, and government service hotlines, the system fails to conduct a comprehensive analysis in a timely manner and return them to the staff for further processing. For example, if a complaint involves multiple issues such as service quality and personnel attitude, the existing system may assign them to different processing departments separately, resulting in fragmented problem handling and duplication of work. Summary of the invention

[0004] The purpose of the present invention is to provide a multimodal data closed-loop management method and system based on government affairs, aiming to solve the problems mentioned in the background technology.

[0005] In order to solve the above technical problems, the technical solution of the present invention is as follows:

[0006] In a first aspect, a multimodal data closed-loop management method based on government affairs services, the method comprising:

[0007] Obtain multimodal data from multiple information channels and parse it into structured data;

[0008] Perform modality classification on structured data, and extract features of text information, voice information, and image information respectively to generate a modality feature set;

[0009] According to the modal feature set, feature conversion and normalization are performed to generate a cross-modal unified feature vector set;

[0010] Based on the cross-modal unified feature vector set, data matching is performed according to the similarity of the complaint content, cross-channel complaint information of the same event is identified, and merged to generate fusion analysis data, including the associated complaint number, timestamp, complainant information, and problem description;

[0011] Based on the fusion analysis data, the preset complaint category labels are used for matching, the complaint categories are identified, and the classification results are generated;

[0012] Determine the responsible department according to the classification result and generate a task assignment instruction, which includes the target responsible department, processing priority, recommended processing time limit and associated complaint number;

[0013] Obtain the processing status corresponding to the task assignment instruction and build a task progress record, which includes the processing progress reported by the responsible department, the complaint processing time and the corresponding interim results, and generate warning information when the task times out or the processing progress is abnormal;

[0014] Generate feedback information based on the final processing results of the responsible department, return the feedback information through the corresponding communication channel to the complainant, and update the processing status corresponding to the task assignment instruction;

[0015] Based on historical task progress records and historical feedback information, optimization suggestions are generated, and complaint category labels and task allocation strategies are adjusted.

[0016] Preferably, the modality classification of the structured data and the extraction of features of text information, voice information and image information to generate a modality feature set include:

[0017] According to the data type in the structured data, the data is divided into text data, voice data and image data;

[0018] Perform word segmentation, syntactic analysis and sentiment recognition on text data, extract topic keywords, syntactic structure and sentiment features, and generate text feature data;

[0019] Convert the speech data into audio signals, convert the speech data into text data, extract intonation features, speech speed features and emotion features, and generate speech feature data;

[0020] Perform target detection and optical character recognition on image data, extract facial features, scene elements, text content and object features, and generate image feature data;

[0021] The text feature data, speech feature data and image feature data are combined to generate a modal feature set.

[0022] Preferably, the step of performing feature conversion and normalization processing according to the modal feature set to generate a cross-modal unified feature vector set includes:

[0023] According to the modal feature set, the text feature data is converted into a semantic vector of fixed dimension to generate a text vector dataset;

[0024] According to the modal feature set, the speech feature data is converted into a text vector and encoded with the emotional features to generate a speech vector dataset;

[0025] According to the modal feature set, the image features are converted into vectors of fixed dimensions and combined with the extracted text content to generate an image vector dataset;

[0026] According to the text vector dataset, speech vector dataset and image vector dataset, a normalization method is adopted to convert all modal data into feature vectors of the same dimension to generate a cross-modal unified feature vector set.

[0027] Preferably, the method of performing data matching according to the similarity of complaint contents based on the cross-modal unified feature vector set, identifying cross-channel complaint information of the same event, and merging to generate fusion analysis data includes:

[0028] Based on the cross-modal unified feature vector set, the similarities of the text vector, the speech vector and the image vector are calculated respectively, and the text similarity matrix, the speech similarity matrix and the image similarity matrix are constructed;

[0029] According to the preset modal weights, the text similarity matrix, the speech similarity matrix and the image similarity matrix are fused to construct a multimodal data association matrix;

[0030] Set a similarity matching threshold, and based on the multimodal data association matrix, identify cross-channel complaint information for the same event and generate a unified event number;

[0031] Based on the unified event number, complaint information from different information channels is merged, and the timestamp, complainant information and problem description are integrated to remove redundant complaint content, extract core demands, and generate integrated analysis data.

[0032] Preferably, the fusion analysis data is matched with preset complaint category labels to identify complaint categories and generate classification results, including:

[0033] According to the text content in the fusion analysis data, the matching degree between the text vector and the preset complaint category label is calculated, and the text category matching result is generated;

[0034] According to the speech content in the fusion analysis data, the matching degree between the speech vector and the preset complaint category label is calculated, and the speech category matching result is generated;

[0035] According to the image content in the fusion analysis data, the matching degree between the image vector and the preset complaint category label is calculated, and the image category matching result is generated;

[0036] According to the text category matching results, voice category matching results and image category matching results, a modal fusion strategy is adopted to calculate the classification matching score, generate the classification result, and associate it with the corresponding complaint category label; the calculation formula of the classification matching score is: ,in, is the classification matching score, which is used to determine the category to which the complaint belongs. is the modal type, Represents text, Indicates voice, Represents an image, For modal The characteristic vector of is the modal vector of complaint category labels, is the modality fusion weight, and is the normalization parameter, is the norm of the mode vector, is the absolute difference between the modal vector and the class label vector.

[0037] Preferably, the responsible department is determined according to the classification result, and a task assignment instruction is generated, wherein the task assignment instruction includes the target responsible department, the processing priority, the recommended processing time limit, and the associated complaint number, including:

[0038] According to the complaint categories in the classification results, query the responsible department database, determine the responsible department corresponding to the complaint category, and generate preliminary task allocation information;

[0039] Calculate the urgency of tasks and set processing priorities based on the current task load, historical processing efficiency and average response time of the responsible department;

[0040] The calculation formula of the task urgency is: ,in, For the task urgency, A collection of available responsible departments. For a specific responsible department, Responsible department The current backlog of tasks, Responsible department The historical average processing efficiency, The impact scope of the complaint, ranging from 0 to 1. is the adjustment coefficient, The time when the complaint was filed, Responsible department The average response time, and is the normalization parameter, is the number of currently available responsible departments;

[0041] in, , is the set of complaint channels involved in this complaint. For complaint channels, is the channel weight coefficient, For the channel The number of complaints forwarded on For this complaint in the channel The first exposure time on is the base time, used for normalization, and is the normalization parameter, For this complaint in the channel Number of user interactions on is the number of historical occurrences of the complaint. is the number of persons involved in the complaint, is the base number, used for normalization, The number of complaint channels involved in the complaint;

[0042] The calculation formula of the processing priority is: ,in, is the task priority, ranging from 0 to 1, For the task urgency, is the magnification factor, is the smoothing coefficient, The trigger threshold for task priority ensures that low-priority tasks are not processed prematurely;

[0043] Set a recommended time limit for handling the task based on the urgency of the task, the scope of the complaint and the feedback from the complainant;

[0044] Generate task assignment instructions, including responsible department, processing priority, recommended processing time limit and associated complaint number, and push the task assignment instructions to the responsible department's work system.

[0045] Preferably, the processing status corresponding to the task assignment instruction is obtained, and a task progress record is constructed. The task progress record includes the processing progress fed back by the responsible department, the complaint processing time and the corresponding interim results, and generates warning information when the task times out or the processing progress is abnormal, including:

[0046] According to the target responsible department in the task assignment instruction, query the task execution status of the responsible department and obtain task progress data in real time;

[0047] According to the task progress data, record the processing progress, complaint handling time and corresponding interim results reported by the responsible department, and build a task progress record;

[0048] According to the task progress record, when the task exceeds the time limit or the processing progress is abnormal, an early warning message is generated and sent to the responsible department and the superior supervisory department.

[0049] In a second aspect, a multimodal data closed-loop management system based on government affairs services comprises:

[0050] Multimodal data acquisition module, used to obtain multimodal data from multiple information channels and parse it into structured data;

[0051] The feature extraction module is used to perform modality classification on structured data and extract features of text information, voice information and image information respectively to generate a modality feature set;

[0052] The cross-modal data unification module is used to perform feature conversion and normalization processing according to the modal feature set to generate a cross-modal unified feature vector set;

[0053] The cross-modal data matching module is used to match data according to the similarity of complaint content based on the cross-modal unified feature vector set, identify cross-channel complaint information of the same event, and merge and generate fusion analysis data, including the associated complaint number, timestamp, complainant information, and problem description;

[0054] The complaint classification module is used to match the fusion analysis data with the preset complaint category labels, identify the complaint category, and generate classification results;

[0055] A task assignment module, used to determine the responsible department according to the classification results and generate a task assignment instruction, wherein the task assignment instruction includes the target responsible department, processing priority, recommended processing time limit and associated complaint number;

[0056] The task progress management module is used to obtain the processing status corresponding to the task assignment instruction and build the task progress record, which includes the processing progress feedback from the responsible department, the complaint processing time and the corresponding interim results, and generates warning information when the task times out or the processing progress is abnormal;

[0057] The feedback and optimization module is used to generate feedback information based on the final processing results of the responsible department, return the feedback information through the corresponding communication channel to the complainant, and update the processing status corresponding to the task assignment instruction;

[0058] The information optimization module is used to generate optimization suggestions and adjust complaint category labels and task allocation strategies based on historical task progress records and historical feedback information.

[0059] The above solution of the present invention includes at least the following beneficial effects:

[0060] First, this method obtains multimodal data from multiple information channels and parses it into structured data to ensure that data from different sources can be uniformly stored and analyzed. Compared with the existing technology that relies on independent data sources for decentralized processing, this method can integrate data from different sources such as social media, online complaint platforms, government service hotlines, etc., to avoid loss or processing deviations due to inconsistent data formats. For example, in the prior art, voice complaints usually rely on manual recording of text information, while this method uses automatic speech-to-text technology to standardize voice data into structured text data, so that it can be processed together with text and image information, improving data integrity and consistency.

[0061] Secondly, this method performs modal classification on structured data, and extracts the features of text information, voice information, and image information respectively to form a modal feature set, ensuring that data of different modalities are comparable during the calculation process. In the prior art, data of multiple modalities are usually stored and analyzed independently, lacking a cross-modal fusion mechanism, which makes it impossible for the system to comprehensively analyze the content of the complaint. For example, a complaint contains a text description of "park bench damaged" and a picture of the damaged bench. In traditional systems, this information may be processed separately, while this method can extract text keywords, analyze image visual features, and fuse this information to ensure the integrity of the complaint content and make classification and task allocation more accurate.

[0062] In addition, this method maps data of different modalities to a unified feature space through feature conversion and normalization processing, forming a cross-modal unified feature vector set. This mechanism effectively solves the computational difficulties caused by the different dimensions of data of different modalities in the prior art. For example, text data is usually represented in the form of word vectors, while image data is high-dimensional pixel features. This method uses normalization technology to calculate the similarity of all data in the same dimensional space, thereby ensuring that complaint data from different sources can be matched under the same standard and improving the fusion of cross-modal data.

[0063] In the data matching stage, this method calculates the similarity of complaint content based on a cross-modal unified feature vector set, and performs data matching to identify cross-channel complaint information for the same event, and merges and generates fusion analysis data. Compared with the prior art, due to the lack of an effective association mechanism for complaints from different sources, complaints about the same event are repeatedly recorded and processed. This method can accurately identify similar complaints and avoid information fragmentation. For example, if multiple users complain about the damage of the same park facilities through telephone, online platforms, and social media, this method can automatically identify the similarities of these complaints and merge them for processing, reducing duplicate task allocation and improving the overall efficiency of problem handling.

[0064] During the complaint classification stage, this method uses preset complaint category labels for matching to ensure that complaint information can be accurately classified into the corresponding business category, thereby improving the accuracy of subsequent task assignment. Compared with existing technologies that rely on keyword matching or manual review, this method combines text semantic analysis, speech emotion recognition, image content analysis and other technologies to automatically identify the core issues of the complaint. For example, in existing systems, similar "fallen trees in the park" and "fallen trees on the road" may be mistakenly classified into the same category, but this method can distinguish the two into "landscape maintenance" and "municipal road management" by combining image features and location information, thereby improving the accuracy of complaint classification.

[0065] In the task assignment stage, this method automatically determines the responsible department based on the classification results and generates task assignment instructions to ensure that the complaint task can be quickly transferred to the correct processing unit. Compared with the prior art, some complaints are split and assigned to different departments because they involve multiple issues, resulting in duplicate work and increased communication costs. This method can accurately identify the main responsible departments involved in the complaint through cross-modal analysis, and intelligently adjust the allocation strategy to reduce unnecessary inter-departmental transfers. For example, a complaint involves two issues, "damaged trash cans" and "garbage cleaning". In the traditional system, they may be assigned to the municipal facilities management department and the environmental sanitation management department respectively. This method can analyze historical processing records and determine that such complaints can be handled uniformly by the municipal facilities management department, thereby improving the efficiency of task execution.

[0066] In the task tracking and feedback stage, this method can obtain the task status in real time, build a task progress record, and generate warning information when the task times out or is abnormal, ensuring that the complaint task can be completed on time. Existing technologies usually rely on regular manual inspections of task progress, resulting in some complaints being discovered after a long period of time without being processed. This method can monitor the execution of tasks in real time, push reminders to the responsible department in a timely manner, and even dynamically adjust the task processing strategy based on the progress of the task. For example, if a complaint task is delayed due to a shortage of personnel, the system can automatically suggest adjusting the processing priority or transferring it to other departments to ensure that the complaint can be resolved as soon as possible.

[0067] Finally, this method continuously optimizes complaint classification and task allocation strategies through historical data analysis, so that the system has adaptive learning capabilities and improves the efficiency of future complaint management. For example, by analyzing past complaint data, the system can identify which categories of complaints are prone to misclassification and automatically adjust the classification rules to improve the accuracy of future complaint classification. At the same time, the system can also analyze the task completion efficiency of different responsible departments and optimize the task allocation strategy accordingly. For example, urgent complaints are preferentially assigned to departments with higher historical processing efficiency to ensure that important complaints can be resolved first.

[0068] In summary, compared with the existing technology, this method can more efficiently integrate multimodal complaint data, improve the accuracy of complaint classification, optimize the task allocation process, and realize real-time monitoring and feedback, ultimately improving the intelligence level of government complaint management, reducing duplication of work, improving government response speed, and enhancing citizen satisfaction. BRIEF DESCRIPTION OF THE DRAWINGS

[0069] Figure 1 It is a flowchart of a multimodal data closed-loop management method based on government affairs provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0070] The exemplary embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although the exemplary embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be limited by the embodiments set forth herein. On the contrary, these embodiments are provided to enable a more thorough understanding of the present disclosure and to fully convey the scope of the present disclosure to those skilled in the art.

[0071] like Figure 1 As shown, an embodiment of the present invention proposes a multimodal data closed-loop management method based on government affairs, the method comprising:

[0072] Obtain multimodal data from multiple information channels and parse it into structured data;

[0073] Perform modality classification on structured data, and extract features of text information, voice information, and image information respectively to generate a modality feature set;

[0074] According to the modal feature set, feature conversion and normalization are performed to generate a cross-modal unified feature vector set;

[0075] Based on the cross-modal unified feature vector set, data matching is performed according to the similarity of the complaint content, cross-channel complaint information of the same event is identified, and merged to generate fusion analysis data, including the associated complaint number, timestamp, complainant information, and problem description;

[0076] Based on the fusion analysis data, the preset complaint category labels are used for matching, the complaint categories are identified, and the classification results are generated;

[0077] Determine the responsible department according to the classification result and generate a task assignment instruction, which includes the target responsible department, processing priority, recommended processing time limit and associated complaint number;

[0078] Obtain the processing status corresponding to the task assignment instruction and build a task progress record, which includes the processing progress reported by the responsible department, the complaint processing time and the corresponding interim results, and generate warning information when the task times out or the processing progress is abnormal;

[0079] Generate feedback information based on the final processing results of the responsible department, return the feedback information through the corresponding communication channel to the complainant, and update the processing status corresponding to the task assignment instruction;

[0080] Based on historical task progress records and historical feedback information, optimization suggestions are generated, and complaint category labels and task allocation strategies are adjusted.

[0081] In an embodiment of the present invention, during the data collection phase, the system can obtain multimodal data from multiple information channels and parse it into structured data to ensure that the complaint information can be efficiently processed by the machine. Since the data formats of different channels are different, for example, voice complaints need to be converted through voice recognition, and image complaints may contain OCR extracted text, the data structuring process is crucial. Through this processing method, information loss or processing deviation caused by inconsistent data formats can be avoided, making subsequent classification and analysis more accurate.

[0082] In the data classification stage, the features of text information, voice information, and image information can be extracted separately through modal classification methods to form a modal feature set. In this way, each type of data can be standardized to ensure compatibility between cross-modal data. At the same time, through feature conversion and normalization processing, data of different modalities can be converted into a vector set of the same dimension, ensuring the stability of subsequent similarity calculation and data fusion.

[0083] In the data matching stage, a similarity calculation method is used to compare the complaint content based on a cross-modal unified feature vector set, identify cross-channel complaint information for the same event, and merge them for processing. This can avoid repeated processing of the same event, reduce the waste of government resources, and ensure that all relevant information can be fully integrated to improve the comprehensiveness of task processing. For example, a citizen may upload a photo through social media to complain about infrastructure damage, and at the same time call the government hotline to make a voice complaint. Through similarity calculation, these complaints can be identified as belonging to the same event and automatically merged to avoid repeated acceptance by multiple departments.

[0084] In the classification stage, the preset complaint category label matching method is used to automatically identify the complaint category and generate the classification results. The traditional manual review classification method usually relies on manual judgment and is inefficient. Through this method, multimodal information can be combined for automatic classification to ensure the accuracy of classification. Text, voice and image information can all be used to match the most appropriate complaint category, making the classification results more accurate. For example, complaints involving road problems will automatically be classified into the "traffic management" category, while complaints related to damage to municipal facilities will be classified into the "infrastructure maintenance" category.

[0085] In the task allocation stage, the responsible department is determined based on the classification results, and task allocation instructions are automatically generated, so that tasks can be quickly transferred to the correct responsible department, reducing processing delays caused by human intervention. Task allocation instructions not only include the target responsible department, but also include processing priority, recommended processing time limit, etc., making task scheduling more efficient.

[0086] During the task tracking phase, the system can obtain the task processing status in real time, build a task progress record, and generate warning information when the task times out or is abnormal. Through this function, government managers can grasp the execution of tasks in real time to ensure that problems are handled in a timely manner. Compared with traditional task tracking methods, this method can proactively warn and avoid complaints that have not been handled for a long time, leading to dissatisfaction among citizens.

[0087] In the feedback stage, the system can generate feedback information based on the final processing results and return the processing results through the original communication channel of the complainant. At the same time, the task processing status will be automatically updated to ensure the integrity of the task life cycle. In this way, citizens can obtain feedback on complaint processing in a timely manner, which improves the public's satisfaction with government affairs.

[0088] In addition, through the accumulation of historical data, the system can optimize and analyze based on historical task progress records and feedback information, and automatically adjust complaint category labels and task allocation strategies. Through continuous optimization, the system can adapt to the needs of different government environments, continuously improve processing efficiency, and reduce task processing time. Compared with the traditional static classification and task allocation model, this method has stronger adaptive capabilities, making government business management more intelligent and automated.

[0089] In one case of this embodiment, the acquiring of multimodal data from multiple information channels and parsing into structured data includes:

[0090] To ensure that complaint information from different sources can be processed uniformly, this method uses automated data collection and analysis technology:

[0091] Text data: Extract raw text content directly from online complaint platforms, social media, emails, etc. If there are formatting issues, such as emoticons and non-standard characters, the system can format them through text cleaning techniques, such as regular expression matching or natural language processing (NLP) preprocessing, so that the text information can be parsed uniformly.

[0092] Voice data: The audio data collected from telephone complaints is first converted into text data through automatic speech recognition (ASR) technology, and relevant emotional features such as tone and speed are extracted to assist in judging the urgency of the complaint. For example, if the complainant is excited and speaks quickly, the system can mark the complaint as high urgency.

[0093] Image data: For images uploaded via social media or mobile applications, the system can use computer vision technology to analyze them. For example, object detection and optical character recognition (OCR) can be used to identify key information in the image. For example, a photo of a damaged street can be used to identify "potholes" through object detection, and street signs in the photo can be extracted through OCR for subsequent positioning and classification.

[0094] Video data: If the complaint information contains videos, such as short videos on social media platforms, the system can first extract key frames through frame capture, and then parse them with OCR and object detection technology to extract key content.

[0095] The parsed data needs to be stored in a unified data format for subsequent analysis and processing. Structured data can be stored in relational databases such as MySQL, PostgreSQL, or non-relational databases such as MongoDB, Elasticsearch. The data fields usually include:

[0096] Complainant information, including name, contact information, complaint source channel; complaint time, geographic location (if applicable); complaint content, including text, speech-to-text, and OCR-extracted text; complaint attachments, including pictures, video frames, and related metadata; urgency and classification labels, which can be predicted in the preprocessing stage.

[0097] Through the above method, it can be ensured that multimodal data from different sources can be converted into structured information, which is convenient for subsequent classification, analysis and task allocation.

[0098] In one case of this embodiment, the feedback information is generated according to the final processing result of the responsible department, and the feedback information is returned through the communication channel corresponding to the complainant, and the processing status corresponding to the task assignment instruction is updated, including:

[0099] After the responsible department handles the complaint, it needs to feedback the handling results to the system. This method supports multiple feedback methods, including:

[0100] Manual input feedback: The staff manually fills in the processing results in the government affairs management system, such as "repair completed" or "not processed yet, reason: coordination with other departments is required."

[0101] Automatic feedback: For complaints involving physical processing results, such as infrastructure maintenance, the system can automatically obtain the progress of task execution through IoT devices and remote monitoring systems. For example, if the task involves street light maintenance, the system can use smart sensors to detect whether the lights are working properly and automatically update the task status to "maintenance completed."

[0102] Confirmation and feedback from responsible departments: The system regularly sends reminders to the responsible departments, asking them to confirm whether the tasks have been completed, and triggers an early warning if no feedback is given within the time limit.

[0103] In order to allow complainants to understand the progress of issue handling in a timely manner, the system will automatically generate feedback information based on the handling results and send it through the original communication channel used by the complainant.

[0104] The processing status of each complaint will be updated in real time in the task management system to reflect the progress of the task. For example: "Processing" → "Completed"; "Processing" → "Awaiting Review"; "Processing" → "Need Coordination";

[0105] The update of task status ensures that complaints will not be processed repeatedly, and avoids complainants mistakenly believing that the problem has not been solved and submitting complaints repeatedly due to the responsible department's failure to update information in a timely manner. In addition, if the progress of task execution is abnormal, such as not being completed for a long time, the system can automatically trigger an abnormal alarm and notify the superior management department to intervene.

[0106] In one case of this embodiment, generating optimization suggestions based on historical task progress records and historical feedback information, and adjusting complaint category labels and task allocation strategies include:

[0107] In order to improve the overall efficiency of complaint management, this method can analyze historical data and optimize task allocation and classification rules to ensure that the system can continuously improve its degree of automation.

[0108] As complaint data accumulates, the system can analyze historical complaint records and optimize the classification model. For example:

[0109] Count the misclassification rates of complaint categories and identify which complaint categories are often misclassified, such as "road construction complaints" being misclassified as "noise complaints."

[0110] Through semantic analysis techniques, such as LDA topic modeling and BERT pre-training models, new complaint categories can be automatically discovered. For example, "intelligent traffic signal failure" may be an emerging complaint category, and the system's category labels can be adjusted accordingly.

[0111] The rationality of task allocation affects the efficiency of complaint handling. The system can analyze historical data:

[0112] Task processing time of different responsible departments: For example, one department takes an average of 7 days to process a road repair task, while another department only takes 3 days. The system can automatically adjust the allocation strategy to prioritize urgent tasks to the department with faster processing speed.

[0113] Priority optimization for different complaint types: For example, if a certain type of complaint, such as "traffic light failure", has a greater impact on citizens during peak hours, the system can increase the default priority of this type of complaint to ensure that it is handled more quickly.

[0114] Intelligent task load balancing: Analyze the task backlog of each department and automatically adjust task allocation to ensure that no department will be delayed due to excessive task volume.

[0115] Through these optimization mechanisms, the system can continuously improve the accuracy of complaint classification and task allocation, making government management more efficient and intelligent, ensuring that complaints can be resolved more quickly and reasonably, and improving citizens' satisfaction with government services.

[0116] In a preferred embodiment of the present invention, the modality classification of structured data and the extraction of features of text information, voice information and image information to generate a modality feature set include:

[0117] According to the data type in the structured data, the data is divided into text data, voice data and image data;

[0118] Perform word segmentation, syntactic analysis and sentiment recognition on text data, extract topic keywords, syntactic structure and sentiment features, and generate text feature data;

[0119] Convert the speech data into audio signals, convert the speech data into text data, extract intonation features, speech speed features and emotion features, and generate speech feature data;

[0120] Perform target detection and optical character recognition on image data, extract facial features, scene elements, text content and object features, and generate image feature data;

[0121] The text feature data, speech feature data and image feature data are combined to generate a modal feature set.

[0122] In the embodiment of the present invention, when processing government affairs complaint information, the classification and feature extraction of multimodal data are important basic links in data processing. Different modal data have different characteristics. For example, text data is structured information, voice data needs to be converted into text for analysis, and image data relies on visual feature recognition. Therefore, through modal classification and feature extraction, an accurate data foundation can be provided for subsequent analysis and task allocation.

[0123] First, modal classification of structured data can effectively distinguish text data, voice data, and image data, ensuring that different modal data can be processed in a targeted manner. For example, social media complaints may contain text descriptions and accompanying pictures, while hotline complaints exist in the form of voice data. Through modal classification, it can be ensured that the most appropriate processing method is used for each data type.

[0124] In the text feature extraction stage, word segmentation, syntactic analysis and sentiment recognition methods are used to extract the subject keywords, grammatical structure and sentiment tendency in the text. For example, for the complaint content "the road is seriously potholed and has affected traffic", the system can extract the keywords "road potholes" and "affect traffic" and identify the complaint sentiment as negative, thereby helping the classification system to more accurately identify the complaint category.

[0125] In the speech data processing stage, the speech is first converted into structured text data through speech-to-text technology, and the intonation features, speech speed features and emotional features are extracted. The intonation features can reflect the emotional intensity of the complainant. For example, a high-pitched tone may mean that the complainant is emotionally excited, and a fast speech speed may indicate a high degree of urgency. This information can be combined with text data to further optimize the classification and task allocation strategies.

[0126] In the image data processing stage, by adopting object detection and optical character recognition technologies, key elements in the image can be extracted. For example, if the complainant uploads a photo of a damaged road surface, the system can identify the damaged area of the road surface through object detection and, combined with the text information extracted by OCR, confirm that the complaint belongs to the road maintenance problem. In this way, the image data can not only serve as supplementary information but also improve the accuracy of complaint category matching.

[0127] Finally, the text feature data, voice feature data, and image feature data are merged to form a modal feature set, ensuring that all modal data can be calculated and analyzed in the same system. In this way, the information isolation between modalities can be reduced, the cross-modal data fusion ability can be improved, and the complaint handling can be made more intelligent.

[0128] In one case of this embodiment, the tokenization, syntactic analysis, and sentiment recognition of the text data, and the extraction of topic keywords, syntactic structures, and sentiment features to generate text feature data include:

[0129] Since natural language text is usually composed of multiple words and computers cannot directly understand continuous strings, it is necessary to tokenize the text. Tokenization is to split the continuous text into meaningful words and remove stop words, such as words that do not affect the semantics like "de", "le", "shi", etc.

[0130] Tokenization for English text: Words can be extracted based on space splitting or regular expression matching. For example, "Thestreetlightisbroken" will be split into [The, streetlight, is, broken].

[0131] Tokenization for Chinese text: Since there are no spaces in Chinese, it is necessary to use dictionary matching algorithms such as HanLP, Jieba, or neural network tokenization based on deep learning such as the BERT tokenizer for tokenization. For example, "路灯坏了" can be split into [路灯, 坏了], where "路灯" is an important keyword.

[0132] Syntactic analysis is used to identify the subject-predicate-object relationship, modifying components, and sentence structure in the text in order to extract important semantic information. Common methods include:

[0133] Method based on sentiment dictionary: For example, if the text contains words such as "dissatisfied", "seriously affect", "urgently need to be processed", then the complaint may carry negative emotions.

[0134] Machine learning-based methods: BERT, LSTM, or CNN can be used to train sentiment classification models. After inputting the complaint text, the sentiment category is output. For example: "The trash can is full, no one cleans it, and it smells bad!" → negative sentiment; "The repair was very fast and the problem was solved, thank you!" → positive sentiment; "When will this road be repaired?" → neutral sentiment;

[0135] In one case of this embodiment, the voice data is converted into audio signals, the voice data is converted into text data, and intonation features, speech speed features and emotion features are extracted to generate voice feature data, including:

[0136] The voice data first needs to go through automatic speech recognition to convert the audio into text.

[0137] Speech signal preprocessing: noise reduction and speech signal framing.

[0138] Speech-to-text: Traditional ASR based on Hidden Markov Model-Gaussian Mixture Model, such as Kaldi. End-to-end ASR based on deep learning, such as Wav2Vec and DeepSpeech, can improve recognition accuracy, especially in noisy environments.

[0139] For example: original speech: "The community's trash can is full, when will it be cleaned?"; speech recognition result: "The community's trash can is full, when will it be cleaned?"; Intonation features can be used to determine the tone of the complaint and distinguish whether it is an urgent complaint. For example: high-pitched and loud voice: it may indicate that the complainant is emotional and needs to increase the urgency. Low-pitched and slow voice: it may indicate a general consultation complaint. Mel-frequency cepstral coefficients and short-time Fourier transform can be used to extract pitch and volume information.

[0140] The speech speed feature can reflect the urgency of the complaint: Fast speech speed (>200wpm): The complainant is anxious, which may indicate an urgent complaint. Slow speech speed (<120wpm): It may be a general complaint or consultation.

[0141] Through sentiment classification models such as the sentiment recognition network combined with CNNRNN, the emotions of the complainant can be judged, such as: anger (high volume + high speaking speed): "Why haven't you fixed it yet?"; anxiety (low volume + high speaking speed): "The street light has been broken for several days. It is very dangerous at night."; calm (normal speaking speed): "I hope the garbage can be cleaned up."; Finally, after the voice data is processed, text data can be generated with intonation, speaking speed, and emotional features for subsequent classification.

[0142] In one case of this embodiment, target detection and optical character recognition are performed on image data to extract facial features, scene elements, text content and object features to generate image feature data, including:

[0143] Object detection is used to identify key elements in an image. For example, YOLO, FasterRCNN, and SSD can be used to identify traffic signs, trash cans, and damaged public facilities.

[0144] Example:

[0145] Input: A picture of a broken park bench.

[0146] Test result: [Item: Bench, Status: Damaged], which can be matched to the "Damage to public facilities" complaint category.

[0147] OCR can be used to extract text information from images, such as:

[0148] Road signs, store signs, equipment labels.

[0149] Ultimately, the feature data after image processing can be used to assist in classification, complaint category optimization, and task scheduling, thereby improving the intelligence level of complaint management.

[0150] In a preferred embodiment of the present invention, the step of performing feature conversion and normalization processing according to the modal feature set to generate a cross-modal unified feature vector set includes:

[0151] According to the modal feature set, the text feature data is converted into a semantic vector of fixed dimension to generate a text vector dataset;

[0152] According to the modal feature set, the speech feature data is converted into a text vector and encoded with the emotional features to generate a speech vector dataset;

[0153] According to the modal feature set, the image features are converted into vectors of fixed dimensions, and combined with the text content extracted by OCR to generate an image vector dataset;

[0154] According to the text vector dataset, speech vector dataset and image vector dataset, a normalization method is adopted to convert all modal data into feature vectors of the same dimension to generate a cross-modal unified feature vector set.

[0155] In the embodiment of the present invention, in multimodal data processing, the feature dimensions of text, voice and image data are often different. Direct similarity calculation may lead to data scale mismatch, affecting the accuracy of the calculation results. Therefore, through feature conversion and normalization processing, data of different modalities can be converted into feature vectors of the same dimension, so that they can be analyzed in the same calculation space.

[0156] In the process of text feature conversion, the text embedding method is used to convert the text feature data into a fixed-dimensional semantic vector, so that texts of different lengths can be mapped to a unified vector space. For example, "the road is seriously damaged" may be converted into a fixed-dimensional vector, so that it can be used for similarity calculation with other text information.

[0157] In the process of voice data conversion, text information is first obtained through the voice-to-text method, and then encoded in combination with emotional features. In this way, voice data can not only provide text information, but also retain emotional information to improve matching accuracy. For example, the tone of "urgent processing" in voice complaints is urgent, and the system can give it a higher weight to ensure that urgent complaints are handled first.

[0158] In the process of image data conversion, deep learning methods are used to extract visual features, and combined with the text information recognized by OCR, they are stored in the image vector dataset. In this way, even non-text data can be mapped to a unified vector space, which is convenient for subsequent analysis.

[0159] Finally, through the normalization method, all modal data are converted into feature vectors of the same dimension, forming a unified cross-modal feature vector set, ensuring that all data can be efficiently matched and improving the accuracy and consistency of complaint handling.

[0160] In one case of this embodiment, the text feature data is converted into a semantic vector of a fixed dimension according to the modal feature set to generate a text vector data set, including:

[0161] Text vectorization is the process of converting text into numerical representation. Common methods include:

[0162] Bag of Words model: By counting the frequency of occurrence of words in the text, the text is represented as a high-dimensional vector. For example: Text A: "The street light is broken and needs to be repaired."; Text B: "The community trash can is full.";

[0163] BoW may be converted to: text A → [1,0,1,1,0,0]; text B → [0,1,0,0,1,1]; this method is simple and easy to implement, but it cannot preserve the relationship between words.

[0164] Word2Vec / FastText / Glove (representation based on word embedding): These methods are based on deep learning and map words to a low-dimensional space, so that words with similar semantics are close in the vector space. For example, although the words "traffic light failure" and "signal light damage" are different in terms of word meaning, their vector representations may be very close.

[0165] In order to make all text data comparable in the same computational space, the dimensions of all text vectors need to be fixed. Common fixed-dimensional strategies include: setting a fixed vector length, such as 128 or 256 dimensions. Regardless of the length of the input text, the final output vector is of fixed dimension. Truncating or padding text: Text that is too short can be padded with special symbols. Text that is too long can be truncated to retain only the most important part.

[0166] In one case of this embodiment, the speech feature data is converted into a text vector according to the modal feature set, and encoded in combination with the emotion feature to generate a speech vector data set, including:

[0167] Since the core information of speech data is usually in the form of language expression, the first step is to convert the speech data into text data using automatic speech recognition:

[0168] End-to-end speech recognition, such as Wav2Vec2.0 and DeepSpeech: Directly convert audio to text to improve accuracy, especially suitable for government complaint scenarios. For example: "The traffic light on this road has been broken for three days."; Recognition result: "The traffic light on this road has been broken for three days"; The converted text can be directly used for text vectorization, which is consistent with the text complaint data.

[0169] In addition to text content, voice data also contains features such as pitch, volume, and speaking speed. These features can be used to determine the sentiment of the complaint:

[0170] Pitch: A high pitch may indicate that the complainant is emotional, while a low pitch may indicate calmness or dissatisfaction.

[0171] Speed: Fast speech may indicate an urgent complaint, while slow speech may indicate a general inquiry.

[0172] Volume: High volume may indicate a more emotional complainant, while low volume may indicate a less urgent complaint.

[0173] These emotional features can be extracted using methods such as short-time Fourier transform and Mel-frequency cepstral coefficients, and combined with text vectors to form a speech vector dataset.

[0174] Similar to text vectors, the dimension of speech vectors needs to be kept fixed: LSTM, CNN, or Transformer is used to process audio features and compress them into fixed dimensions, such as 128 or 256. The text vector (converted from ASR) and emotional features are combined to form the final speech vector dataset.

[0175] In one case of this embodiment, the image features are converted into vectors of fixed dimensions according to the modal feature set, and combined with the extracted text content to generate an image vector data set, including:

[0176] Image features can be extracted using deep learning methods:

[0177] Object detection: Use YOLO and FasterRCNN to identify key objects in the image, such as "broken road surface" and "dumped trash can".

[0178] Scene analysis: Extract global scene information such as "park environment" and "road facilities" through ResNet, VGG, and EfficientNet.

[0179] Optical character recognition: Identify text content in images, such as "Traffic light failure at XX intersection".

[0180] CNN is used for feature extraction and converted to a fixed dimension, such as 128 or 256, through a fully connected layer. Combined with the text content recognized by OCR, the text is converted into a vector and concatenated with the image vector to enhance the semantic information.

[0181] In a preferred embodiment of the present invention, the method of performing data matching according to the similarity of complaint content based on the cross-modal unified feature vector set, identifying cross-channel complaint information of the same event, and merging to generate fusion analysis data includes:

[0182] Based on the cross-modal unified feature vector set, the similarities of the text vector, the speech vector and the image vector are calculated respectively, and the text similarity matrix, the speech similarity matrix and the image similarity matrix are constructed;

[0183] According to the preset modal weights, the text similarity matrix, the speech similarity matrix and the image similarity matrix are fused to construct a multimodal data association matrix;

[0184] Set a similarity matching threshold, and based on the multimodal data association matrix, identify cross-channel complaint information for the same event and generate a unified event number;

[0185] Based on the unified event number, complaint information from different information channels is merged, and the timestamp, complainant information and problem description are integrated to remove redundant complaint content, extract core demands, and generate integrated analysis data.

[0186] In an embodiment of the present invention, in the closed-loop management of multimodal data of government affairs, accurately identifying cross-channel complaint information of the same event is the key to improving the efficiency of complaint handling. Since the complaint content of different channels may contain information of different modalities such as text, voice and image, matching directly based on raw data is easily affected by the format, expression and data dimension. Therefore, calculating the similarity of complaint content based on a cross-modal unified feature vector set and establishing a multimodal data association matrix can effectively solve this problem, improve the integration capability of complaint data, and avoid repeated processing.

[0187] In the matching calculation stage, the similarity of text, speech and image vectors is first calculated respectively, and the text similarity matrix, speech similarity matrix and image similarity matrix are constructed. The vector cosine similarity method can be used for text similarity calculation to ensure that even if the complaints are expressed differently, as long as the semantics are similar, they can be identified as the same event. For example, complaint A contains "the traffic light is damaged and no one has repaired it for many days", and complaint B contains "the signal light is broken and the traffic is chaotic". Although the two are expressed differently, they are semantically close and can be identified as the same event through similarity calculation. In the process of voice similarity calculation, in addition to converting to text to calculate semantic similarity, it will also be calculated in combination with intonation features, speech speed, etc., so that complaints with the same expression but strong emotions can obtain higher matching weights. For image similarity calculation, visual features such as object shape and color are extracted through deep learning methods, and combined with text information recognized by OCR to construct a visual matching matrix. For example, photos of the same damaged road uploaded by different users can be identified as the same complaint through visual feature comparison even if they are taken from different angles.

[0188] In the data fusion stage, a weighted fusion method is used to integrate the similarity matrices of text, speech, and images to generate a multimodal data association matrix. Since the importance of different modalities may vary, for example, text and speech may be more direct than images in expressing the content of complaints, different modal weighting coefficients can be set to make the similarity calculation more accurate. Through this method, the similarity calculation can be made to not only rely on a single modality, but also to integrate the matching degree of multimodal data to improve recognition accuracy.

[0189] In the same event identification stage, a similarity matching threshold is set, and based on the matching score of the multimodal data association matrix, complaint information belonging to the same event is identified and assigned a unified event number. In this way, even if complaints come from different channels, they can be automatically merged by the system. For example, complaint A comes from a telephone hotline, complaint B comes from an online reporting platform, and complaint C comes from social media, but the content involves the same infrastructure problem. The system can automatically identify that these complaints belong to the same event and generate a unique event number for them so that they can be processed centrally later to avoid repeated processing by multiple departments.

[0190] In the final data integration stage, the system will merge the complaint information of the same event, integrate the timestamp, complainant information and problem description, and remove redundant information. For example, when multiple users complain about the same problem, the system will extract valid information from different complaints to form a complete complaint description without repeatedly recording the same content. This mechanism ensures the integrity and accuracy of complaint information, reduces unnecessary data redundancy, and improves the efficiency of subsequent task allocation.

[0191] Through the above method, the automatic identification and merging of cross-channel complaint information of the same incident can be effectively realized, the level of intelligence in government business processing can be improved, and the waste of government resources caused by isolated or repeated processing of information can be avoided. At the same time, the speed of complaint response can be accelerated and citizen satisfaction can be improved.

[0192] In one case of this embodiment, the similarities of the text vector, the speech vector, and the image vector are calculated respectively according to the cross-modal unified feature vector set to construct a text similarity matrix, a speech similarity matrix, and an image similarity matrix, including:

[0193] Text similarity calculation is mainly used to identify complaints with similar text descriptions. Common similarity calculation methods include:

[0194] Cosine similarity: Calculates the angle between two text vectors. The higher the similarity, the more similar the complaint content is.

[0195] Semantic matching: Use deep learning models such as BERT, RoBERTa, and SentenceBERT for text embedding and calculate semantic similarity to avoid misjudgment caused by relying solely on keyword matching.

[0196] Jaccard similarity: Based on vocabulary overlap, it is suitable for short text complaints. For example: Complaint A: "The street light is broken" Complaint B: "The community street light is not on" has a high Jaccard similarity and can be judged as the same type of complaint.

[0197] Speech similarity calculation not only depends on the text content after speech recognition, but also needs to be combined with speech features, such as:

[0198] After converting speech to text, calculate text similarity, such as using BERT embedding.

[0199] Similarity of speech features: pitch similarity, speaking speed similarity, etc.

[0200] Mel spectrum analysis: Mel spectrum is used to compare voice content. If the audio content of the complaint is similar, even if there is a slight deviation in the speech-to-text conversion, it can be identified as a similar complaint.

[0201] For complaints that include images, it is necessary to analyze the image content to determine whether the complaints are similar. Common similarity calculation methods include:

[0202] Object detection: Use algorithms such as YOLO and FasterRCNN to detect key objects in images. For example, Complaint 1 picture: a photo of an overflowing trash can; Complaint 2 picture: a photo of an overflowing trash can at the same location. If the same object is detected, it means that the two pictures belong to the same event.

[0203] Feature extraction and hash comparison: ResNet, VGG, and EfficientNet are used for feature extraction, images are converted into fixed-dimensional vectors, and cosine similarity is calculated. Perceptual hashing is used to determine the similarity of image structures, so that similar images can be identified even if they are taken from different angles.

[0204] OCR recognition and comparison: Extract text information from the image, such as road signs and logos, and match them to determine whether they are complaints about the same location.

[0205] In one case of this embodiment, the setting of a similarity matching threshold, identifying cross-channel complaint information of the same event based on a multimodal data association matrix, and generating a unified event number includes:

[0206] Fixed threshold setting: Set a fixed similarity threshold, for example, 0.7. When the similarity scores of two complaints exceed 0.7, they are considered to be the same event.

[0207] Dynamic threshold setting: Combined with historical complaint data analysis, the threshold is adjusted dynamically. For example, in municipal infrastructure complaints, the text similarity threshold can be set to 0.8 to ensure that the complaint descriptions are close enough. In environmental sanitation complaints, since image similarity is more important, the text similarity threshold can be lowered to 0.6.

[0208] According to the multimodal data association matrix, the information of new complaints is compared with that of existing complaints. If the similarity between the new complaint and the existing complaint exceeds the set threshold, the new complaint will be classified as the same event.

[0209] For all matching complaint information, the system assigns a unified event number to ensure that tasks are not duplicated during subsequent processing. For example: Complaint 1 (social media): "The traffic light at XX intersection is not on.";

[0210] Complaint 2 (hotline): "The traffic light on XX Road has been broken for three days." After the system matches, a unified event number is generated: E20240201123, and all related complaints are classified as the same event for processing.

[0211] In a preferred embodiment of the present invention, the method of matching the fusion analysis data with a preset complaint category label, identifying the complaint category, and generating a classification result includes:

[0212] According to the text content in the fusion analysis data, the matching degree between the text vector and the preset complaint category label is calculated, and the text category matching result is generated;

[0213] According to the speech content in the fusion analysis data, the matching degree between the speech vector and the preset complaint category label is calculated, and the speech category matching result is generated;

[0214] According to the image content in the fusion analysis data, the matching degree between the image vector and the preset complaint category label is calculated, and the image category matching result is generated;

[0215] According to the text category matching results, voice category matching results and image category matching results, a modal fusion strategy is adopted to calculate the classification matching score, generate the classification result, and associate it with the corresponding complaint category label; the calculation formula of the classification matching score is: ,in, is the classification matching score, which is used to determine the category to which the complaint belongs. is the modal type, Represents text, Indicates voice, Represents an image, For modal The characteristic vector of is the modal vector of complaint category labels, is the modality fusion weight, and is the normalization parameter, is the norm of the mode vector, is the absolute difference between the modal vector and the class label vector.

[0216] In the embodiment of the present invention, in the government affairs business management, accurate classification of complaint categories is crucial to improving the efficiency of complaint handling. Government complaints usually contain information in multiple modes, such as text descriptions, voice calls, picture uploads, etc. The classification method of a single mode is easily limited, resulting in a decrease in accuracy. By matching through fusion analysis data, the complaint category can be identified more accurately, the task allocation can be more precise, and the overall automation of complaint handling can be improved.

[0217] In the classification stage, the text information is first matched and calculated. By calculating the matching degree between the text vector and the preset complaint category label, the category of the complaint text can be determined. For example, the complaint text "The street light is broken, it is dark at night, affecting residents' travel" can be matched with the category vector of "Municipal lighting facility problems" and their similarity is calculated. Categories with higher similarity will be used as possible complaint category candidates.

[0218] In the voice information classification stage, the voice information is first converted into text and classified and calculated in combination with emotional features. For example, if a citizen calls the hotline to complain that "the trash can is full, no one cleans it, and it smells too bad", the text may be matched with the "environmental sanitation problem" category after text conversion, but if the voice emotion is more excited, indicating that the problem may be more urgent, the matching weight of this category can be increased, so that it will be selected as the complaint category first.

[0219] In the image information classification stage, the visual feature matching method is used to analyze the elements in the complaint image. For example, if the complainant uploads a picture of a damaged park bench, the system will automatically detect the elements contained in the picture, such as broken wood, seat shape, etc., and combine the text information recognized by OCR to determine that the complaint may belong to the "damage to public facilities" category.

[0220] In the final classification decision stage, the modal fusion strategy is used to integrate the classification matching results of text, voice and image, and calculate the final classification matching score. In the end, the system will select the category with the highest matching score as the final classification result of the complaint. For example, if a complaint has a matching score of 1.2 in the "road problem" category, a matching score of 2.4 in the "municipal facility damage" category, and a matching score of 1.8 in the "sanitation problem" category, then the complaint will eventually be classified as the "municipal facility damage" category and subsequent tasks will be assigned.

[0221] This method can greatly improve the accuracy of complaint classification and avoid misclassification caused by insufficient single-modal information. For example, it may be difficult to judge the difference between "broken traffic lights" and "broken road signs" based on text alone, but combined with image information, the nature of the problem can be judged more accurately. Through multimodal fusion, not only the accuracy of classification is improved, but also the reasonable allocation of complaint tasks is ensured, avoiding misjudgment of responsible departments due to classification errors, and improving the automation and intelligence level of government complaint processing.

[0222] In addition, the method also supports optimizing classification rules based on historical data. By analyzing the classification accuracy of historical complaint data, the system can dynamically adjust the weighting coefficients of different modalities. For example, if the accuracy of voice information is found to be low, its weight can be appropriately reduced to enhance the influence of text and images to ensure that the classification results are more in line with the actual situation. This optimization mechanism enables the system to continuously improve classification accuracy, making complaint classification more efficient and accurate, and ultimately improving the overall efficiency of government complaint handling.

[0223] In a preferred embodiment of the present invention, the responsible department is determined according to the classification result, and a task assignment instruction is generated. The task assignment instruction includes the target responsible department, the processing priority, the recommended processing time limit and the associated complaint number, including:

[0224] According to the complaint categories in the classification results, query the responsible department database, determine the responsible department corresponding to the complaint category, and generate preliminary task allocation information;

[0225] Calculate the urgency of tasks and set processing priorities based on the current task load, historical processing efficiency and average response time of the responsible department;

[0226] The calculation formula of the task urgency is: ,in, For the task urgency, A collection of available responsible departments. For a specific responsible department, Responsible department The current backlog of tasks, Responsible department The historical average processing efficiency, The impact scope of the complaint, ranging from 0 to 1. is the adjustment coefficient, The time when the complaint was filed, Responsible department The average response time, and is the normalization parameter, is the number of currently available responsible departments;

[0227] in, , is the set of complaint channels involved in this complaint. For complaint channels, is the channel weight coefficient, For the channel The number of complaints forwarded on For this complaint in the channel The first exposure time on is the base time, used for normalization, and is the normalization parameter, For this complaint in the channel Number of user interactions on is the number of historical occurrences of the complaint. is the number of persons involved in the complaint, is the base number, used for normalization, The number of complaint channels involved in the complaint;

[0228] The calculation formula of the processing priority is: ,in, is the task priority, ranging from 0 to 1, For the task urgency, is the magnification factor, is the smoothing coefficient, The trigger threshold for task priority ensures that low-priority tasks are not processed prematurely;

[0229] Set a recommended time limit for handling the task based on the urgency of the task, the scope of the complaint and the feedback from the complainant;

[0230] Generate task assignment instructions, including responsible department, processing priority, recommended processing time limit and associated complaint number, and push the task assignment instructions to the responsible department's work system.

[0231] In an embodiment of the present invention, in the closed-loop management process of multimodal data of government affairs, the classified complaint information needs to be quickly and accurately assigned to the appropriate responsible department for processing. Since different types of complaints involve different departments, such as road problems may belong to the traffic management department, while environmental problems may belong to the urban environment management department, the accuracy of task allocation directly affects the efficiency of complaint processing. By generating task allocation instructions based on classification results, intelligent task allocation can be achieved, the speed and accuracy of government response can be improved, and task processing delays or incorrect allocations caused by human factors can be avoided.

[0232] In the process of task allocation, the system first determines the responsible department based on the classification results. Each complaint category corresponds to at least one responsible department. The system determines the mapping relationship between the complaint category and the responsible department by querying the responsible department database. For example, if the category of a complaint is "Road Pothole Repair", the system can automatically match it to the "Municipal Road Maintenance and Management Center" as the responsible department to avoid allocation errors that may be caused by manual judgment. In actual applications, the responsibilities of different responsible departments may overlap. To ensure accurate task allocation, the system will also combine historical task processing to analyze the best department to which similar complaints were best assigned in past processing to improve the accuracy of task allocation.

[0233] During the task priority setting stage, the system calculates the task priority based on the current task load of the responsible department, historical processing efficiency, and complaint urgency. The calculation of complaint urgency is mainly based on factors such as the scope of complaint impact, emotional analysis of the complainant, and the processing time of similar historical complaints. For example, if a complaint involves a road collapse around a school that affects the passage of teachers and students, the system will automatically assign it a higher urgency to ensure that the task is given priority. Compared with the traditional fixed priority method, this method can dynamically adjust the priority according to actual conditions, making task scheduling more scientific and reasonable.

[0234] Setting a time limit for task processing is also an important part of task allocation. The system will calculate a reasonable recommended time limit for task processing based on factors such as the urgency of the complaint content, the historical average response time of the responsible department, and the availability of processing resources. For example, if the historical average processing time for a certain type of complaint is 48 hours, and the current responsible department has a high backlog of tasks, the system may appropriately extend the recommended processing time limit to ensure the feasibility of task allocation. On the contrary, if the complaint involves an emergency such as a water supply interruption, the system may shorten the recommended time limit and attach an "urgent" label to encourage the responsible department to give priority to it.

[0235] Finally, the system generates a task assignment instruction and pushes it to the task management system of the responsible department. The task assignment instruction not only contains the information of the responsible department, but also includes the processing priority, recommended processing time limit and the associated complaint number, ensuring that the responsible department can manage and execute tasks efficiently. In addition, the task assignment instruction will also be synchronized to the government affairs management platform to facilitate the supervision and tracking of task execution and ensure the transparency of task processing.

[0236] This method greatly improves the intelligence level of government task allocation. Compared with the traditional manual allocation method, this method can dynamically adjust the task allocation strategy according to the actual situation, improve the efficiency of task allocation, and reduce allocation errors caused by human factors. At the same time, through the dynamic setting of task priority and time limit, it ensures that emergency complaints can be handled first, improves the response speed of government services, and makes government services more efficient and accurate.

[0237] In a preferred embodiment of the present invention, the processing status corresponding to the task assignment instruction is obtained, and a task progress record is constructed. The task progress record includes the processing progress fed back by the responsible department, the complaint processing time and the corresponding interim results, and generates warning information when the task times out or the processing progress is abnormal, including:

[0238] According to the target responsible department in the task assignment instruction, query the task execution status of the responsible department and obtain task progress data in real time;

[0239] According to the task progress data, record the processing progress, complaint handling time and corresponding interim results reported by the responsible department, and build a task progress record;

[0240] According to the task progress record, when the task exceeds the time limit or the processing progress is abnormal, an early warning message is generated and sent to the responsible department and the superior supervisory department.

[0241] In the embodiment of the present invention, in the process of handling government complaints, simply assigning tasks does not guarantee that the tasks can be executed smoothly. The execution status of the tasks needs to be monitored in real time to ensure that the tasks are completed within the specified time limit. Traditional task management methods often rely on manual records and regular reports, and cannot monitor the progress of tasks in real time, resulting in some tasks not being processed in time due to forgetting or delaying. By obtaining the task status in real time and building a task progress record, the transparency and controllability of government task management can be greatly improved, ensuring that complaints are resolved in a timely and effective manner.

[0242] In the task progress monitoring stage, the system first queries the task execution status from the task management system based on the target responsible department in the task assignment instruction, and obtains the task progress data in real time. The task progress data includes feedback information from the responsible department, task execution progress, completed processing steps, etc. For example, for a road maintenance task, the system can obtain detailed status information such as "construction has been completed 50%" or "materials have been allocated and are waiting for construction". These task progress data can help managers intuitively understand the processing status of tasks and ensure that tasks are executed as planned.

[0243] During the task progress recording stage, the system records the task completion rate, interim results and processing time of the responsible department based on the task progress data, and stores this information in the task management system. For example, if the goal of a complaint task is to repair a damaged bench in a park, the system will record the different stages of the task, such as "task received", "maintenance personnel dispatched", "maintenance completed, waiting for acceptance", etc. Through task progress recording, a complete task execution chain can be formed to ensure the traceability of the task and prevent the task from being lost or forgotten during execution.

[0244] In the task timeout warning mechanism, the system sets the task timeout threshold and monitors the task execution time in real time. If the task processing time exceeds the recommended processing time limit, the system will automatically trigger a timeout warning and notify the responsible department and the superior regulatory agency. For example, if the recommended processing time limit for a complaint task is 48 hours, and the current task has not been completed for more than 48 hours, the system will generate a timeout alarm and send it to the relevant responsible personnel to remind them to handle it as soon as possible. In addition, the system also supports the detection of abnormal task processing progress. For example, if the status of a task stays in the "received, not processed" stage for a long time, it indicates that the task may be forgotten or shelved. The system can automatically generate an abnormal warning and report it to the superior department to urge the relevant responsible department to speed up the processing.

[0245] In the task abnormality escalation mechanism, if a task has not been processed for a long time or the responsible department has delayed feedback, the system can automatically adjust the task allocation strategy and push the abnormal task to the backup responsible department or the superior management agency. For example, if a complaint task has been assigned to Department A for processing, but Department A is unable to handle it in time due to staff shortage, the system can automatically transfer the task to Department B to ensure that the task can still be completed on time. In addition, the system can also optimize the task allocation rules based on historical task execution. For example, if a responsible department often has processing delays in similar complaint tasks, the system can reduce the amount of tasks assigned to the department in the future to optimize the overall task execution efficiency.

[0246] This method greatly improves the controllability of government task execution. Compared with traditional manual management methods, this method can monitor task status in real time, ensure that tasks are completed within the specified time, and provide automated timeout warnings and exception handling mechanisms to reduce complaint backlogs and processing delays. In addition, through the accumulation of historical task data, the system can continuously optimize task allocation strategies, improve the efficiency of government task management, and make the government complaint handling process more intelligent and efficient.

[0247] An embodiment of the present invention further provides a multimodal data closed-loop management system based on government affairs, the system comprising:

[0248] Multimodal data acquisition module, used to obtain multimodal data from multiple information channels and parse it into structured data;

[0249] The feature extraction module is used to perform modality classification on structured data and extract features of text information, voice information and image information respectively to generate a modality feature set;

[0250] The cross-modal data unification module is used to perform feature conversion and normalization processing according to the modal feature set to generate a cross-modal unified feature vector set;

[0251] The cross-modal data matching module is used to match data according to the similarity of complaint content based on the cross-modal unified feature vector set, identify cross-channel complaint information of the same event, and merge and generate fusion analysis data, including the associated complaint number, timestamp, complainant information, and problem description;

[0252] The complaint classification module is used to match the fusion analysis data with the preset complaint category labels, identify the complaint category, and generate classification results;

[0253] A task assignment module, used to determine the responsible department according to the classification results and generate a task assignment instruction, wherein the task assignment instruction includes the target responsible department, processing priority, recommended processing time limit and associated complaint number;

[0254] The task progress management module is used to obtain the processing status corresponding to the task assignment instruction and build the task progress record, which includes the processing progress feedback from the responsible department, the complaint processing time and the corresponding interim results, and generates warning information when the task times out or the processing progress is abnormal;

[0255] The feedback and optimization module is used to generate feedback information based on the final processing results of the responsible department, return the feedback information through the corresponding communication channel to the complainant, and update the processing status corresponding to the task assignment instruction;

[0256] The information optimization module is used to generate optimization suggestions and adjust complaint category labels and task allocation strategies based on historical task progress records and historical feedback information.

[0257] It should be noted that the system is a system corresponding to the above method, and all implementation methods in the above method embodiment are applicable to this embodiment and can achieve the same technical effect.

[0258] The embodiment of the present invention further provides a computing device, including: a processor, a memory storing a computer program, and when the computer program is executed by the processor, the method described above is executed. All implementations in the above method embodiment are applicable to this embodiment and can achieve the same technical effect.

[0259] The embodiment of the present invention further provides a computer-readable storage medium storing instructions, which, when executed on a computer, enable the computer to execute the above method. All implementations in the above method embodiment are applicable to this embodiment and can achieve the same technical effect.

[0260] The above is a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.

Claims

1. A multimodal data closed-loop management method based on government affairs, characterized in that: The method comprises: Obtain multimodal data from multiple information channels and parse it into structured data; Perform modality classification on structured data, and extract features of text information, voice information, and image information respectively to generate a modality feature set; According to the modal feature set, feature conversion and normalization are performed to generate a cross-modal unified feature vector set; Based on the cross-modal unified feature vector set, data matching is performed according to the similarity of the complaint content, cross-channel complaint information of the same event is identified, and merged to generate fusion analysis data, including the associated complaint number, timestamp, complainant information, and problem description; Based on the fusion analysis data, the preset complaint category labels are used for matching, the complaint categories are identified, and the classification results are generated; Determine the responsible department according to the classification result and generate a task assignment instruction, which includes the target responsible department, processing priority, recommended processing time limit and associated complaint number; Obtain the processing status corresponding to the task assignment instruction and build a task progress record, which includes the processing progress reported by the responsible department, the complaint processing time and the corresponding interim results, and generate warning information when the task times out or the processing progress is abnormal; Generate feedback information based on the final processing results of the responsible department, return the feedback information through the corresponding communication channel to the complainant, and update the processing status corresponding to the task assignment instruction; Based on historical task progress records and historical feedback information, optimization suggestions are generated, and complaint category labels and task allocation strategies are adjusted.

2. According to claim 1, a closed-loop management method for multimodal data based on government affairs is characterized in that: The modality classification of the structured data is performed, and the features of text information, voice information and image information are extracted respectively to generate a modality feature set, including: According to the data type in the structured data, the data is divided into text data, voice data and image data; Perform word segmentation, syntactic analysis and sentiment recognition on text data, extract topic keywords, syntactic structure and sentiment features, and generate text feature data; Convert the speech data into audio signals, convert the speech data into text data, extract intonation features, speech speed features and emotion features, and generate speech feature data; Perform target detection and optical character recognition on image data, extract facial features, scene elements, text content and object features, and generate image feature data; The text feature data, speech feature data and image feature data are combined to generate a modal feature set.

3. A closed-loop management method for multimodal data based on government affairs according to claim 2, characterized in that: The step of performing feature conversion and normalization processing according to the modal feature set to generate a cross-modal unified feature vector set includes: According to the modal feature set, the text feature data is converted into a semantic vector of fixed dimension to generate a text vector dataset; According to the modal feature set, the speech feature data is converted into a text vector and encoded with the emotional features to generate a speech vector dataset; According to the modal feature set, the image features are converted into vectors of fixed dimensions and combined with the extracted text content to generate an image vector dataset; According to the text vector dataset, speech vector dataset and image vector dataset, a normalization method is adopted to convert all modal data into feature vectors of the same dimension to generate a cross-modal unified feature vector set.

4. A closed-loop management method for multimodal data based on government affairs services according to claim 3, characterized in that: According to the cross-modal unified feature vector set, data matching is performed according to the similarity of the complaint content, cross-channel complaint information of the same event is identified, and fusion analysis data is generated by merging, including: Based on the cross-modal unified feature vector set, the similarities of the text vector, the speech vector and the image vector are calculated respectively, and the text similarity matrix, the speech similarity matrix and the image similarity matrix are constructed; According to the preset modal weights, the text similarity matrix, the speech similarity matrix and the image similarity matrix are fused to construct a multimodal data association matrix; Set a similarity matching threshold, and based on the multimodal data association matrix, identify cross-channel complaint information for the same event and generate a unified event number; Based on the unified event number, complaint information from different information channels is merged, and the timestamp, complainant information and problem description are integrated to remove redundant complaint content, extract core demands, and generate integrated analysis data.

5. A closed-loop management method for multimodal data based on government affairs services according to claim 4, characterized in that: According to the fusion analysis data, the preset complaint category labels are used for matching, the complaint category is identified, and the classification results are generated, including: According to the text content in the fusion analysis data, the matching degree between the text vector and the preset complaint category label is calculated, and the text category matching result is generated; According to the speech content in the fusion analysis data, the matching degree between the speech vector and the preset complaint category label is calculated, and the speech category matching result is generated; According to the image content in the fusion analysis data, the matching degree between the image vector and the preset complaint category label is calculated, and the image category matching result is generated; According to the text category matching results, voice category matching results and image category matching results, a modal fusion strategy is adopted to calculate the classification matching score, generate the classification result, and associate it with the corresponding complaint category label; the calculation formula of the classification matching score is: ,in, is the classification matching score, which is used to determine the category to which the complaint belongs. is the modal type, Represents text, Indicates voice, Represents an image, For modal The characteristic vector of is the modal vector of complaint category labels, is the modality fusion weight, and is the normalization parameter, is the norm of the mode vector, is the absolute difference between the modal vector and the class label vector.

6. A closed-loop management method for multimodal data based on government affairs services according to claim 5, characterized in that: The responsible department is determined according to the classification results, and a task assignment instruction is generated. The task assignment instruction includes the target responsible department, processing priority, recommended processing time limit and associated complaint number, including: According to the complaint categories in the classification results, query the responsible department database, determine the responsible department corresponding to the complaint category, and generate preliminary task allocation information; Calculate the urgency of tasks and set processing priorities based on the current task load, historical processing efficiency and average response time of the responsible department; The calculation formula of the task urgency is: ,in, For the task urgency, A collection of available responsible departments. For a specific responsible department, Responsible department The current backlog of tasks, Responsible department The historical average processing efficiency, The impact scope of the complaint, ranging from 0 to 1. is the adjustment coefficient, The time when the complaint was filed, Responsible department The average response time, and is the normalization parameter, is the number of currently available responsible departments; in, , is the set of complaint channels involved in this complaint. For complaint channels, is the channel weight coefficient, For the channel The number of complaints forwarded on For this complaint in the channel The first exposure time on is the base time, used for normalization, and is the normalization parameter, For this complaint in the channel Number of user interactions on is the number of historical occurrences of the complaint. is the number of persons involved in the complaint, is the base number, used for normalization, The number of complaint channels involved in the complaint; The calculation formula of the processing priority is: ,in, is the task priority, ranging from 0 to 1, For the task urgency, is the magnification factor, is the smoothing coefficient, is the trigger threshold of the task priority; Set a recommended time limit for handling the task based on the urgency of the task, the scope of the complaint and the feedback from the complainant; Generate task assignment instructions, including responsible department, processing priority, recommended processing time limit and associated complaint number, and push the task assignment instructions to the responsible department's work system.

7. A closed-loop management method for multimodal data based on government affairs according to claim 6, characterized in that: The processing status corresponding to the task assignment instruction is obtained, and a task progress record is constructed. The task progress record includes the processing progress fed back by the responsible department, the complaint processing time and the corresponding interim results, and generates warning information when the task times out or the processing progress is abnormal, including: According to the target responsible department in the task assignment instruction, query the task execution status of the responsible department and obtain task progress data in real time; According to the task progress data, record the processing progress, complaint handling time and corresponding interim results reported by the responsible department, and build a task progress record; According to the task progress record, when the task exceeds the time limit or the processing progress is abnormal, an early warning message is generated and sent to the responsible department and the superior supervisory department.

8. A multimodal data closed-loop management system based on government affairs, characterized in that: Applied to the method according to any one of claims 1 to 7, the system comprises: Multimodal data acquisition module, used to obtain multimodal data from multiple information channels and parse it into structured data; The feature extraction module is used to perform modality classification on structured data and extract features of text information, voice information and image information respectively to generate a modality feature set; The cross-modal data unification module is used to perform feature conversion and normalization processing according to the modal feature set to generate a cross-modal unified feature vector set; The cross-modal data matching module is used to match data according to the similarity of complaint content based on the cross-modal unified feature vector set, identify cross-channel complaint information of the same event, and merge and generate fusion analysis data, including the associated complaint number, timestamp, complainant information, and problem description; The complaint classification module is used to match the fusion analysis data with the preset complaint category labels, identify the complaint category, and generate classification results; A task assignment module, used to determine the responsible department according to the classification results and generate a task assignment instruction, wherein the task assignment instruction includes the target responsible department, processing priority, recommended processing time limit and associated complaint number; The task progress management module is used to obtain the processing status corresponding to the task assignment instruction and build the task progress record, which includes the processing progress feedback from the responsible department, the complaint processing time and the corresponding interim results, and generates warning information when the task times out or the processing progress is abnormal; The feedback and optimization module is used to generate feedback information based on the final processing results of the responsible department, return the feedback information through the corresponding communication channel to the complainant, and update the processing status corresponding to the task assignment instruction; The information optimization module is used to generate optimization suggestions and adjust complaint category labels and task allocation strategies based on historical task progress records and historical feedback information.

9. A computing device, characterized in that include: one or more processors; A storage device for storing one or more programs, when the one or more programs are executed by the one or more processors, the one or more processors implement the method as claimed in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a program, which, when executed by a processor, implements the method according to any one of claims 1 to 7.

Images