Multi-mode intelligent disposal method and system based on one-network unified management city event work order
Through the multimodal intelligent handling method, combined with the deep learning model, the urban event work tickets are automatically classified and allocated, which solves the shortcomings of traditional systems in multimodal data processing and classification accuracy, and realizes efficient and accurate work ticket management and exception handling, meeting the real-time and efficient needs of smart city governance.
Patent Information
- Application Number
- CN202411860196.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-12-17
- Publication Date
- 2025-04-08
AI Technical Summary
Traditional urban event work order processing systems have shortcomings in processing multimodal data, classification accuracy, resource allocation efficiency and exception handling, and it is difficult to meet the real-time and efficient needs of smart city governance.
Multimodal intelligent disposal method is adopted, combined with the BERT model and ResNet-18 network to deeply extract text and pictures, automatically classify them through multimodal feature fusion and attention mechanism, and real-time monitoring and dynamic urgency are achieved through intelligent process design and one-network managed SMS platform.
It significantly improves the accuracy of work order classification and efficient allocation, reduces classification errors and delay problems, improves the efficiency of abnormal work orders, reduces dependence on human resources, and meets the real-time and efficient needs of smart city governance.
Smart Images
Figure CN119784319A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of urban management and intelligent work order processing, and in particular to a multimodal intelligent processing method and system for managing urban event work orders based on a single network. Background Art
[0002] With the acceleration of urbanization, the complexity of urban governance continues to increase. Especially in the management of public affairs and the handling of social events, city managers are faced with massive data, diverse problems and complex work order circulation requirements. Traditional urban event work order processing systems usually rely on manual review, manual allocation and targeted disposal. This method has many shortcomings in processing efficiency, accuracy and resource allocation. Although there are some intelligent work order management systems on the market, most systems can only perform simple classification and processing for single-modal data (such as text or pictures), which is difficult to cope with the complex multimodal data requirements in actual work orders. For example, some work order applicants will not only describe the text information of the event, but may also attach on-site photos as evidence. The single-modal processing method easily ignores the relationship between multimodal information, resulting in inaccurate work order classification results or delays that are difficult to solve in a timely manner. In addition, the existing system lacks the ability to coordinate and link in data processing, classification algorithms and work order status monitoring, and has limitations in overall efficiency and reliability.
[0003] In the prior art, text data is usually processed through traditional feature extraction methods such as TF-IDF or keyword matching. These methods have certain advantages when processing structured data, but they are weak in understanding text semantics, and are particularly difficult to accurately identify contextual relationships and key information in long texts. In addition, traditional text processing methods are difficult to use in conjunction with image data, resulting in low processing capabilities for multimodal work order information. In terms of image data processing, although deep learning methods are gradually being applied to work order systems, most systems are still at the basic image recognition stage, failing to effectively combine image information with text information, and failing to fully explore the semantic relevance between multimodal data. The lack of multimodal feature fusion directly limits the performance of the classification model, resulting in low accuracy in work order classification in complex scenarios, affecting the flow efficiency and processing quality of work orders.
[0004] In addition, traditional work order status monitoring methods are mainly based on manual management, relying on manual input and limited data support from management tools, and cannot achieve real-time status updates and intelligent feedback. Once a work order is delayed, failed, or rejected, it is often necessary to track and supervise it through manual intervention. This method not only consumes human resources, but also easily leads to untimely information transmission, further exacerbating the lag in work order processing. In terms of urging and supervising, existing systems usually use simple timed reminders, which cannot be dynamically adjusted according to changes in work order status, and cannot flexibly respond to abnormal situations that may occur during the work order circulation process.
[0005] In addition, some work order systems in the existing technology support automated training and optimization of classification models, but most of them use online training methods, which are difficult to process large-scale historical work order data. Although online training can adapt to new data in a timely manner, it is prone to model performance degradation or overfitting problems. Especially in multimodal data processing and classification tasks, the model is more dependent on the amount of data. The limitations of online training lead to the inability to fully guarantee the stability and accuracy of the model. At the same time, the existing model deployment methods lack flexibility, and the model update and call efficiency is low. Especially when faced with sudden large-scale work order flows, the system is prone to response delays or service interruptions, which makes it difficult to meet the real-time and efficiency requirements of smart city governance.
[0006] Therefore, how to provide a multimodal intelligent handling method and system based on a single network to manage urban event work orders is an urgent problem that technical personnel in this field need to solve. Summary of the invention
[0007] One purpose of the present invention is to propose a multimodal intelligent handling method and system based on a one-network unified management of urban event work orders. The present invention improves the accuracy of work order classification, the efficiency of distribution, and the automatic management capability of the whole process by combining multimodal data processing technology, deep learning model, multimodal feature fusion and classification technology, and intelligent process design. Through the deep feature mining and fusion of text and pictures, the present invention can accurately classify complex work orders, and through the configurable design of prefabricated processes, it can realize the rapid distribution and disposal of work orders, avoiding the problems of classification errors, missed orders, and delays in traditional work order systems. At the same time, the present invention realizes real-time monitoring of the work order status through a one-network unified management SMS platform, combines dynamic urging and intelligent supervision functions, ensures the timely processing of work orders and the efficient resolution of abnormal work orders, and significantly reduces the dependence on human resources. Combined with cloud model deployment and offline training optimization technology, the present invention enhances the flexibility and scalability of the system, and can be widely applied to multiple fields such as smart city governance, public affairs management, and enterprise services, providing innovative technical support for improving social governance efficiency and service quality.
[0008] According to an embodiment of the present invention, a multi-modal intelligent handling method based on one-network management of city event work orders includes the following steps:
[0009] S1. Obtain the text and image information of the work order, pre-process the work order text information, complete the key information and store it in the Redis database;
[0010] S2, read the completed text information from the Redis database, use the BERT model to perform bidirectional Transformer encoding on it, extract text semantic features, and generate a text feature vector;
[0011] S3, read the original image information from the Redis database, use the ResNet-18 deep learning network to extract features of the image information, and generate an image feature vector;
[0012] S4, inputting the text feature vector and the image feature vector into the contrastive learning model, performing feature fusion, generating a multimodal feature vector, and storing it in the multimodal feature space;
[0013] S5. Input the multimodal feature vector into the multimodal attention mechanism and the Bi-LSTM classification model to automatically classify the work order, generate the classification result, and assign it to the predefined disposal process;
[0014] S6. Use the problem acceptance module to monitor the classification results and handling processes in real time, and manage the time limit and status information of work orders.
[0015] Optionally, the S2 specifically includes:
[0016] S21. Read the completed text information from the Redis database, extract the complete fields and semantic context as input data for the BERT model.
[0017] S22, input the completed text information into the word embedding layer of the BERT model, map it into a vector of fixed dimension after word segmentation, and generate a word embedding vector sequence;
[0018] S23, input the generated word embedding vector into the multi-head self-attention mechanism, generate attention weights by calculating the degree of association between words, and generate weighted feature representations for each word;
[0019] S24. Perform nonlinear transformation on the output vector of the multi-head attention mechanism through the fully connected layer:
[0020] FFN(x)=ReLU(W1x+b1)W2+b2;
[0021] Among them, FFN is a feedforward neural network, x is the input feature vector, W1, W2 are weight matrices, b1, b2 are bias terms, and ReLU is the activation function;
[0022] S25, passing the feature vector of the text sequence through multiple stacked Transformer encoder layers in sequence, extracting global semantic information layer by layer, including vocabulary, syntax and semantic features, and integrating them into a unified high-dimensional feature representation;
[0023] S26. Based on a unified high-dimensional feature representation, a pooling method is used to generate global semantic features of the entire text and store them in the Redis database.
[0024] Optionally, the S3 specifically includes:
[0025] S31, read the original image information from the Redis database, adjust the resolution and normalize the pixel value range;
[0026] S32, input the adjusted image into the first convolution layer of the ResNet-18 model, extract low-level features through the convolution kernel, and output a preliminary feature map;
[0027] S33. The generated feature map is passed to the residual block structure of ResNet-18. Each residual block extracts deeper features through multi-layer convolution and jump connection operations while retaining low-level features:
[0028] y=F(x,{W i})+x;
[0029] Among them, F represents a mapping function, x is the input feature map, and W i is the weight matrix and bias term, y is the output of the residual block;
[0030] S34, input the feature map processed by the residual block to the global pooling layer of ResNet-18, and the global pooling averages the pixel values of each feature map to reduce the feature dimension while retaining key semantic information to generate a feature vector of a specific dimension;
[0031] S35. Use a fully connected layer to further process the feature vector, compress the feature vector to a specified dimension, and store the optimized feature vector in a Redis database.
[0032] Optionally, the S4 specifically includes:
[0033] S41, inputting the text feature vector and the image feature vector into a mapping module of a multimodal feature space respectively;
[0034] S42, using a dot product operation to calculate the similarity between each pair of text vectors and image vectors, and generate a similarity matrix;
[0035] S43. In each row of the similarity matrix, the Softmax function is used for normalization to highlight the main correlation between the text vector and the image vector:
[0036]
[0037] Among them, Softmax represents the normalized exponential function, S i,j represents the value of the i-th row and j-th column in the similarity matrix, exp represents the exponential function, N is the total number of categories, and k is the index of the category;
[0038] S44, calculating a bidirectional loss function based on the normalized similarity matrix, and combining the loss function values for model training optimization;
[0039] S45. Use the optimized model to generate a fusion feature vector, recalculate the weights of the text features and the image features through the fusion model, generate a multimodal fusion feature vector, and store the generated multimodal feature vector in the multimodal feature space:
[0040] F multi =W T ·T+W W W;
[0041] Where W T and W W is the weight matrix, T is the text vector, W is the image vector, F multi is the multimodal feature vector.
[0042] Optionally, the S5 specifically includes:
[0043] S51, input the multimodal feature vector into the multimodal attention mechanism, calculate the internal correlation using the query matrix, key matrix and value matrix, and generate a preliminary weighted multimodal feature vector:
[0044]
[0045] Among them, Attention is a function for calculating attention distribution, softmax is a normalization function, and d k is the dimension of the key vector, Q is the query matrix, K is the key matrix, and K T represents the transpose operation of the key matrix K, V is the value matrix;
[0046] S52, inputting the preliminarily weighted multimodal feature vector into the multi-head attention mechanism, capturing multi-dimensional context features through parallel processing, and concatenating and fusing the output results of multiple attention heads into a context-sensitive global feature vector;
[0047] S53, inputting the global feature vector into the Bi-LSTM model, using the forward and backward LSTM units to process the past and future information of the multimodal feature sequence respectively, and generating a feature representation containing the complete context through a concatenation operation;
[0048] S54, extract global features from the output of Bi-LSTM, and select the hidden state of the last time step as the global classification feature of the work order;
[0049] S55. Input the global classification features into the Softmax function, calculate the classification probability distribution, normalize each category, and generate probability values belonging to different categories:
[0050]
[0051] Where P(y=c|h T ) represents the possibility that the input feature belongs to category C, h T is the global classification feature vector, is the weight matrix of the classifier, represents the matrix row vector, b k represents the bias term of category K, b c represents the bias term of category C, C is the total number of categories, and exp represents the exponential function;
[0052] S56. According to the classification probability output by Softmax, the category with the highest probability is selected as the classification result, and the work order is assigned to the predefined disposal process according to the classification result, so as to complete the classification and work order flow operation;
[0053] S57, using an offline training method to optimize the multimodal information vectorization representation and classification model through a joint loss function;
[0054] S58. The classification model is trained through the Pytorch framework and then serialized and stored in the cloud server. When the user calls it, the model is restored to its original state through deserialization to complete multimodal feature classification and work order allocation.
[0055] Optionally, the S6 specifically includes:
[0056] S61. Automatically generate work order handling time limit parameters according to the work order application time and expected completion time, and divide the time limit into primary, secondary and tertiary thresholds, which are set to half of the work order time limit respectively, initialize the countdown monitoring state, and enter the real-time monitoring process;
[0057] S62. During the countdown monitoring process, when the countdown status of the work order approaches the time limit threshold, a reminder notification is automatically triggered, and a reminder message is sent to the relevant administrator through the one-stop SMS platform to urge him to speed up the handling progress of the work order;
[0058] S63. During the work order handling process, the status information is monitored in real time, including successful handling, delayed and rejected status; for delayed work orders, the urging process is continued to be triggered; for successfully handled work orders, the completion status is recorded; for rejected work orders, an exception record is generated;
[0059] S64. For rejected work orders, generate supervision tasks by analyzing the reasons for rejection, and assign them to the upper-level responsible person or designated processing personnel for collaborative handling in combination with the task list to be completed and processing suggestions;
[0060] S65. After the work order is successfully handled, the handling result is notified to the work order applicant and administrator through the One-Network SMS platform or system message, including the work order number, completion time, handling result and operation suggestions;
[0061] S66. Work orders that are not completed on time and are rejected multiple times are marked as abnormal work orders, and a report record is generated to record the abnormal cause and recommended processing plan, and report it to the system administrator or designated personnel for subsequent tracking and processing.
[0062] The multimodal intelligent handling system based on one network to manage city event work orders according to an embodiment of the present invention includes the following modules:
[0063] The work order information mining module is responsible for extracting key information from the work order text and image information;
[0064] The work order distribution module is used to fuse text and image feature vectors to generate multimodal feature vectors and automatically distribute work orders through the classification model;
[0065] The problem acceptance module is used to monitor the status and time limit of work orders in real time and handle timeouts and abnormal situations;
[0066] The classification result feedback and allocation module is used to provide feedback on the disposal results and automatically allocate the classified work orders;
[0067] The model training and optimization module is used to optimize system model performance through offline training and deployment.
[0068] The beneficial effects of the present invention are:
[0069] The present invention significantly improves the efficiency and accuracy of urban event work order management by introducing a multimodal intelligent handling method and system, and overcomes many deficiencies in the traditional work order handling system. In terms of classification accuracy, the present invention combines the BERT model and the ResNet-18 network to perform deep semantic feature extraction on the work order text and images, effectively mines the correlation between text and image information through multimodal feature fusion technology, and adopts a multimodal attention mechanism and a Bi-LSTM classification model to improve the accuracy of work order classification in complex scenarios, solving the problem of insufficient classification accuracy caused by single modal processing in the prior art. The efficient fusion of multimodal features enables work orders to be accurately classified in a variety of scenarios, laying a solid foundation for subsequent process allocation.
[0070] In terms of work order distribution, the present invention realizes the rapid distribution of work orders and efficient matching of disposal paths through configurable prefabricated process design combined with intelligent classification results. This mechanism ensures the flexibility and adaptability of work order distribution, and effectively avoids classification errors, missed orders and delays in traditional work order disposal. The system's built-in process configuration module ensures the clarity and unification of work order processing rules, and enhances the ability to support complex business needs. At the same time, with the help of a unified SMS platform, the present invention realizes dynamic monitoring and status feedback of work orders, and can send reminders in real time according to the time limit and status of the work order, promote the progress of work order processing, and avoid disposal delays caused by timeouts.
[0071] In addition, the present invention performs well in handling abnormal work orders. Through real-time monitoring and status analysis of work orders, the system can accurately capture rejected or delayed work orders, and combined with the generated supervision tasks and processing suggestions, assign the problem work orders to the responsible person for rapid processing, ensuring the smooth operation of the work order process. At the same time, work orders that are not completed on time or rejected multiple times are marked and reported as abnormal, combined with the subsequent processing of the system administrator, the management ability of abnormal work orders and the efficiency of problem response are improved, and the decline in governance efficiency caused by the backlog of abnormal work orders is significantly reduced.
[0072] In terms of model training and deployment, the present invention adopts offline training and joint loss optimization strategies, and periodically optimizes the multimodal feature extraction and classification model through the continuously accumulated historical data, thereby improving the stability and classification performance of the model. Combined with the cloud deployment method of the Pytorch framework, the system supports dynamic calling and rapid updating of the model, realizes efficient and reliable service response capabilities, and provides technical support for the intelligent management of work orders. BRIEF DESCRIPTION OF THE DRAWINGS
[0073] The accompanying drawings are used to provide a further understanding of the present invention and constitute a part of the specification. Together with the embodiments of the present invention, they are used to explain the present invention and do not constitute a limitation of the present invention. In the accompanying drawings:
[0074] Figure 1 This is a flow chart of the multi-modal intelligent handling method for city event work orders based on one network management proposed by the present invention;
[0075] Figure 2 This is a structural diagram of the multimodal intelligent handling system proposed by the present invention for managing city event work orders through a single network. DETAILED DESCRIPTION
[0076] The present invention will now be described in further detail with reference to the accompanying drawings. These drawings are simplified schematic diagrams, which only illustrate the basic structure of the present invention in a schematic manner, and therefore only show the components related to the present invention.
[0077] refer to Figure 1 , a multi-modal intelligent handling method based on one network to manage city event work orders includes the following steps:
[0078] S1. Obtain the text and image information of the work order, pre-process the work order text information, complete the key information and store it in the Redis database;
[0079] S2, read the completed text information from the Redis database, use the BERT model to perform bidirectional Transformer encoding on it, extract text semantic features, and generate a text feature vector;
[0080] S3, read the original image information from the Redis database, use the ResNet-18 deep learning network to extract features of the image information, and generate an image feature vector;
[0081] S4, inputting the text feature vector and the image feature vector into the contrastive learning model, performing feature fusion, generating a multimodal feature vector, and storing it in the multimodal feature space;
[0082] S5. Input the multimodal feature vector into the multimodal attention mechanism and the Bi-LSTM classification model to automatically classify the work order, generate the classification result, and assign it to the predefined disposal process;
[0083] S6. Use the problem acceptance module to monitor the classification results and handling processes in real time, and manage the time limit and status information of work orders.
[0084] In this implementation, S2 specifically includes:
[0085] S21. Read the completed text information from the Redis database, extract the complete fields and semantic context as input data for the BERT model.
[0086] S22, input the completed text information into the word embedding layer of the BERT model, map it into a vector of fixed dimension after word segmentation, and generate a word embedding vector sequence;
[0087] S23, input the generated word embedding vector into the multi-head self-attention mechanism, generate attention weights by calculating the degree of association between words, and generate weighted feature representations for each word;
[0088] S24. Perform nonlinear transformation on the output vector of the multi-head attention mechanism through the fully connected layer:
[0089] FFN(x)=ReLU(W1x+b1)W2+b2;
[0090] Among them, FFN is a feedforward neural network, x is the input feature vector, W1, W2 are weight matrices, b1, b2 are bias terms, and ReLU is the activation function;
[0091] S25, passing the feature vector of the text sequence through multiple stacked Transformer encoder layers in sequence, extracting global semantic information layer by layer, including vocabulary, syntax and semantic features, and integrating them into a unified high-dimensional feature representation;
[0092] S26. Based on a unified high-dimensional feature representation, a pooling method is used to generate global semantic features of the entire text and store them in the Redis database.
[0093] In this implementation, S3 specifically includes:
[0094] S31, read the original image information from the Redis database, adjust the resolution and normalize the pixel value range;
[0095] S32, input the adjusted image into the first convolution layer of the ResNet-18 model, extract low-level features through the convolution kernel, and output a preliminary feature map;
[0096] S33. The generated feature map is passed to the residual block structure of ResNet-18. Each residual block extracts deeper features through multi-layer convolution and jump connection operations while retaining low-level features:
[0097] y=F(x,{W i})+x;
[0098] Among them, F represents a mapping function, x is the input feature map, and W i is the weight matrix and bias term, y is the output of the residual block;
[0099] S34, input the feature map processed by the residual block to the global pooling layer of ResNet-18, and the global pooling averages the pixel values of each feature map to reduce the feature dimension while retaining key semantic information to generate a feature vector of a specific dimension;
[0100] S35. Use a fully connected layer to further process the feature vector, compress the feature vector to a specified dimension, and store the optimized feature vector in a Redis database.
[0101] In this implementation, S4 specifically includes:
[0102] S41, inputting the text feature vector and the image feature vector into a mapping module of a multimodal feature space respectively;
[0103] S42, using a dot product operation to calculate the similarity between each pair of text vectors and image vectors, and generate a similarity matrix;
[0104] S43. In each row of the similarity matrix, the Softmax function is used for normalization to highlight the main correlation between the text vector and the image vector:
[0105]
[0106] Among them, Softmax represents the normalized exponential function, S i,j represents the value of the i-th row and j-th column in the similarity matrix, exp represents the exponential function, N is the total number of categories, and k is the index of the category;
[0107] S44, calculating a bidirectional loss function based on the normalized similarity matrix, and combining the loss function values for model training optimization;
[0108] S45. Use the optimized model to generate a fusion feature vector, recalculate the weights of the text features and the image features through the fusion model, generate a multimodal fusion feature vector, and store the generated multimodal feature vector in the multimodal feature space:
[0109] F multi =W T ·T+W W W;
[0110] Where W T and W W is the weight matrix, T is the text vector, W is the image vector, F multi is the multimodal feature vector.
[0111] In this implementation manner, S5 specifically includes:
[0112] S51, input the multimodal feature vector into the multimodal attention mechanism, calculate the internal correlation using the query matrix, key matrix and value matrix, and generate a preliminary weighted multimodal feature vector:
[0113]
[0114] Among them, Attention is a function for calculating attention distribution, softmax is a normalization function, and d k is the dimension of the key vector, Q is the query matrix, K is the key matrix, and K T represents the transpose operation of the key matrix K, V is the value matrix;
[0115] S52, inputting the preliminarily weighted multimodal feature vector into the multi-head attention mechanism, capturing multi-dimensional context features through parallel processing, and concatenating and fusing the output results of multiple attention heads into a context-sensitive global feature vector;
[0116] S53, inputting the global feature vector into the Bi-LSTM model, using the forward and backward LSTM units to process the past and future information of the multimodal feature sequence respectively, and generating a feature representation containing the complete context through a concatenation operation;
[0117] S54, extract global features from the output of Bi-LSTM, and select the hidden state of the last time step as the global classification feature of the work order;
[0118] S55. Input the global classification features into the Softmax function, calculate the classification probability distribution, normalize each category, and generate probability values belonging to different categories:
[0119]
[0120] Where P(y=c|h T ) represents the possibility that the input feature belongs to category C, h T is the global classification feature vector, is the weight matrix of the classifier, represents the matrix row vector, b k represents the bias term of category K, b c represents the bias term of category C, C is the total number of categories, and exp represents the exponential function;
[0121] S56. According to the classification probability output by Softmax, the category with the highest probability is selected as the classification result, and the work order is assigned to the predefined disposal process according to the classification result, so as to complete the classification and work order flow operation;
[0122] S57, using an offline training method to optimize the multimodal information vectorization representation and classification model through a joint loss function;
[0123] S58. The classification model is trained through the Pytorch framework and then serialized and stored in the cloud server. When the user calls it, the model is restored to its original state through deserialization to complete multimodal feature classification and work order allocation.
[0124] In this implementation manner, S6 specifically includes:
[0125] S61. Automatically generate work order handling time limit parameters according to the work order application time and expected completion time, and divide the time limit into primary, secondary and tertiary thresholds, which are set to half of the work order time limit respectively, initialize the countdown monitoring state, and enter the real-time monitoring process;
[0126] S62. During the countdown monitoring process, when the countdown status of the work order approaches the time limit threshold, a reminder notification is automatically triggered, and a reminder message is sent to the relevant administrator through the one-stop SMS platform to urge him to speed up the handling progress of the work order;
[0127] S63. During the work order handling process, the status information is monitored in real time, including successful handling, delayed and rejected status; for delayed work orders, the urging process is continued to be triggered; for successfully handled work orders, the completion status is recorded; for rejected work orders, an exception record is generated;
[0128] S64. For rejected work orders, generate supervision tasks by analyzing the reasons for rejection, and assign them to the upper-level responsible person or designated processing personnel for collaborative handling in combination with the task list to be completed and processing suggestions;
[0129] S65. After the work order is successfully handled, the handling result is notified to the work order applicant and administrator through the One-Network SMS platform or system message, including the work order number, completion time, handling result and operation suggestions;
[0130] S66. Work orders that are not completed on time and are rejected multiple times are marked as abnormal work orders, and a report record is generated to record the abnormal cause and recommended processing plan, and report it to the system administrator or designated personnel for subsequent tracking and processing.
[0131] refer to Figure 2 , a multi-modal intelligent handling system based on one network to manage city event work orders, including the following modules:
[0132] The work order information mining module is responsible for extracting key information from the work order text and image information;
[0133] The work order distribution module is used to fuse text and image feature vectors to generate multimodal feature vectors and automatically distribute work orders through the classification model;
[0134] The problem acceptance module is used to monitor the status and time limit of work orders in real time and handle timeouts and abnormal situations;
[0135] The classification result feedback and allocation module is used to provide feedback on the disposal results and automatically allocate the classified work orders;
[0136] The model training and optimization module is used to optimize system model performance through offline training and deployment.
[0137] Embodiment 1:
[0138] In order to verify the feasibility of the present invention in implementation, the present invention is applied to a smart city management platform. In urban social governance, citizens submit event work orders through various channels, such as road damage repair, public facility maintenance or safety hazard inspection. The submission form of these work orders usually includes text descriptions, picture evidence and additional information. The traditional work order handling system relies on manual distribution, which consumes a lot of time and manpower. At the same time, there are obvious deficiencies in classification accuracy, especially in the collaborative processing of multimodal data. Traditional methods are difficult to effectively integrate text and image features, resulting in low efficiency in work order distribution and prone to delays or missed orders. This embodiment takes a road damage repair incident in a certain city as an example to introduce how the multimodal intelligent handling method and system of the present invention based on a one-network management of urban event work orders are applied in actual scenarios, and demonstrates its significant beneficial effects through comparative data.
[0139] The road management department of a certain city received a work order submitted by a citizen on the smart city management platform, which was described as "large cracks appeared on the road, affecting the safety of vehicle traffic" and attached with multiple on-site photos. The traditional system requires manual reviewers to first analyze the text content, then compare the image information one by one to determine the type of problem, and then manually distribute it to the maintenance department. Due to the large amount of information involved, manual analysis and classification are time-consuming, usually requiring 1-2 hours to complete the distribution, and the classification results may be biased due to inconsistent manual judgments. The multimodal intelligent disposal system of the present invention can complete this process quickly and accurately.
[0140] In this scenario, the work order information mining module of the present invention first extracts text information and image features. The system extracts key semantic features such as "road", "crack", and "vehicle traffic safety" by bidirectional Transformer encoding of the text description through the BERT model. At the same time, the ResNet-18 deep learning network performs semantic analysis on the on-site pictures, extracts the location, area, and eigenvalues of the crack area, and generates a multimodal feature vector in combination with the text features. After the multimodal work order distribution module fuses the text and image features through the multimodal attention mechanism, it uses the Bi-LSTM classification model to complete the accurate classification of the problem type. The system marks the work order as "road maintenance-emergency" and automatically distributes it to the emergency response team of the road management department. The whole process takes less than 5 minutes.
[0141] During the work order processing, the problem acceptance module dynamically monitors the status and time limit of the work order. The system generates a processing time limit based on the work order application time and the expected completion time, and sets the first, second, and third level time limit thresholds. When the work order is at the second level threshold, the system sends a reminder to the relevant person in charge through the one-stop SMS platform to remind them to speed up the processing. After the emergency team completes the maintenance task, the system automatically feeds back the work order processing results to the applicant via SMS, including the work order number, maintenance time, and task completion status.
[0142] To verify the beneficial effects of the present invention, a certain city evaluated the system performance during a pilot period from January to March 2023, and recorded a number of key indicators including work order classification accuracy, distribution efficiency, and exception handling efficiency.
[0143] Table 1 Performance comparison data between the traditional system and the system of the present invention during the pilot period
[0144]
[0145] It can be seen from the above data table 1 that the system of the present invention shows excellent processing efficiency in the dispatch time of emergency work orders. The traditional system takes about 90 minutes to complete the dispatch of emergency work orders, while the system of the present invention shortens this time to only 7 minutes, reducing the dispatch time by 92.2%. This shows that the present invention has greatly improved the response speed to emergency work orders through multimodal data fusion and intelligent classification technology, and can dispatch solution resources for key problems in a short time, effectively ensuring the timeliness of emergency response.
[0146] Secondly, the system of the present invention also shows a significant improvement in the dispatch time of ordinary work orders. The traditional system takes an average of 120 minutes, while the system of the present invention can complete the dispatch in only 15 minutes, reducing the time by 87.5%. This result proves that the multimodal intelligent dispatch technology of the present invention is not only suitable for the rapid response of emergency work orders, but also can show high efficiency in daily complex business scenarios.
[0147] In terms of classification accuracy, the traditional system is limited by the subjectivity of manual judgment, with a classification accuracy of 85%. This invention extracts text and image features through deep learning technology, and combines the multimodal attention mechanism with the Bi-LSTM classification model to increase the classification accuracy to 98%, an increase of 15.3%. This significant improvement not only reduces the delay of work orders or waste of resources caused by classification errors, but also lays the foundation for the accuracy of subsequent process allocation.
[0148] The efficiency of handling abnormal work orders is an important indicator for evaluating system capabilities. The success rate of handling abnormal work orders in traditional systems is 60%, while the present invention increases the success rate of abnormal handling to 95%, an increase of 58.3%, through functions such as real-time monitoring, intelligent supervision and dynamic task allocation. In addition, the system of the present invention also performs well in controlling the number of work orders that are not completed on time, from 400 in traditional systems to 80, a reduction of 80%. These data fully demonstrate the intelligence and efficiency of the present invention in handling abnormal work orders, which helps to avoid backlogs of problems and improve overall work order management capabilities.
[0149] The above description is only a preferred specific implementation manner of the present invention, but the protection scope of the present invention is not limited thereto. Any technician familiar with the technical field can make equivalent replacements or changes according to the technical scheme and inventive concept of the present invention within the technical scope disclosed by the present invention, which should be covered by the protection scope of the present invention.
Claims
1. A multi-modal intelligent handling method based on one network to manage city event work orders, characterized in that: The steps include: S1. Obtain the text and image information of the work order, pre-process the work order text information, complete the key information and store it in the Redis database; S2, read the completed text information from the Redis database, use the BERT model to perform bidirectional Transformer encoding on it, extract text semantic features, and generate a text feature vector; S3, read the original image information from the Redis database, use the ResNet-18 deep learning network to extract features of the image information, and generate an image feature vector; S4, inputting the text feature vector and the image feature vector into the contrastive learning model, performing feature fusion, generating a multimodal feature vector, and storing it in the multimodal feature space; S5. Input the multimodal feature vector into the multimodal attention mechanism and the Bi-LSTM classification model to automatically classify the work order, generate the classification result, and assign it to the predefined disposal process; S6. Use the problem acceptance module to monitor the classification results and handling processes in real time, and manage the time limit and status information of work orders.
2. The multi-modal intelligent handling method based on one-network management of city event work orders according to claim 1 is characterized in that: The S2 specifically includes: S21. Read the completed text information from the Redis database, extract the complete fields and semantic context as input data for the BERT model. S22, input the completed text information into the word embedding layer of the BERT model, map it into a vector of fixed dimension after word segmentation, and generate a word embedding vector sequence; S23, input the generated word embedding vector into the multi-head self-attention mechanism, generate attention weights by calculating the degree of association between words, and generate weighted feature representations for each word; S24. Perform nonlinear transformation on the output vector of the multi-head attention mechanism through the fully connected layer: FFN(x)=ReLU(W1x+b1)W2+b2; Among them, FFN is a feedforward neural network, x is the input feature vector, W1, W2 are weight matrices, b1, b2 are bias terms, and ReLU is the activation function; S25, passing the feature vector of the text sequence through multiple stacked Transformer encoder layers in sequence, extracting global semantic information layer by layer, including vocabulary, syntax and semantic features, and integrating them into a unified high-dimensional feature representation; S26. Based on a unified high-dimensional feature representation, a pooling method is used to generate global semantic features of the entire text and store them in the Redis database.
3. The multi-modal intelligent handling method based on one-network management of city event work orders according to claim 1 is characterized in that: The S3 specifically includes: S31, read the original image information from the Redis database, adjust the resolution and normalize the pixel value range; S32, input the adjusted image into the first convolution layer of the ResNet-18 model, extract low-level features through the convolution kernel, and output a preliminary feature map; S33. The generated feature map is passed to the residual block structure of ResNet-18. Each residual block extracts deeper features through multi-layer convolution and jump connection operations while retaining low-level features: y=F(x,{W i })+x; Among them, F represents a mapping function, x is the input feature map, and W i is the weight matrix and bias term, y is the output of the residual block; S34, input the feature map processed by the residual block to the global pooling layer of ResNet-18, and the global pooling averages the pixel values of each feature map to reduce the feature dimension while retaining key semantic information to generate a feature vector of a specific dimension; S35. Use a fully connected layer to further process the feature vector, compress the feature vector to a specified dimension, and store the optimized feature vector in a Redis database.
4. The multi-modal intelligent handling method based on one-network management of city event work orders according to claim 1 is characterized in that: The S4 specifically includes: S41, inputting the text feature vector and the image feature vector into a mapping module of a multimodal feature space respectively; S42, using a dot product operation to calculate the similarity between each pair of text vectors and image vectors, and generate a similarity matrix; S43. In each row of the similarity matrix, the Softmax function is used for normalization to highlight the main correlation between the text vector and the image vector: Among them, Softmax represents the normalized exponential function, S i,j represents the value of the i-th row and j-th column in the similarity matrix, exp represents the exponential function, N is the total number of categories, and k is the index of the category; S44, calculating a bidirectional loss function based on the normalized similarity matrix, and combining the loss function values for model training optimization; S45. Use the optimized model to generate a fusion feature vector, recalculate the weights of the text features and the image features through the fusion model, generate a multimodal fusion feature vector, and store the generated multimodal feature vector in the multimodal feature space: F multi =W T ·T+W W ·W; Where W T and W W is the weight matrix, T is the text vector, W is the image vector, F multi is the multimodal feature vector.
5. The multi-modal intelligent handling method based on one-network management of city event work orders according to claim 1 is characterized in that: The S5 specifically includes: S51, input the multimodal feature vector into the multimodal attention mechanism, calculate the internal correlation using the query matrix, key matrix and value matrix, and generate a preliminary weighted multimodal feature vector: Among them, Attention is a function for calculating attention distribution, softmax is a normalization function, and d k is the dimension of the key vector, Q is the query matrix, K is the key matrix, and K T represents the transpose operation of the key matrix K, V is the value matrix; S52, inputting the preliminarily weighted multimodal feature vector into the multi-head attention mechanism, capturing multi-dimensional context features through parallel processing, and concatenating and fusing the output results of multiple attention heads into a context-sensitive global feature vector; S53, inputting the global feature vector into the Bi-LSTM model, using the forward and backward LSTM units to process the past and future information of the multimodal feature sequence respectively, and generating a feature representation containing the complete context through a concatenation operation; S54, extract global features from the output of Bi-LSTM, and select the hidden state of the last time step as the global classification feature of the work order; S55. Input the global classification features into the Softmax function, calculate the classification probability distribution, normalize each category, and generate probability values belonging to different categories: Where P(y=c|h T ) represents the possibility that the input feature belongs to category C, h T is the global classification feature vector, is the weight matrix of the classifier, represents the matrix row vector, b k represents the bias term of category K, b c represents the bias term of category C, C is the total number of categories, and exp represents the exponential function; S56. According to the classification probability output by Softmax, the category with the highest probability is selected as the classification result, and the work order is assigned to the predefined disposal process according to the classification result, so as to complete the classification and work order flow operation; S57, using an offline training method to optimize the multimodal information vectorization representation and classification model through a joint loss function; S58. The classification model is trained through the Pytorch framework and then serialized and stored in the cloud server. When the user calls it, the model is restored to its original state through deserialization to complete multimodal feature classification and work order allocation.
6. The multi-modal intelligent handling method based on one-network management of city event work orders according to claim 1 is characterized in that: The S6 specifically includes: S61. Automatically generate work order handling time limit parameters according to the work order application time and expected completion time, and divide the time limit into primary, secondary and tertiary thresholds, which are set to half of the work order time limit respectively, initialize the countdown monitoring state, and enter the real-time monitoring process; S62. During the countdown monitoring process, when the countdown status of the work order approaches the time limit threshold, a reminder notification is automatically triggered, and a reminder message is sent to the relevant administrator through the one-stop SMS platform to urge him to speed up the handling progress of the work order; S63. During the work order handling process, the status information is monitored in real time, including successful handling, delayed and rejected status; for delayed work orders, the urging process is continued to be triggered; for successfully handled work orders, the completion status is recorded; for rejected work orders, an exception record is generated; S64. For rejected work orders, generate supervision tasks by analyzing the reasons for rejection, and assign them to the upper-level responsible person or designated processing personnel for collaborative handling in combination with the task list to be completed and processing suggestions; S65. After the work order is successfully handled, the handling result is notified to the work order applicant and administrator through the One-Network SMS platform or system message, including the work order number, completion time, handling result and operation suggestions; S66. Work orders that are not completed on time and are rejected multiple times are marked as abnormal work orders, and a report record is generated to record the abnormal cause and recommended processing plan, and report it to the system administrator or designated personnel for subsequent tracking and processing.
7. A multimodal intelligent handling system based on a one-network management of city event work orders, a multimodal intelligent handling method based on a one-network management of city event work orders according to any one of claims 1 to 6, characterized in that: Includes the following modules: The work order information mining module is responsible for extracting key information from the work order text and image information; The work order distribution module is used to fuse text and image feature vectors to generate multimodal feature vectors and automatically distribute work orders through the classification model; The problem acceptance module is used to monitor the status and time limit of work orders in real time and handle timeouts and abnormal situations; The classification result feedback and allocation module is used to provide feedback on the disposal results and automatically allocate the classified work orders; The model training and optimization module is used to optimize system model performance through offline training and deployment.