Precise livelihood work order distribution method fusing multi-modal semantic understanding

Abstract

The invention relates to the technical field of livelihood service, and particularly discloses an accurate work order distribution method fusing multi-modal semantic understanding. A semantic vector is generated by an extraction module through multi-modal features including texts, voices, images and the like; the cross-modal alignment module uses a dynamic attention weight distribution mechanism, and the formula is alpha k = softmax (Wk. [Modaliyk] + bk) to fuse features; the semantic portrait generation module constructs structure, time, space and semantic four-dimensional decision features; the dynamic matching engine adopts a near-end strategy optimization algorithm to calculate a department matching probability; the exception handling module is based on a confidence threshold (Confuse; and 0.6) triggering man-machine collaborative auditing. The distribution accuracy is 93.2% and is improved by 24.3%, and the manual intervention rate is reduced to be below 5%.

Description

TECHNICAL FIELD

[0001] The application relates to application of artificial intelligence in government service, in particular to a work order dispatching system based on multi-modal semantic understanding and reinforcement learning decision. BACKGROUND

[0002] Existing livelihood work orders, such as municipal complaints, water and electricity faults, etc., rely on manual classification and dispatch, which is low in efficiency and high in error rate, about 15%-20%. Traditional text classification models cannot understand multi-modal information such as voice description, on-site pictures, etc., resulting in inaccurate matching of work orders and processing departments.

[0003] According to the 2025Q1 National Government Service Efficiency Evaluation Report, the existing technology causes two systemic defects: 1. Time loss: the average response time is 120 minutes, and more than 30% of urgent work orders are delayed due to dispatch errors; 2. Resource waste: the proportion of manual review is close to 100%, and municipal departments increase operating costs by about 1.2 billion yuan per year.

[0004] Therefore, there is an urgent need for a work order dispatching method that can deeply integrate text, voice, image multi-modal semantics and has dynamic decision-making capability to solve the cross-modal understanding deficiency and rigid dispatch mechanism in the prior art. SUMMARY

[0005] In view of the above existing problems, the application is proposed.

[0006] The application proposes a multi-modal semantic fusion-dynamic decision dispatching framework to realize accurate dispatching through the following steps: multi-modal feature extraction: synchronously analyzing work order text, voice, image information; cross-modal semantic alignment: mapping different modal information to a unified semantic space; reinforcement decision matching: dynamically optimizing dispatch rules combined with historical work order data. The specific processing flow is as follows:

[0007] Multi-modal feature extraction

[0008]

[0009]

[0010] Step 2: Cross-modal semantic fusion

[0011] The multi-modal semantic fusion method based on dynamic attention mechanism aims to solve the semantic fragmentation problem caused by isolated processing of text, voice, image information in traditional work order systems. The specific implementation is divided into three levels:

[0012] Modal feature vectorization: Text features are encoded into 768-dimensional semantic vectors by a pre-trained language model (e.g., BERT-wwm), capturing key entities such as "cable failure" and intent labels including urgency and responsible parties. Speech features, after being converted by ASR, are processed by an acoustic feature extractor (e.g., OpenSMILE) to generate emotion intensity vectors, specifically numerical representations of anger and anxiety, and are concatenated with transcribed text vectors to form 256-dimensional hybrid features. Image features are extracted by a ResNet-50 convolutional layer to capture spatial features, combined with instance segmentation results from Mask R-CNN, such as identifying the proportion of "water pipe corrosion area," resulting in 1024-dimensional visual semantic vectors.

[0013] Dynamic weight distribution mechanism: A trainable multi-layer perceptron (MLP) weight network is designed to automatically learn the contribution coefficients of each modality based on ticket type. Taking device failure tickets as an example:

[0014] Image feature weight (α_image) is increased to 0.65-0.75 (as visual information directly reflects the damage level)

[0015] Speech emotion weight (α_audio) is reduced to 0.1-0.2 (low relevance of emotions in device failure)

[0016] Text description weight (α_text) remains at 0.15-0.25 (used to locate the model of the faulty device)

[0017] The weight calculation follows the formula:

[0018] α k = softmax(W k ·[Modalityk]+b k ), k ∈ {text, audio, image}

[0019] where W k is a trainable parameter matrix optimized through backpropagation.

[0020] Cross-modal semantic alignment: In the feature fusion stage, a cross-modal contrastive loss function is introduced forcing similar samples of different modalities to be close in vector space:

[0021]

[0022] positive sample pairs for tickets of the same type, such as text and image describing the same "water pipe break" event;

[0023] negative sample pairs, such as "water pipe break" text and "power failure" image;

[0024] delta is the boundary threshold (default 0.5). This loss ensures that the "leakage" speech description is within a distance of 0.1 from the leakage picture feature, significantly improving modality consistency.

[0025] Fused feature output: the weighted modality feature vectors are element-wise added to generate a 128-dimensional unified semantic vector:

[0026] v fused = alpha text · v text + alpha audio · v audio + alpha image · v image

[0027] This vector encodes multi-modal complementary semantics as input for the work order semantic portrait.

[0028] Technical advantages compared to traditional solutions: compared to fixed weight addition, the dynamic weight mechanism improves the dispatch accuracy of complex scenarios such as "night construction noise + broken roadblock picture" by 19.2%. The cross-modal alignment loss function reduces semantic ambiguity error rate by 42%, avoiding misjudgment of "cable sag" picture as "tree collapse" related work order.

[0029] Innovations:

[0030] Dynamic weight distribution: adaptively adjust modality importance according to work order type, e.g. device failure -> image weight 0.7, complaint -> speech weight 0.6

[0031] Semantic alignment loss function:

[0032]

[0033] Ensure consistency of different modalities in vector space, e.g. "leakage" text and "leakage" picture feature distance <= 0.1

[0034] Step 3: Work order semantic portrait generation

[0035] After completing cross-modal semantic fusion, the system enters the work order semantic portrait generation phase, which is the core knowledge representation layer for precise dispatch. This phase adopts a three-level generation architecture:

[0036] Multi-modal feature vectorization

[0037] Construct the portrait base through the 128-dimensional fused feature vector vfused, where:

[0038] Semantic label predictor (Bi-GRU classification network) analyzes entity, attribute and scene information in feature vector

[0039] The key label generation follows the "three-level priority principle": physical entities, such as "water pipes"> state attributes, such as "breakage"> spatiotemporal features, such as "night", and the feature space optimized by Lalign loss ensures that visual "rust" and textual "corrosion" are within a vector distance of ≤0.2

[0040] Scenario image modeling

[0041] Based on multi-modal collaborative representation: spatial modeling: use graph attention network (GAT) to analyze visual spatial relationships, such as "road collapse-0.8m deep" manhole cover location; temporal modeling: combine speech timestamp features to mark event emergency gradient, such as: speech acceleration>2.5m / s 2 Trigger red alert; causal modeling: knowledge graph embedding technology establishes fault links, such as "cable sagging→contacting trees→short circuit risk"

[0042] Dynamic image optimization

[0043] The generation process adopts a triple feedback mechanism: confidence reinforcement: when ∣∣Modalityi-Modalityj∣∣22>1.2, trigger cross-modal verification, call pre-trained contrast model to recalibrate; historical mapping: match the historical similar work order library through k-nearest neighbor algorithm, automatically supplement potential treatment schemes, trigger when the similarity is>85%; image evolution: deploy online incremental learning module, update label predictor parameters every 24 hours, image accuracy improves with system running, actual measurement +2.1% per month.

[0044] The final generated semantic image has four-dimensional decision features: structural dimension: JSON-LD format image contains 8 decision fields; time dimension: real-time labeling processing time window, such as "water and electricity failure: response<2h"; spatial dimension: high-precision map coordinate binding + impact range radius calculation; semantic dimension: through cross-modal alignment loss constraint, ensure that text "water accumulation", picture "water hole", speech "flooded" share the same semantic slot. The feedback JSON structure is as follows:

[0045]

[0046] Step 4: Dynamic matching engine

[0047] The dynamic matching engine uses the proximal policy optimization (PPO) algorithm to build the core decision framework of the dynamic matching engine, and realizes the optimal matching of work orders and processing departments through reinforcement learning mechanism. Its technical implementation contains four key design levels:

[0048] 1. State space modeling

[0049] Construct joint state space s with work order semantic image and real-time department load t :

[0050] Ticket representation: encode multi-modal fusion features, e.g. fault type, urgency level, spatial coordinates, into 128-dimensional decision vector

[0051] Department load: dynamically capture processing capacity of each agency, e.g. load weight wd=0.8 when city management department has >50 pending tickets

[0052] Normalization: Z-score standardization on heterogeneous data such as ticket priority, department saturation, to eliminate dimension differences

[0053] 2. Action space design

[0054] Define discrete action set at∈{water bureau, power grid, city management, environmental protection, …}, innovatively introduce departmental collaborative action:

[0055] Single-department assignment: regular tickets are directly assigned to the responsible department, e.g. "cable repair" → power grid;

[0056] Cross-department joint action: complex scenarios trigger collaborative mechanisms, e.g. "road collapse + gas leak"

[0057] Simultaneous assignment to municipal and gas companies;

[0058] Action space size is reduced from the original 200+ departments to 35 decision units through department clustering compression technology, improving training efficiency by 40%

[0059] 3. Piecewise reward function (core innovation)

[0060] Design a three-level reward and punishment mechanism to drive policy optimization:

[0061]

[0062] Innovative design points:

[0063] Time efficiency reward rspeed decays exponentially, forcing the engine to prioritize 3-hour golden window period tickets

[0064] Penalty coefficient (-0.5) for returning back, which can reduce 20% of responsibility shirking through Monte Carlo simulation verification

[0065] Success reward (+1.0) aligns with municipal KPI assessment, requiring a response resolution rate of ≥95%

[0066] 4. Three-stage training protocol

[0067] Stage 1: Network initialization

[0068] Build Actor-Critic dual network architecture:

[0069] Actor output department selection probability distribution πθ(a∣s)

[0070] Critic evaluates state value Vφ(s)

[0071] Network parameters θ,φ use Xavier initialization and orthogonal regularization phase 2: offline pre-training

[0072] Sample 100,000 labeled data from the historical ticket library

[0073] Training key parameters:

[0074] batch_size = 512,

[0075] clip_epsilon = 0.2, # PPO clipping parameter

[0076] gamma = 0.99, # discount factor

[0077] entropy_coef = 0.01 # exploration incentive

[0078] Pre-training goal: baseline accuracy > 82%

[0079] Phase 3: online learning

[0080] Incremental update mechanism: update network parameters every 1000 new tickets

[0081] Data backflow design:

[0082] Department processing results automatically generate <s,a,r,s'> transition tuples

[0083] Priority replay buffer (PER) stores high-value samples, such as r < 0 returned tickets

[0084] Safety update constraint: KL divergence threshold δ < 0.05 to prevent policy shock

[0085] Step 5: exception handling and manual intervention

[0086] In abnormal handling and manual intervention, a two-level fault-tolerant processing mechanism is constructed to ensure the efficiency of automation and solve complex abnormal situations through artificial intelligence collaboration. The core operation process is as follows:

[0087] 1. Dynamic confidence threshold

[0088] Confidence calculation principle:

[0089] Based on the entropy value fluctuation of the multi-modal fusion vector, the decision reliability is quantified to generate a confidence score:

[0090]

[0091] where v nearest is the nearest neighbor vector in the historical successful work order feature library, max_dist is the maximum distance threshold in the feature space, which is optimized to 0.6 through grid search.

[0092] Trigger rules:

[0093] When Conf fused <0.6, for example, the multimodal feature conflict rate is >40%, the system automatically freezes the dispatch process, and the work order is handed over to the manual review queue.

[0094] 2. Human-computer collaborative review mechanism

[0095] Three-channel collaborative decision-making:

[0096]

[0097] Review timeliness control: through dynamic priority scheduling, the response delay of an urgent work order is <3 minutes.

[0098] 3. Closed-loop feedback optimization system

[0099] In the subsequent dispatching results, if there is data anomaly, the accuracy of the feedback results can be enhanced through manual correction, and five tuples are generated as follows:

[0100]

[0101] Incremental learning strategy:

[0102] High-priority samples (retrain_priority>0.7) are injected into the training pool in real time

[0103] Every 500 corrected data triggers model fine-tuning:

[0104]

[0105] Where λ=1.2 to strengthen the punishment for the labeled error type.

[0106] The present application has the following beneficial effects:

[0107] Indicators Traditional text classification schemes The present patent solution Lifting effect Work order dispatch accuracy 75％ 93.2％ ↑24.3％ Mean response time 120 minutes 8.5 minutes ↓93％ Cross-departmental collaboration error rate 18％ 3.7％ ↓79.4％ Manual intervention rate 100％ Below 5% ↓95％ BRIEF DESCRIPTION OF DRAWINGS Figure 1 It is a flowchart of an intelligent work order processing system based on multimodal semantic understanding.

Claims

1. A method for precise dispatch of public service orders integrating multimodal semantic understanding, characterized in that... The technical steps are executed in the following order: Multimodal feature extraction: The BERT-wwm pre-trained language model is used to extract 768-dimensional semantic vectors from the work order text, and the BiLSTM+CRF model is used to identify key entities; the Conformer end-to-end speech recognition model is used to parse the work order speech information, and the OpenSMILE tool is used to quantify acoustic features such as speech rate and pitch, as well as emotion intensity coefficients; the ResNet-50 convolutional neural network and Mask R-CNN instance segmentation model are used to extract image visual features and identify spatial relationships between objects. Cross-modal semantic alignment: A dynamic attention weight allocation mechanism is designed, which generates the modal weight coefficients α through a trainable multilayer perceptron. k Construct a cross-modal contrastive loss function Force the Euclidean distance between cross-modal features of the same event in the vector space to be ≤0.1; output a 128-dimensional fused feature vector v. fused =α text ·v text +α audio ·v audio +α image ·v image ; Work order semantic profile generation: based on v fused Construct a three-level semantic tagging layer: physical entity layer, such as water pipes and cables; status attribute layer, such as rust area ≥ 80%; and spatiotemporal feature layer, such as GPS coordinates and timeliness window. Generate a structured profile in JSON-LD format, containing four-dimensional decision features: structural dimension: 8 semantic fields; time dimension: response timeliness requirements; spatial dimension: GIS coordinate binding; and semantic dimension: cross-modal slot alignment. Dynamic matching decision-making: A department matching policy network is established using a proximal policy optimization reinforcement learning algorithm; a piecewise reward function R(s,a) is defined: successful distribution of reward R +1 (Resolved within 3 hours), department returns penalty R -0.5 Time-based decay reward r speed =0.3×e -t / 180 ; Anomaly handling mechanism: When the confidence level of fused features... When the time is right, a three-channel manual review is triggered; a five-tuple feedback data {error_type, correct_label, model_vs_human, new_features, retrain_priority} is generated for closed-loop optimization.

2. The method as described in claim 1, wherein the dynamic weight allocation is characterized by: In the dynamic attention weight allocation mechanism, α k The coefficients are adaptively adjusted by the MLP network based on the work order type: for equipment failure work orders, such as burst water pipes or sagging cables, the image modality weights are prioritized and α is set. image ∈[0.65,0.75]; For complaint-type work orders (such as noise pollution, street vending), the voice modality weight is increased first, and α is set. audio ∈[0.55,0.65]; Text modal weight α text As a basic semantic carrier, according to α text =1-α audio -α image Dynamic calculation.

3. Cross-modal alignment loss, as described in claim 1, characterized in that: The cross-modal contrast loss function is defined as follows: in: It is a set of positive sample pairs, containing cross-modal data describing the same event, such as text descriptions of "water leakage" and pictures of the water leakage site; It is a set of negative sample pairs containing semantically unrelated cross-modal data, such as the text "water leak" and the image "power failure"; δ is the boundary threshold, which is set to 0.5 after grid search optimization.

4. A reinforced learning reward mechanism, as described in claim 1, characterized in that... The reward function R(s,a) of the PPO reinforcement learning includes a three-level reward and punishment mechanism: Time-saving bonus: R speed =0.3×e -t / 180 Where t is the number of minutes of work order response delay, the exponential decay design forces the system to prioritize work orders during the 3-hour golden window period; Refund penalty: R is triggered when a work order is returned by the department. -0.5 At the same time, the departmental collaborative action compensation mechanism is activated, such as automatically transferring the "road collapse + gas leak" work order to the municipal and gas companies for joint handling; Cooperative action rewards: For work orders that trigger cross-departmental joint processing, R is given +1.3 The bonus, verified by Monte Carlo simulations, shows that this value can reduce the collaborative error rate by 79.4%.

5. Closed-loop feedback optimization, as described in claim 1, characterized in that: The closed-loop feedback optimization includes: The quintuple data structure includes: `error_type`: labeling the type of error identified by the human, including modality comprehension bias, outdated departmental knowledge base, and misjudgment of urgency; `model_vs_human`: recording the difference vector between the model's predicted probability and the human's decision probability; and `retrain_priority`: retraining weights calculated based on the severity of the error type, specifically calculated as: 0.4 × I. 紧急 +0.6×I 跨部门 ; Incremental learning trigger condition: When retrain_priority > 0.7, the training pool is injected in real time and parameter updates are triggered. Where η is the learning rate, and I(error_type) is the penalty coefficient for the error type; Security update constraints: Verify KL divergence D after parameter update KL (π old ||π new <0.05, to prevent strategy network oscillation.

Images