Customer service reply method and system of multi-modal fusion architecture, and storage medium

Abstract

The invention relates to the technical field of artificial intelligence, in particular to a customer service reply method and system of a multi-modal fusion architecture and a storage medium, and the method comprises the following steps: obtaining initial input data of a user, preprocessing the initial input data, and generating a standard data source; performing decision compliance analysis on the standard data source to generate a decision judgment result; and generating a corresponding decision scheme according to the decision judgment result. According to the invention, through a space-semantic double-constraint graph network, the technical defects of fragmentation and low standardization degree of multi-modal data identification by traditional intelligent customer service are effectively solved, and the problem of insufficient adaptation of a general identification technology to a government affair format is overcome; in a semantic constraint dimension, spatial features and a government and enterprise professional semantic dictionary are deeply fused based on a node association mechanism of a graph network, conversion from multi-modal data to structured policy elements is automatically completed, and the recognition accuracy is improved by more than 40% compared with that of a traditional NLP technology.

Description

TECHNICAL FIELD

[0001] The application relates to the technical field of artificial intelligence, in particular to a customer service reply method and system based on a multi-modal fusion architecture and a storage medium. BACKGROUND

[0002] With the deep promotion of digital government construction and enterprise digital transformation, artificial intelligence customer service has become a core supporting technology for optimizing service efficiency and reducing operation cost of governments and enterprises. In the government field, intelligent customer service construction has become a core task of digital government construction. In 2025, the intelligent customer service system accounted for more than 90% in the AI market of the Chinese government. In the enterprise field, the surge in consultation volume brought about by business expansion and the high demand of customers for service response speed have promoted the upgrading of traditional customer service mode to intelligent mode, and artificial intelligence customer service has become a key tool for enterprises to improve customer experience. The current technical implementation of artificial intelligence customer service mainly relies on natural language processing, machine learning, large models and other core technologies. Natural language processing technology realizes human-computer language interaction through word segmentation and semantic analysis. Large models have advantages in parameter scale and large-scale data training, and have achieved breakthroughs in language understanding and knowledge reasoning, providing possibilities for complex scenario question and answer. At the same time, knowledge graph technology helps to integrate structured and unstructured data in the government and enterprise fields, and constructs a specialized knowledge base to provide data support for accurate answering.

[0003] A Chinese invention patent with publication number CN119577459B discloses an intelligent customer service training method, device and storage medium of a multi-modal large model. The method of the patent comprises: collecting multi-modal data; performing data duplication checking on the multi-modal data; preprocessing the duplicate data; performing data enhancement on the preprocessed data; constructing an initialized multi-modal large model based on a Transformer architecture; training the initialized multi-modal large model based on the enhanced data to obtain a target multi-modal large model; obtaining user inquiry information; extracting weight text from the user inquiry information; inputting the weight text into the target multi-modal large model to obtain associated information; preprocessing the associated information to generate a feature vector; obtaining a similarity value between the feature vector and the weight text according to a similarity algorithm; arranging the similarity values in order to obtain a sorting result; extracting the sorting result; determining target associated information based on the target similarity value; and displaying the target associated information.

[0004] The above-mentioned patent has a blank in formula format and red head file recognition capability, and cannot accurately extract key information such as policy time limit, scope of application and responsible department. The patent scheme does not deploy a corresponding format analysis module and text structuring algorithm, so that when a government user uploads or quotes a policy file, the core provisions cannot be quickly located, which is out of line with the demand for "precise government services". SUMMARY

[0005] In order to solve the above technical problems existing in the prior art, the application provides a customer service reply method and system based on a multi-modal fusion architecture and a storage medium.

[0006] In order to achieve the above-mentioned purpose, the technical scheme of the application is as follows: In a first aspect, the application provides a customer service reply method based on a multi-modal fusion architecture, comprising the following steps: S1: obtaining initial input data of a user, preprocessing the initial input data, and generating a standard data source; S2: performing decision compliance analysis on the standard data source to generate a decision judgment result; S3: generating a corresponding decision scheme according to the decision judgment result.

[0007] Further, the step of preprocessing the initial input data to generate a standard data source is: determining the data type of the initial input data, and selecting a corresponding processing model for standardized data processing according to different data types.

[0008] Further, the step of performing decision compliance analysis on the standard data source to generate a decision judgment result is: dividing the standard data source into demand data and policy data; associating the demand data with the policy data, determining whether the demand data conforms to the policy data, and outputting the decision judgment result.

[0009] Further, the step of generating a corresponding decision scheme according to the decision judgment result is: if the decision judgment result is that the user's demand is met, then the demand data is optimized according to the policy data to generate an optimal decision scheme; if the decision judgment result is that the user's demand is not met, then the demand data is modified according to the policy data to generate a compliance decision scheme.

[0010] Further, the step of selecting a corresponding processing model for standardized data processing according to different data types is: if the data type is an image, a YOLOv7_doc_parser function is called for processing; if the data type is audio, a Conformer_ASR function is called for processing; if the data type is text, a BERT_entity_extractor function is called for processing.

[0011] Further, a fusion correlation model is used when associating the demand data with the policy data, and a construction step of the fusion correlation model is: Constructing a graph network model G: ; ; ; wherein V represents a node of an image, E represents an edge of an image, represents an nth text type node, represents an mth image node, represents an kth audio type node, represents a spatial position coordinate of a node p, represents a spatial position coordinate of a node q, represents semantic information of a node p, represents semantic information of a node q; Constructing a dynamic weight adjustment model: if the policy data is a tax policy, then the weight ratio of a text weight α, an image weight β and an audio weight is 0.6:0.3:0.1; if the policy data is a subsidy policy, then the weight ratio of a text weight α, an image weight β and an audio weight is 0.3:0.5:0.2; if the policy data is other policy, then the weight ratio of a text weight α, an image weight β and an audio weight is 0.4:0.4:0.2; integrating the graph network model and the dynamic weight adjustment model to generate the fusion correlation model :

[0012] wherein, represents a position mask matrix of an image.

[0013] Further, a conflict detection model is used for the judgment method of the decision judgment result, and a construction step of the conflict detection model is: Conflict detection model C:

[0014] wherein, represents a version validity indicator, Wi represents a weight, R represents a user demand, Pi represents an ith policy data, and n represents a total number of policy data; if the policy data is a central policy, Wi takes a value of 1.0; if the policy data is a non-central policy, Wi takes a value of 0.6.

[0015] Further, the method further comprises a step S4 of sending the decision scheme to a policy declaration terminal, and uploading relevant materials according to a process regulation of the policy declaration terminal until the policy declaration is completed.

[0016] In a second aspect, the application provides a customer service reply system based on a multi-modal fusion architecture, comprising a memory, a processor and a computer program stored in the memory, and the processor executes the computer program to implement the steps of the customer reply method.

[0017] In a third aspect, the application provides a computer readable storage medium, and the computer readable storage medium stores a computer program, and the computer program is executed by a processor to implement the steps of the customer reply method.

[0018] Compared with the prior art, the application has the following beneficial effects: The customer service reply method based on a multi-modal fusion architecture provided by the application effectively solves the technical defects of fragmented multi-modal data recognition and low standardization of traditional intelligent customer service by using a space-semantic double constraint graph network (SS-DCGN). In the space constraint dimension, the space topology structure of data is constructed to extract the space features of multi-modal data frequently occurring in the government-enterprise-scene, such as picture policy files (such as red-head file scans and business form screenshots), voice consultation recordings and text appeals, accurately locate the key areas such as the title of the document, the document number and the clause paragraph, and overcome the problem of insufficient adaptation of general recognition technology to government formats; in the semantic constraint dimension, the space features are deeply fused with the government-enterprise professional semantic dictionary based on the node association mechanism of the graph network, and the conversion of multi-modal data to structured policy elements (such as the applicable subject, the policy time limit and the core requirements) is automatically completed, and the recognition accuracy is improved by more than 40% compared with traditional NLP technology.

[0019] In addition, in the decision guidance process, the system generates suggestions based on structured policy data and traceable comparison logic throughout the process, ensuring that each decision guidance has a clear policy basis, which improves the transparency and credibility of government services. In addition, the standardized data processing process can directly interface with the data hierarchical desensitization engine, which is compatible with GB / T 39725-2020 government data security requirements, improving service efficiency while ensuring data security, and realizing the unity of technical value and compliance value. BRIEF DESCRIPTION OF DRAWINGS

[0020] Figure 1 The figure is a schematic diagram of the modules of the application. DETAILED DESCRIPTION

[0021] The technical solutions of the present application will be clearly described below with reference to the drawings. Obviously, the described embodiments are not all the embodiments of the present application, and all other embodiments obtained by those of ordinary skill in the art without creative efforts fall within the protection scope of the present application.

[0022] It should be noted that, unless otherwise specified, the relative arrangement of components and steps, numerical expressions set forth in these embodiments should not be understood as limiting the scope of the present application.

[0023] The following description of exemplary embodiments is merely illustrative in nature and is in no way intended to limit the application or its application or use. Techniques, methods, and devices known to those of ordinary skill in the relevant art may not be discussed in detail herein, but where applicable, such techniques, methods, and devices should be considered as part of this specification.

[0024] As shown in Figure 1 The present application provides a multi-modal fusion architecture customer service reply system, which includes a user interaction layer, a multi-modal fusion analysis engine, a dynamic knowledge hub, a compliance decision engine, a policy atlas database, a cross-domain collaborative executor, a transaction flow knowledge base, and a government affairs system interface group.

[0025] The user interaction layer is used to obtain the initial input data of the user and transmit the initial input data to the multi-modal fusion analysis engine. In the multi-modal fusion analysis engine, when the input data type is IMAGE (image, such as scanned copy, screenshot, etc.), the YOLOv7_doc_parser function is called for processing. This function is based on the YOLOv7 target detection algorithm and is specially used for document analysis. It outputs JSON data containing table coordinates and seal positions, achieving the positioning and extraction of key visual elements in images.

[0026] When the input type is AUDIO (audio, such as voice consultation), the Conformer_ASR function is called. This function uses the Conformer model for speech recognition and specifically specifies the dialect as Cantonese, converting Cantonese speech into standard term text to solve the problem of dialect recognition and standardization.

[0027] For other types of input (presumably text type, such as user input text appeal, policy document text, etc.), the BERT_entity_extractor function is called. This function uses the BERT model for entity extraction, extracting key entities related to policies (such as policy name, applicable conditions, responsible department, etc.). The running logic of multi-modal input standardization is as follows: def multimodal_normalization(input): if input.type == "IMAGE": return YOLOv7_doc_parser(input) # Output JSON {"Table coordinates": (x1, y1, x2, y2)", "Official seal location", (x, y)"} elif input.type == "AUDIO": return Conformer_ASR(input, dialect="cantonese") # Convert Cantonese to Standard Terminology else: return BERT_entity_extractor(input) # Extract policy entities Cross-modal correlation modeling, constructing a graph network model G: ; ; ; Where V represents a node in the image, and E represents an edge in the image. This represents the nth text type node. This represents the m-th image node. This represents the k-th audio type node. This represents the spatial coordinates of node p. This represents the spatial coordinates of node q. Represents the semantic information of node p. Represents the semantic information of node q; Construct a dynamic weight adjustment model: If the policy data is a tax policy, then the text weight α, image weight β, and audio weight are... The weighting ratio is 0.6:0.3:0.1; If the policy data refers to subsidy policies, then the text weight α, image weight β, and audio weight are... The weighting ratio is 0.3:0.5:0.2; If the policy data is other policies, then the text weight α, image weight β, and audio weight are... The weight ratio is 0.4:0.4:0.2; the operating logic of the dynamic weight adjustment algorithm is as follows: function [α,β, ]= adjust_weights(policy type) switch policy_type case "TAX_POLICY" a = 0.6; b = 0.3; = 0.1 / / text dominant case "SUBSIDY APPLICATION" a = 0.3; b = 0.5; = 0.2 / / image dominant otherwise a = 0.4; b = 0.4; = 0.2 / / balanced mode end end integrating the graph network model with the dynamic weight adjustment model to generate the fusion correlation model :

[0028] wherein, represents the position mask matrix of the image. The final multi-modal fusion analysis engine divides the initial input data into demand data and policy data, and transmits the demand data and policy data to the dynamic knowledge hub.

[0029] The dynamic knowledge hub transmits the demand data to the compliance decision engine, and stores the policy data as a knowledge graph in the policy graph database, facilitating quick queries of the same policy in the future. An example of a policy graph is as follows: { "node": { "id": "POL-2024-086", "type": "central regulations", "clause": "R&D expenses of technology-based small and medium-sized enterprises are added by 150%": "effective date": "2024-01-01", "conflict relationship": ["CONFLICT: LOC-2023-772"] } } The dynamic knowledge hub also includes a dynamic updating mechanism that uses a crawler engine to monitor the latest policy changes on government websites and update the content of the policy graph and the process of the transaction flow knowledge base. The running logic of the dynamic updating mechanism is as follows: sequenceDiagram Government website ->+ Crawler engine: Monitor policy updates; Crawler engine -> BERT-PolicyDiff: New policy text; BERT-PolicyDiff ->>> Knowledge Hub: Change marker (ADD / MODIFY / DELETE); Knowledge Hub ->>> Transaction Flow Knowledge Base: Trigger associated transaction update.

[0030] The transaction flow knowledge base stores enterprise opening transaction flows for querying related processes of policy declaration. An example of a transaction flow is as follows: Step 1: Department: Business Bureau; API: biz / register; Input: ["Enterprise name", "Registered capital"] Output: ["Unified credit code"]; Timeout: 120s.

[0031] Step 2: Department: Public Security Bureau; API: seal / apply; Precondition: $step1.output.credit_code!= null.

[0032] Exception handling: Error code: ERR TAX TIMEOUT; Retry strategy: [interval 10s, retry 3 times]; Alternative: transfer to manual agent.

[0033] The compliance decision engine performs compliance judgment on the requirement data and the corresponding policy graph to generate a decision scheme. It queries the transaction flow corresponding to the decision scheme and transmits the data to the cross-domain collaborative executor.

[0034] The compliance judgment uses a conflict detection model C:

[0035] wherein, represents a version validity indicator (currently valid = 1, abolished = 0), Wi represents a weight, R represents user requirements, Pi represents the i-th policy data, and n represents the total number of policy data; If the policy data is a central policy, Wi is 1.0; if the policy data is a non-central policy, Wi is 0.6. The operation logic of Wi is as follows: def weight_calc(policy level): return 1.0 if policy_level==“CENTRAL” else 0.6 The logic for generating a decision scheme is as follows: generate plan(company): industry(company, Industry), region(company, Region), findall(step, eligible_step(Industry, Region, step), steps), Optimize_order(steps) The generated decision-making plan is shown in Table 1: Table 1: Generated decision-making plan

[0036] The cross-domain collaborative executor executes the transaction flow corresponding to the decision-making plan, interfaces with the government system interface group, and performs policy declaration. The policy declaration uses the process model PN: ; ; ; Where F represents the arc set, Q represents the state or condition in the system, T represents the event or operation occurring in the system, represents the initial marking.

[0037] The running logic of the process model PN is as follows: while(is_final_marking(m)){ Enabled_transitions = get_enabled(T, M); Fire_transition(select(enabled transitions)); if(api_timeout>threshold){trigger_rollback(); activate backup path();} } The cross-domain collaborative executor also includes an exception handling mechanism, which indicates errors in the policy declaration process through error codes and makes corresponding processing strategies. An example of error code mapping is shown in Table 2: Table 2: Error code mapping example

[0038] Example 1: Establishment of foreign high-tech enterprises Step 1: Multimodal material analysis.

[0039] (1) Input: Scan: Page 12 of the "Foreign Investment Access Negative List (2024 Edition)" (including the prohibited industry table); Voice: "Our company plans to invest in quantum computing research and development. Is it a restricted category?"

[0040] (2) Analysis process: Image analysis: YOLOv7 locates the "quantum information technology" cell in the "restricted category" column of the table; Voice translation: Conformer identifies "quantum computing" as a subcategory of "quantum information technology"; Semantic fusion: constructs an association edge (voice node → table cell, weight = 0.93).

[0041] Step 2: Compliance decision engine response.

[0042] (1) Conflict detection: The company's investment direction "quantum computing" is in the "restricted category" of the negative list; Trigger orange warning (conflict index C = 0.72).

[0043] (2) Generate solutions: "action": "Adjust investment proportion to <50%", "steps": [ {"department": "Ministry of Commerce", "operation": "Foreign investment filing (holding 49%)"}, {"department": "Science and Technology Bureau", "operation": "Special R&D qualification approval"} ], "risk note": "Need to submit the \"Technical Safety Commitment\".

[0044] Step 3: Cross-domain collaborative execution.

[0045] (1) API scheduling process: System >> + Ministry of Commerce API: submit foreign investment filing; Ministry of Commerce API >> - System: return filing number FDI2024-08756; System >> + Foreign Exchange Administration Bureau: capital account pre-registration (filing number); Foreign Exchange Administration Bureau >> - System: generate account number CNY88***; System >> enterprise email: send material list (including English company charter template).

[0046] Exception handling: Foreign exchange registration timeout triggers automatic retry (3 times, 10 seconds interval).

[0047] Embodiment 2: Application for environmental protection technical transformation subsidy of manufacturing industry Step 1: Intelligent verification of multi-modal materials.

[0048] (1) Input: EIA report PDF (including pollutant emission data table); Equipment list Excel (air compressor model: SA-2300).

[0049] (2) Analysis process: PDF table extraction: identify SO2 emission value 35 mg / m³ (lower than national standard 50 mg / m³); Equipment compliance verification: knowledge hub searches "Energy-saving Technology Equipment Directory"; confirms that SA-2300 is a national promotion model (energy efficiency level 1).

[0050] Step 2: Dynamic policy matching.

[0051] (1) Real-time detection changes: Crawler captures "New Rules for Technical Transformation Subsidy" of a province: "Enterprises with SO2 <40 mg / m³ are subsidized by 10%"; Automatic update transaction flow: add "Environmental Protection Standard Achievement Award" application node.

[0052] Generate optimal path: Original path: technical transformation record → equipment verification → subsidy application; New path: technical transformation record → environmental protection standard recognition → equipment verification → subsidy application (+10%).

[0053] Step 3: Risk control and execution.

[0054] (1) Compliance verification: Calculate equipment investment (5 million) <25% of enterprise net assets (20 million) → meet safety threshold.

[0055] (2) Cross-system execution: Call the API of the Ministry of Industry and Information Technology to submit the equipment list (second-level feedback); Synchronize data from the Ecological Environment Bureau: automatically fill in the EIA report number in the subsidy application form.

[0056] The above specific embodiments are only used to illustrate the technical solutions of the present application and not to limit. Although the present application has been described in detail with reference to the examples, it should be understood by those skilled in the art that the technical solutions of the present application can be modified or replaced by equivalents without departing from the scope of the technical solutions of the present application, which should be covered by the scope of the claims of the present application.

Claims

1. A customer service reply method of a multi-modal fusion architecture, characterized in that, The method comprises the following steps: S1: obtaining initial input data of a user, preprocessing the initial input data, and generating a standard data source; S2: performing decision compliance analysis on the standard data source, and generating a decision judgment result; S3: generating a corresponding decision scheme according to the decision judgment result.

2. The method of claim 1, wherein the method further comprises: The step of preprocessing the initial input data to generate a standard data source is: judging the data type of the initial input data, and selecting a corresponding processing model for standardized data processing according to different data types.

3. The method of claim 1, wherein the method further comprises: The step of performing decision compliance analysis on the standard data source to generate a decision judgment result is: dividing the standard data source into demand data and policy data; associating the demand data with the policy data, judging whether the demand data meets the policy data, and outputting the decision judgment result.

4. The method of claim 3, wherein, The step of generating a corresponding decision scheme according to the decision judgment result is: if the decision judgment result is that the user's demand is met, then optimizing the demand data according to the policy data to generate an optimal decision scheme; if the decision judgment result is that the user's demand is not met, then modifying the demand data according to the policy data to generate a compliance decision scheme.

5. The method of claim 2, wherein, The step of selecting a corresponding processing model for standardized data processing according to different data types is: if the data type is an image, then a YOLOv7_doc_parser function is called for processing; if the data type is audio, then a Conformer_ASR function is called for processing; if the data type is text, then a BERT_entity_extractor function is called for processing.

6. The method of claim 3, wherein the method further comprises: A fusion correlation model is used when associating the demand data with the policy data, and the construction steps of the fusion correlation model are: constructing a graph network model G: wherein V represents a node of the image, E represents an edge of the image, represents an nth text type node, represents an mth image node, represents an kth audio type node, represents a spatial position coordinate of node p, represents a spatial position coordinate of node q, represents semantic information of node p, represents semantic information of node q; constructing a dynamic weight adjustment model: if the policy data is tax policy, the weight ratio of text weight α, image weight β and audio weight γ is 0.6:0.3:0.1 ; If the policy data is a subsidy policy, the weight ratio of the text weight a, the image weight β and the audio weight is 0.3:0.5:0.2; If the policy data is neither a tax policy nor a subsidy policy, then the weight ratio of the text weight a, the image weight β, and the audio weight is 0.4:0.4:0.2; Integrating the graph network model with a dynamic weight adjustment model to generate the fusion correlation model : wherein, represents a position mask matrix of the image.

7. The method of claim 4, wherein, The decision judgment result uses a conflict detection model, and the construction steps of the conflict detection model are: a conflict detection model C: wherein, denotes a version validity indicator, Wi denotes a weight, R denotes a user demand, Pi denotes the i-th policy data, and n denotes the total number of policy data; if the policy data is a central policy, then Wi is 1.0; if the policy data is a non-central policy, then Wi is 0.

6.

8. The method of claim 1, wherein the method further comprises: It also includes step S4: sending the decision scheme to a policy declaration terminal, uploading materials according to the process regulations of the policy declaration terminal, and completing the policy declaration.

9. A customer service reply system of a multi-modal fusion architecture, comprising a memory, a processor and a computer program stored on the memory, characterized in that, The processor executes the computer program to implement the customer reply method of any one of claims 1-8.

10. A computer-readable storage medium having stored thereon a computer program, characterized in that The computer program is executed by the processor to implement the steps of the customer reply method of any one of claims 1-8.

Images