A multi-modal virtual digital human interaction method for intelligent customer service

Abstract

The application discloses a kind of multi-modal virtual digital human interaction methods for intelligent customer service, it is related to artificial intelligence interaction technical field, the method receives the business material picture and user consultation content in current session, material state semantics is generated by parsing business material picture, and the user appeal semantics is generated by being tokenized to consultation content;Material state semantics and user appeal semantics are encapsulated as semantic unit sequence, the attention weight of each semantic unit is calculated, and action type configuration is formed by organizing and registering item to form semantic registration sequence, and material appeal semantic editing fragment is formed after sorting;Round mark is generated and fragment evaluation result is formed, evidence acceptance action is screened out to form digestion unit, and backfill position and execution mode are determined;Editing action and evidence phrase are verified and evidence explanation or artificial connection evaluation conclusion is output, and then material appeal semantic editing fragment is recalculated according to execution mode, and virtual digital human is controlled to output connection prompt or interaction completion prompt.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence interaction technology, specifically to a multimodal virtual digital human interaction method for intelligent customer service. Background Technology

[0002] The field of artificial intelligence interactive technology is based on speech recognition, natural language processing, image recognition, semantic understanding, and human-computer interaction control, enabling machines to understand, analyze, and provide feedback based on user input. With the continuous expansion of online business processing, remote consultation, and automated question-and-answer scenarios, the field of intelligent customer service is gradually shifting from simple text-based question and answer to a comprehensive service model combining voice, image, business knowledge, and interactive processes. Building on this, multimodal virtual digital human interaction has been introduced into intelligent customer service scenarios. Through virtual avatars carrying voice, facial expressions, actions, and text feedback, users can obtain an interactive experience closer to that of a human customer service representative when uploading business materials images and making inquiries.

[0003] In multimodal virtual digital human interaction scenarios, user inquiries often simultaneously include images of business materials and text or voice requests. Existing processing methods typically focus on recognizing text in images or understanding the user's question itself, with less emphasis on unified analysis of material status, user requests, evidence phrases, and subsequent interactive actions. Since status information, result information, and evidence fragments in business materials are distributed across different text blocks, and user requests may include different intentions such as processing, inquiry, review, and supplementation, relying solely on fixed question-and-answer templates or single-round semantic matching can easily lead to problems such as disconnected responses from material evidence, unclear supplementary questions, inaccurate status verification triggers, and disjointed processing guidance, thus affecting the interaction quality of virtual digital humans in intelligent customer service. Summary of the Invention

[0004] To address the shortcomings of existing technologies, this invention provides a multimodal virtual digital human interaction method for intelligent customer service, solving the problems mentioned in the background technology.

[0005] To achieve the above objectives, the present invention provides the following technical solution: a multimodal virtual digital human interaction method for intelligent customer service, comprising: Receive business material images and user inquiry content from the current session, parse the business material images to generate material status semantics, and perform word segmentation on the inquiry content to generate user request semantics; The semantics of material status and user demands are encapsulated into a sequence of semantic units. After semantic encoding and linear mapping, the attention weight of each semantic unit is calculated and registration items are organized to form a semantic registration sequence. The action type configuration is performed on the semantic registration sequence to obtain the editing action record. After sorting, the material demand semantic editing fragment is formed. Read the material requests semantic fragments sequentially to generate round markers and form fragment evaluation results. When the fragment evaluation result is an interactive construction evaluation result, filter out evidence acceptance actions to form resolution units, and determine the backfilling position and execution method. When the fragment evaluation result is an evidence response evaluation result or a continuation evaluation result, verify the editing action and evidence phrase and output evidence description or manual continuation evaluation conclusion, then recalculate the semantic editing fragment of the material claim according to the execution method, and control the virtual digital human to output continuation prompts or interactive completion prompts.

[0006] Preferably, the intelligent customer service system receives images of business materials uploaded by the user in the current session, and also receives text inquiries entered by the user; if the user's input is voice, the voice input is converted into speech-to-text. Perform image text recognition on business material images to obtain text blocks in the images; Perform layout semantic analysis on image text blocks to obtain the layout position and reading order of each image text block in the business material images; Named entity recognition is performed on the text blocks in the image to obtain a set of business entities. Then, the business entities are normalized according to the business status template in the business fusion knowledge base component to form material status semantics. The business entity set includes business object entities, state entities, result entities, and evidence entities; the business state template includes business object slots, state word slots, state result slots, evidence phrase slots, state types, state values, and evidence phrases; and the material state semantics includes state types, state values, and evidence phrases.

[0007] Preferably, the text consultation content or speech-to-text is segmented to obtain a set of slots to be matched. Then, the slots are matched with the request template recorded in the intelligent interactive knowledge base component. When the action segment, business object segment, question segment or status segment in the set of slots to be matched are received by the same request template, the request type and target status of the current request template are used as the semantics of the user request. The set of slots to be matched includes verb segments, business object segments, interrogative segments, and state segments; the request template includes word slots, business object slots, interrogative slots, target state slots, request type, and target state; and the user request semantics includes request type and target state.

[0008] Preferably, all material state semantics and user request semantics are encapsulated in a unified format to obtain a semantic unit sequence including material semantic unit sequence and request semantic unit sequence. The same semantic encoder is used to encode the material semantic unit sequence and the request semantic unit sequence to obtain material encoding sequence and request encoding sequence. The request encoding sequence is linearly mapped to obtain query vector, and the material encoding sequence is linearly mapped to obtain key vector and value vector. The query vector and key vector corresponding to each claim semantic unit are subjected to dot product operation to obtain several initial matching values. After scaling the several initial matching values ​​according to the key vector dimension, softmax normalization is performed to obtain the attention weight of the current claim semantic unit relative to each material semantic unit. The material semantic units are sorted according to the attention weight, and the material semantic unit ranked first is selected. Evidence phrases that are homologous to the current material semantic unit are extracted to form registration items. The registration items are organized according to the arrangement order of the claim semantic unit sequence to obtain the semantic registration sequence.

[0009] Preferably, the action type configuration is performed on the semantic registration sequence to obtain the edit action record, including: If the claim semantic unit and the material semantic unit in the registration item have the same unit semantics, and the current material semantic unit has the same source evidence phrase, the claim semantic unit and the material semantic unit in the current registration item are the editing objects, and the same unit semantics and the same source evidence phrase are the evidence phrase identifiers. The action type is configured as evidence acceptance action. If the semantic unit of the claim in the registration item cannot be expressed by the semantic unit of the material, and the current semantic unit of the claim originates from the claim type or the target state, the semantic unit of the claim in the current registration item is the editing object, the source identifier of the semantic unit of the claim is the evidence phrase identifier, and the action type is configured as the claim supplement action. If the claim semantic unit and the material semantic unit in the registration item point to the same state type, and the state value in the material semantic unit is different from the target state in the claim semantic unit, the claim semantic unit and the material semantic unit in the current registration item are used as the editing objects, and the action type is configured as the state review action with the same state type, the state value of the material semantic unit and the target state evidence phrase of the claim semantic unit. If the semantic unit of a request originates from the type of a request, and the type of a request is to continue processing or to inquire about progress, and the status value in the semantic unit of materials indicates that the current business has not entered the end state, then the semantic unit of a request and the semantic unit of materials in the current registration item are the editing objects, and the status value of the type of a request and the semantic unit of materials are the evidence phrase identifiers, and the action type is configured as a process advancement action. Editing action logs include actions for receiving evidence, adding requests, reviewing status, and advancing processes. Based on the arrangement order of the semantic unit sequence of the appeal, the editing action record is organized to obtain the semantic editing fragment of the material appeal.

[0010] Preferably, the editing actions, editing objects, and evidence phrases in the semantic editing segments of the material claims in the same order are read one by one according to the order of the editing action record, and a round mark is generated for each order; wherein, the order indicates that after the sequence of editing actions, the sequence of editing objects, and the sequence of evidence phrases in the semantic editing segments of the material claims are set according to the same arrangement order, the data in the same arrangement position are the same order; If the current editing action is an evidence continuation action and the edited object in the same sequence is complete and the evidence phrase is not empty, the round will be marked as an evidence response round; If the current editing action is a request completion action and the editing object in the same order contains a request semantic unit, the round will be marked as a single-item completion round; If the current editing action is a status verification action and the editing object in the same order contains a claim semantic unit and a material semantic unit, the action basis contains a status value and a target status, and the evidence phrase is not empty, then the round will be marked as a status verification round. If the current editing action is a process advancement action and the editing object in the same sequence contains a demand semantic unit and a material semantic unit, the action basis contains a demand type and a status value, and the evidence phrase is not empty, then the round will be marked as a processing guidance round. If the current position does not meet the conditions for forming any turn mark, the turn is marked as a continuation round.

[0011] Preferably, the fragment evaluation results are formed based on all round markers: When all rounds are marked as evidence response rounds, the fragment evaluation result is an evidence response; If there is a single supplementary question round, a status verification round, or a processing guidance round, and there is no follow-up sorting round, the fragment evaluation result is interactive construction; When there is a continuation round, the fragment evaluation result is continuation.

[0012] Preferably, when the fragment evaluation result is an interactive construction evaluation result, the evidence acceptance action is screened out from the material appeal semantic editing fragment, and the appeal supplementation action, status review action and process advancement action are retained to obtain the sequence of editing actions to be resolved. The editing objects and evidence phrases of each editing action to be resolved in the material appeal semantic editing fragment are read to form a resolution unit. Then, the backfill position of each resolution unit is determined according to the current editing action. When the action to be resolved is the action of adding a request, the object to be edited is the request semantic unit that is not expressed by the material semantic unit, and the backfill position of the current resolution unit is determined as the user's request semantic. When the action to be resolved is a status review action or a process advancement action, the editing objects include claim semantic units and material semantic units, and the evidence phrases include status values ​​and target status. At this time, the backfill position of the current resolution unit is determined as the material status semantic. The execution method of the resolution unit is obtained based on the current editing action to be resolved and the current backfill position, including: When the action to be resolved is a request addition action and the backfill position is the semantics of the user's request, the execution method is to receive user additions; When the action to be resolved is a status verification action and the backfill location is a material status semantic, the execution method is to call the status verification entry. When the action to be resolved is a process advancement action and the backfill position is a material status semantic, the execution method is to call the processing entry.

[0013] Preferably, when the fragment evaluation result is the evidence response evaluation result, it verifies whether the editing action sequence in the semantic editing fragment of the material claim only contains evidence acceptance actions, and verifies whether the evidence phrase sequence in the semantic editing fragment of the material claim has an evidence phrase in the same order as the editing object; when all verifications pass, it indicates that the current virtual digital human is suitable for providing evidence explanation around the business material image; otherwise, it indicates that the current virtual digital human is not suitable for providing evidence explanation, and outputs the human continuation evaluation conclusion. When the fragment evaluation result is a continuation evaluation result, the editing actions and their editing objects that do not meet the turn construction conditions in the semantic editing fragment of the material request are read, and the manual continuation evaluation conclusion is output. Among them, the conclusion of the manual follow-up assessment indicates that the current virtual digital human has stopped independently generating subsequent interactive content. Based on the editing actions, editing objects and evidence phrases in the semantic editing fragments of the material requests, a follow-up summary is generated for human customer service or back-end customer service processes to continue processing.

[0014] Preferably, the output content and backfill position of each digestion unit are executed one by one according to the execution method of the digestion unit; When the resolution unit is executed in the manner of receiving user supplements, the virtual digital human outputs the output content in the current resolution unit. At this time, the user supplements are received and the content is classified into the user's request semantics according to the backfill position in the current resolution unit. When the resolution unit is executed by calling the status verification entry, the virtual digital human outputs the output content in the current resolution unit. At this time, the status verification return content is obtained through the status verification entry, and it is classified into the material status semantics according to the backfill position in the current resolution unit. When the resolution unit is executed by calling the processing entry, the virtual digital human outputs the output content in the current resolution unit. At this time, the processing return content is obtained through the processing entry and is classified into the material status semantics according to the backfill position in the current resolution unit. Iterative analysis is performed based on the updated material claim semantics, the material claim semantic editing fragment is re-acquired, and the editing actions to be resolved and the objects to be resolved in the current resolution unit are retrieved from the re-acquired material claim semantic editing fragment; If there is no current edit action to be resolved and no current object to be resolved, it means that the current resolution unit has passed the recalculation, and the next resolution unit will continue to be executed. If there are still current editing actions to be resolved and current objects to be resolved, it means that the current resolution unit has not passed the recalculation. At this time, the remaining resolution units will be stopped, and the virtual digital human will be controlled to output the continuation prompts according to the output content of the current resolution unit. When all the resolution units in the resolution unit sequence have passed the recalculation, an interactive completion prompt is generated based on the output content of each resolution unit, and the virtual digital human is controlled to output the interactive completion prompt.

[0015] This invention provides a multimodal virtual digital human interaction method for intelligent customer service. It has the following beneficial effects: (1) After receiving the business material images and user consultation content in the current session, this method performs image text recognition, layout semantic analysis, named entity recognition, and state normalization processing on the business material images to obtain image text blocks, layout positions, reading order, business entity sets, state types, state values, evidence phrases, and material state semantics. At the same time, it performs word segmentation processing on the text consultation content or speech-to-text, and obtains the request type, target state, and user request semantics based on the set of word slots to be matched and the request template. Compared with intelligent customer service methods that only rely on user consultation content for intent recognition, this method can simultaneously utilize the evidence phrases in the business material images and the target state in the user consultation content, so that the basis for the virtual digital human's response is transformed from a single text intent to a joint analysis of business material images and user consultation content, which is suitable for business scenarios such as material verification, progress inquiry, and continued processing.

[0016] (2) This method encapsulates the semantics of material status and user requests into a sequence of semantic units, including a sequence of material semantic units and a sequence of request semantic units. It then obtains the material encoding sequence and the request encoding sequence through a semantic encoder. Finally, it calculates the initial matching value and attention weight using query vectors, key vectors, and value vectors, and organizes the registration items according to the attention weights to form a semantic registration sequence. Subsequently, it configures evidence acceptance actions, request supplementation actions, status verification actions, and process advancement actions on the semantic registration sequence, forming an editing action record and a material request semantic editing fragment. Compared to fixed question-and-answer databases, single-round intent classification, or general script generation methods, this method can differentiate processing tasks such as evidence response, information supplementation, status verification, and process advancement around request semantic units, material semantic units, cognate evidence phrases, status values, and target states, making the interactive content of the virtual digital human closer to the facts of the business materials and the user's processing purpose.

[0017] (3) This method generates round markers based on the editing actions, editing objects, and evidence phrases in the semantic editing fragments of material requests, and forms fragment evaluation results. When the fragment evaluation result is an interactive construction evaluation result, evidence acceptance actions are filtered out, and request supplementation actions, status verification actions, and process advancement actions are retained to form a sequence of editing actions to be resolved, resolution units, backfill positions, and execution methods. When the fragment evaluation result is an evidence response evaluation result or a continuation evaluation result, the editing action sequence, evidence phrase sequence, editing actions that do not meet the round construction conditions, and their editing objects are verified, and evidence explanations or manual continuation evaluation conclusions are output. Subsequently, user supplementary content, status verification return content, or processing return content are processed according to the execution method, and based on the updated material status semantics and user request semantics iterative analysis, the semantic editing fragments of material requests are re-acquired, so that the virtual digital human can output interactive completion prompts when the evidence is sufficient, and output continuation prompts or continuation summaries when information is missing, status is inconsistent, or the process cannot continue, so as to achieve the purpose of multimodal material understanding, user request analysis, business evidence explanation, interactive completion, and manual continuation collaborative processing. Attached Figure Description

[0018] Figure 1 This is a schematic diagram illustrating the steps of a multimodal virtual digital human interaction method for intelligent customer service according to the present invention. Figure 2 This is a logic block diagram for a multimodal virtual digital human interaction method for intelligent customer service according to the present invention. Detailed Implementation

[0019] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0020] Example 1 Please see Figure 1 This invention provides a multimodal virtual digital human interaction method for intelligent customer service. To achieve the above objectives, this invention is implemented through the following technical solution: including: Receive business material images and user inquiry content from the current session, parse the business material images to generate material status semantics, and perform word segmentation on the inquiry content to generate user request semantics; The semantics of material status and user demands are encapsulated into a sequence of semantic units. After semantic encoding and linear mapping, the attention weight of each semantic unit is calculated and registration items are organized to form a semantic registration sequence. The action type configuration is performed on the semantic registration sequence to obtain the editing action record. After sorting, the material demand semantic editing fragment is formed. Read the material requests semantic fragments sequentially to generate round markers and form fragment evaluation results. When the fragment evaluation result is an interactive construction evaluation result, filter out evidence acceptance actions to form resolution units, and determine the backfilling position and execution method. When the fragment evaluation result is an evidence response evaluation result or a continuation evaluation result, verify the editing action and evidence phrase and output evidence description or manual continuation evaluation conclusion, then recalculate the semantic editing fragment of the material claim according to the execution method, and control the virtual digital human to output continuation prompts or interactive completion prompts.

[0021] In this embodiment, business material images and user inquiry content are received in the current session. The business material images are parsed to generate material status semantics, and the inquiry content is segmented to generate user request semantics, undertaking the tasks of business material identification, business status extraction, and user request understanding. Specifically, the virtual digital human does not directly generate a generalized response based on the user inquiry content. Instead, it first extracts text, layout information, and evidence content that can represent the business status from the business material images, and then combines this with the action intent, business object, and target status in the user inquiry content to form user request semantics. For example, a user uploads a screenshot of a business process, which shows "Document review failed, ID photo is unclear," while the user's inquiry content is "Why can't I complete this? Can I continue submitting?" After parsing, the business material images generate material status semantics containing content such as "Document review failed" and "ID photo is unclear," and the user inquiry content, after segmentation, generates user request semantics containing the meanings of "reason query" and "continue submitting." Compared to current technologies that only match question-and-answer templates based on user inquiries, this method can integrate the factual evidence in business materials and images with the purpose of handling user inquiries into the same processing chain, so that the virtual digital human's response has material basis, rather than just outputting fixed scripts.

[0022] Subsequently, the semantics of material status and user request are encapsulated into a sequence of semantic units. After semantic encoding and linear mapping of the semantic unit sequence, the attention weight of each semantic unit is calculated, and registration items are organized according to the attention weight to form a semantic registration sequence. Then, action type configuration is performed on the semantic registration sequence to obtain editing action records, which are sorted to form a semantic editing fragment of material request. The role of the current process is to first analyze the matching between the semantics of material status and user request, and then organize different processing tasks into executable editing action records. For example, in the examples of "document review failed, ID photo is unclear" and "why can't it be processed, can I still submit?", the "why can't it be processed?" in the semantics of user request can form a registration item with "document review failed" and "ID photo is unclear" in the semantics of material status, which can be configured as evidence acceptance action; the "can I still submit?" in the semantics of user request and "review failed" in the semantics of material status both point to the subsequent processing process, which can be configured as process advancement action; if the user only says "please process it for me", but the semantics of user request lacks a clear target status, it can be configured as request supplementation action. Compared to current techniques that directly generate responses based on a single intent label, this method separates evidence explanation, request supplementation, status review, and process advancement by using attention weights, registration items, semantic registration sequences, and editing action records. This enables the semantically edited fragments of material requests to clearly express what material evidence, user requests, and business actions the virtual digital human should organize and interact around next.

[0023] Finally, the semantic editing fragments of the material request are read sequentially to generate round markers, and fragment evaluation results are formed based on the round markers. When the fragment evaluation result is an interactive construction evaluation result, evidence acceptance actions are filtered out to form resolution units, and the backfilling position and execution method are determined. When the fragment evaluation result is an evidence response evaluation result or a continuation evaluation result, the editing actions and evidence phrases are verified, and evidence explanations or manual continuation evaluation conclusions are output. Then, the semantic editing fragments of the material request are recalculated according to the execution method, and the virtual digital human is controlled to output continuation prompts or interactive completion prompts. For example, if the semantic editing fragments of the material request contain both evidence acceptance actions and process advancement actions, the fragment evaluation result can form an interactive construction evaluation result. The virtual digital human first explains based on the evidence phrase "The current business material image shows that the ID card photo is not clear, therefore the data review has not passed," then forms resolution units based on the process advancement actions, determines the execution method as calling the processing entry, and recalculates the semantic editing fragments of the material request after obtaining the processing return content. If, after recalculation, no longer the current editing action or object to be resolved exists, the virtual human outputs an interaction completion prompt; if the processing portal returns information indicating a missing new ID photo, the virtual human outputs a follow-up prompt, guiding the user to supplement materials; if the semantic editing fragment of the material request contains content that cannot construct a clear turn, a human follow-up assessment conclusion is output. Compared to the common single-round responses, general guidance, or direct transfer to human agents used by current virtual human agents, this method can output evidence explanations when evidence is sufficient, control the virtual human to ask supplementary questions when information is missing, trigger status review when the status is inconsistent, call the processing portal when further processing is needed, and generate a human follow-up assessment conclusion when it cannot be handled independently. This makes the intelligent customer service interaction process form a clear closed loop around business material images, user consultation content, semantic editing fragments of material requests, and execution methods.

[0024] Example 2 Please refer to Figure 2 Specifically: In the current session, the intelligent customer service system receives images of business materials uploaded by the user and receives text inquiries entered by the user; if the user inputs voice, the voice input is converted into speech-to-text. Perform image text recognition on business material images to obtain text blocks in the images; Perform layout semantic analysis on image text blocks to obtain the layout position and reading order of each image text block in the business material images; Named entity recognition is performed on the text blocks in the image to obtain a set of business entities. Then, the business entities are normalized according to the business status template in the business fusion knowledge base component to form material status semantics. The business entity set includes business object entities, status entities, result entities, and evidence entities; the business status template includes business object slots, status word slots, status result slots, evidence phrase slots, status types, status values, and evidence phrases; the material status semantics includes status types, status values, and evidence phrases; and the business fusion knowledge base component is a pre-configured knowledge processing component within the intelligent customer service system. State normalization processing: Input the business object entity into the business state template for word slot matching. When the same business state template receives the business object entity and the state entity or the business object entity and the result entity, the state type and state value of the current business state template are determined as the material state semantics, and the received evidence entity is used as the evidence phrase of the material state semantics. The status type indicates the business status category to which the business material belongs, the status value indicates the status result presented by the business material, and the evidence phrase indicates the image and text block fragment on which the status value is based.

[0025] The text consultation content or speech-to-text is segmented to obtain a set of slots to be matched. Then, the slots are matched with the request template recorded in the intelligent interactive knowledge base component. When the action segment, business object segment, question segment or status segment in the set of slots to be matched are received by the same request template, the request type and target status of the current request template are used as the semantics of the user request. The set of slots to be matched includes verb segments, business object segments, interrogative segments, and state segments; the request template includes word slots, business object slots, interrogative slots, target state slots, request type, and target state; and the user request semantics includes request type and target state. The request type indicates the business matter that the user wants to handle, and the target status indicates the processing result that the user expects the current business to achieve.

[0026] In this embodiment, the system first receives business material images uploaded by the user in the current session, and also receives text-based consultation content input by the user. When the user inputs voice input, the voice input is converted into speech-to-text. Then, image text recognition is performed on the business material images to obtain image text blocks. Layout semantic analysis is performed on the image text blocks to obtain the layout position and reading order of each image text block within the business material images. Named entity recognition is performed on the image text blocks to obtain a set of business entities, including business object entities, state entities, result entities, and evidence entities. Next, the set of business entities is normalized according to the business state template in the business fusion knowledge base component. The business state template includes business object slots, state word slots, state result slots, evidence phrase slots, state types, state values, and evidence phrases. In the state normalization process, business object entities are input into the business state template for word-slot matching. When the same business state template receives both a business object entity and a state entity, or receives both a business object entity and a result entity, the state type and state value of the current business state template are determined as the material state semantics, and the received evidence entity is used as the evidence phrase of the material state semantics. Simultaneously, the text consultation content or speech-to-text is segmented to obtain a set of slots to be matched, including verb segments, business object segments, question segments, and status segments. Slot matching is then performed between the request template recorded in the intelligent interaction knowledge base component and the set of slots to be matched. When an action segment, business object segment, question segment, or status segment in the set of slots to be matched is received by the same request template, the request type and target state of the current request template are used as the user's request semantics. Through the above processing, the material state semantics can express the state type of the business material image, the state value presented by the business material image, and the evidence phrase on which the state value is generated. The user request semantics can express the business matter that the user wants to handle and the target state that the user hopes the current business will reach. Compared to processing methods that rely solely on textual consultation content for intent recognition or solely on text extraction from business material images, this implementation method places the image text blocks, layout positions, reading order, business entity sets, business status templates, status types, status values, and evidence phrases in the business material images into the same analysis chain as the set of word slots to be matched, request templates, request types, and target status in the textual consultation content or speech-to-text. This enables the virtual digital human to generate subsequent interactive content based on the facts of the business materials and the user's purpose for handling the matter. This achieves the goals of understanding business material images, normalizing business status, analyzing user requests, and building a foundation for evidence-based responses. Furthermore, it provides a clear data source for subsequent evidence explanations, status verification, handling guidance, and manual follow-up.

[0027] Example 3 Please refer to Figure 2Specifically: all material state semantics and user request semantics are encapsulated in a unified format to obtain a semantic unit sequence including material semantic unit sequence and request semantic unit sequence. The same semantic encoder is used to encode the material semantic unit sequence and request semantic unit sequence to obtain material encoding sequence and request encoding sequence. The request encoding sequence is linearly mapped to obtain query vector, and the material encoding sequence is linearly mapped to obtain key vector and value vector. Among them, the vector order in the material vector sequence MV is consistent with the material semantic unit sequence; the vector order in the claim vector sequence NV is consistent with the claim semantic unit sequence. The query vector and key vector corresponding to each claim semantic unit are subjected to dot product operation to obtain several initial matching values. After scaling the several initial matching values ​​according to the key vector dimension, softmax normalization is performed to obtain the attention weight of the current claim semantic unit relative to each material semantic unit. The material semantic units are sorted according to the attention weight, and the material semantic unit ranked first is selected. Evidence phrases that are homologous to the current material semantic unit are extracted to form registration items. The registration items are organized according to the arrangement order of the claim semantic unit sequence to obtain the semantic registration sequence.

[0028] Configure the action type for the semantic registration sequence and obtain the edit action record, including: If the claim semantic unit and the material semantic unit in the registration item have the same unit semantics, and the current material semantic unit has the same source evidence phrase, the claim semantic unit and the material semantic unit in the current registration item are the editing objects, and the same unit semantics and the same source evidence phrase are the evidence phrase identifiers. The action type is configured as evidence acceptance action. If the semantic unit of the claim in the registration item cannot be expressed by the semantic unit of the material, and the current semantic unit of the claim originates from the claim type or the target state, the semantic unit of the claim in the current registration item is the editing object, the source identifier of the semantic unit of the claim is the evidence phrase identifier, and the action type is configured as the claim supplement action. If the claim semantic unit and the material semantic unit in the registration item point to the same state type, and the state value in the material semantic unit is different from the target state in the claim semantic unit, the claim semantic unit and the material semantic unit in the current registration item are used as the editing objects, and the action type is configured as the state review action with the same state type, the state value of the material semantic unit and the target state evidence phrase of the claim semantic unit. If the semantic unit of a request originates from the type of a request, and the type of a request is to continue processing or to inquire about progress, and the status value in the semantic unit of materials indicates that the current business has not entered the end state, then the semantic unit of a request and the semantic unit of materials in the current registration item are the editing objects, and the status value of the type of a request and the semantic unit of materials are the evidence phrase identifiers, and the action type is configured as a process advancement action. Editing action logs include actions for receiving evidence, adding requests, reviewing status, and advancing processes. Based on the arrangement order of the semantic unit sequence of the appeal, the editing action record is organized to obtain the semantic editing fragment of the material appeal.

[0029] In this embodiment, all material state semantics and user request semantics are encapsulated in a unified format to obtain a semantic unit sequence including a material semantic unit sequence and a request semantic unit sequence. The same semantic encoder is then used to encode both the material semantic unit sequence and the request semantic unit sequence, resulting in a material-encoded sequence and a request-encoded sequence. Next, a linear mapping is performed on the request-encoded sequence to obtain a query vector, and a linear mapping is performed on the material-encoded sequence to obtain a key vector and a value vector, thus bringing the material state semantics and user request semantics into the same semantic analysis space. Then, a dot product operation is performed on the query vector and key vector corresponding to each request semantic unit to obtain several initial matching values. These initial matching values ​​are then scaled according to the key vector dimension and subjected to softmax normalization to obtain the attention weight of the current request semantic unit relative to each material semantic unit. The material semantic units are then sorted according to the attention weights, and the material semantic unit ranked first is selected. Evidence phrases from the same source as the current material semantic unit are extracted to form registration terms. These registration terms are then organized according to the order of the request semantic unit sequence to obtain a semantic registration sequence. Subsequently, action type configuration is performed on the semantic registration sequence: when the request semantic unit and the material semantic unit in the registration item have the same unit semantics, and the current material semantic unit has a common evidence phrase, the action type is configured as evidence acceptance action; when the request semantic unit in the registration item cannot be expressed by the material semantic unit, and the current request semantic unit originates from the request type or target state, the action type is configured as request supplementation action; when the request semantic unit and the material semantic unit in the registration item point to the same state type, and the state value in the material semantic unit is different from the target state in the request semantic unit, the action type is configured as state review action; when the request semantic unit originates from the request type, the request type is continue processing or query progress, and the state value in the material semantic unit indicates that the current business has not entered the end state, the action type is configured as process advancement action. Finally, the editing action record is organized according to the arrangement order of the request semantic unit sequence to obtain the material request semantic editing fragment. Through the above implementation methods, the current solution transforms the matching process between material status semantics and user request semantics into a continuous processing process of semantic encoding, linear mapping, attention weight calculation, registration item organization, and action type configuration. This solves the problem that current intelligent customer service relies solely on single intent tags, fixed question-and-answer templates, or general scripts, making it difficult to distinguish between evidence responses, request supplementary inquiries, status verification, and process advancement. This enables the material request semantic editing fragments to clearly express which cognate evidence phrases the virtual digital human should cite, which request semantic units and material semantic units to organize the response around, and which editing action record to trigger. This achieves the goal of providing a clear input basis for subsequent round marking, fragment evaluation results, resolution units, backfill positions, and execution methods, and makes the virtual digital human's response content closer to the factual evidence in the business material images and the handling objectives in the user's consultation content.

[0030] Example 4 Please refer to Figure 2 Specifically: based on the order of the editing action record, the editing actions, editing objects, and evidence phrases in the semantic editing segments of the material claims in the same order are read one by one, and a round mark is generated for each order; where the order indicates that after the sequence of editing actions, the sequence of editing objects, and the sequence of evidence phrases in the semantic editing segments of the material claims are set according to the same arrangement order, the data in the same arrangement position are the same order. If the current editing action is an evidence continuation action and the edited object in the same sequence is complete and the evidence phrase is not empty, the round will be marked as an evidence response round; If the current editing action is a request completion action and the editing object in the same order contains a request semantic unit, the round will be marked as a single-item completion round; If the current editing action is a status verification action and the editing object in the same order contains a claim semantic unit and a material semantic unit, the action basis contains a status value and a target status, and the evidence phrase is not empty, then the round will be marked as a status verification round. If the current editing action is a process advancement action and the editing object in the same sequence contains a demand semantic unit and a material semantic unit, the action basis contains a demand type and a status value, and the evidence phrase is not empty, then the round will be marked as a processing guidance round. If the current position does not meet the conditions for forming any turn mark, the turn is marked as a continuation round.

[0031] Based on the evaluation results of fragments formed by all round markers: When all rounds are marked as evidence response rounds, the fragment evaluation result is an evidence response; If there is a single supplementary question round, a status verification round, or a processing guidance round, and there is no follow-up sorting round, the fragment evaluation result is interactive construction; When there is a continuation round, the fragment evaluation result is continuation.

[0032] When the fragment evaluation result is an interactive construction evaluation result, the evidence acceptance action is screened out from the material appeal semantic editing fragment, and the appeal supplementation action, status review action and process advancement action are retained to obtain the sequence of editing actions to be resolved. The editing objects and evidence phrases of each editing action to be resolved in the material appeal semantic editing fragment are read to form a resolution unit. Then, the backfill position of each resolution unit is determined according to the current editing action. When the action to be resolved is the action of adding a request, the object to be edited is the request semantic unit that is not expressed by the material semantic unit, and the backfill position of the current resolution unit is determined as the user's request semantic. When the action to be resolved is a status review action or a process advancement action, the editing objects include claim semantic units and material semantic units, and the evidence phrases include status values ​​and target status. At this time, the backfill position of the current resolution unit is determined as the material status semantic. The execution method of the resolution unit is obtained based on the current editing action to be resolved and the current backfill position, including: When the action to be resolved is a request addition action and the backfill position is the semantics of the user's request, the execution method is to receive user additions; When the action to be resolved is a status verification action and the backfill location is a material status semantic, the execution method is to call the status verification entry. When the action to be resolved is a process advancement action and the backfill position is a material status semantic, the execution method is to call the processing entry.

[0033] In this embodiment, based on the order of the editing action record, the editing action, editing object, and evidence phrase in the semantic editing segment of the material claim at the same order are read one by one, and a round mark is generated for each order; wherein, the order indicates that after the sequence of editing action, sequence of editing object, and sequence of evidence phrase in the semantic editing segment of the material claim are set according to the same arrangement order, the data in the same arrangement position are the same order. When the current editing action is an evidence acceptance action, and the edited object in the same sequence is complete and the evidence phrase is not empty, the round is marked as an evidence response round; when the current editing action is a request supplement action, and the edited object in the same sequence contains a request semantic unit, the round is marked as a single-item supplementary question round; when the current editing action is a status review action, and the edited object in the same sequence contains a request semantic unit and a material semantic unit, the action basis includes a status value and a target status, and the evidence phrase is not empty, the round is marked as a status verification round; when the current editing action is a process advancement action, and the edited object in the same sequence contains a request semantic unit and a material semantic unit, the action basis includes a request type and a status value, and the evidence phrase is not empty, the round is marked as a processing guidance round; when the current sequence does not meet the formation conditions for any round mark, the round is marked as a continuation and organization round. Subsequently, the intelligent customer service system generates a segment evaluation result based on all round markers: when all round markers are evidence response rounds, the segment evaluation result is evidence response; when there are single-item supplementary question rounds, status verification rounds, or processing guidance rounds, and no follow-up organization rounds, the segment evaluation result is interactive construction; when there are follow-up organization rounds, the segment evaluation result is follow-up. When the segment evaluation result is an interactive construction evaluation result, evidence acceptance actions are filtered out from the material request semantic editing segment, while request supplementation actions, status review actions, and process advancement actions are retained to obtain the sequence of editing actions to be resolved. The system then reads the editing objects and evidence phrases of each editing action to be resolved in the material request semantic editing segment to form resolution units, and determines the backfill position of each resolution unit based on the current editing action; when the editing action to be resolved is a request supplementation action, the editing object is a request semantic unit not expressed by the material semantic unit, and the backfill position of the current resolution unit is determined. When the filling position is determined to be the user's request semantics, the execution method is to receive user supplements; when the editing action to be resolved is a status review action, the editing object includes the request semantic unit and the material semantic unit, the evidence phrase includes the status value and the target status, the filling position of the current resolution unit is determined to be the material status semantics, and the execution method is to call the status verification entry; when the editing action to be resolved is a process advancement action, the editing object includes the request semantic unit and the material semantic unit, the evidence phrase includes the status value and the target status, the filling position of the current resolution unit is determined to be the material status semantics, and the execution method is to call the processing entry.Through the above implementation methods, the current solution transforms the editing actions, editing objects, and evidence phrases in the semantic editing segments of material requests into evidence response rounds, single-item supplementary question rounds, status verification rounds, processing guidance rounds, and continuation organization rounds. It further generates segment evaluation results, a sequence of editing actions to be resolved, resolution units, backfill positions, and execution methods, achieving the goal of distinguishing between direct evidence explanation, user supplementation, status verification, processing guidance, and manual continuation. Compared to the current processing method of directly outputting fixed scripts based on a single intent or directly transferring to a human, this implementation method allows the virtual digital human's interaction process to unfold item by item along the semantic editing segments of material requests. It organizes evidence responses when evidence phrases are sufficient, receives user supplementation when user request semantics are incomplete, invokes the status verification entry when there is a state difference between the material status semantics and the user request semantics, invokes the processing entry when the current business has not entered the closed state, and enters the manual continuation path when a continuation organization round occurs. This ensures that the multimodal virtual digital human interaction process has clear round divisions, business processing paths, and evidence sources.

[0034] Example 5 Please refer to Figure 2 Specifically: When the fragment evaluation result is the evidence response evaluation result, verify whether the editing action sequence in the semantic editing fragment of the material claim only contains evidence acceptance actions, and verify whether the evidence phrase sequence in the semantic editing fragment of the material claim has an evidence phrase in the same order as the edited object; when all verifications pass, it means that the current virtual digital human is suitable for providing evidence explanation around the business material image; otherwise, it means that the current virtual digital human is not suitable for providing evidence explanation, and output the human follow-up evaluation conclusion; When the fragment evaluation result is a continuation evaluation result, the editing actions and their editing objects that do not meet the turn construction conditions in the semantic editing fragment of the material request are read, and the manual continuation evaluation conclusion is output. Among them, the conclusion of the manual follow-up assessment indicates that the current virtual digital human has stopped independently generating subsequent interactive content. Based on the editing actions, editing objects and evidence phrases in the semantic editing fragments of the material requests, a follow-up summary is generated for human customer service or back-end customer service processes to continue processing.

[0035] The output content and backfill location of each digestion unit are executed one by one according to the execution method of the digestion unit; When the resolution unit is executed in the manner of receiving user supplements, the virtual digital human outputs the output content in the current resolution unit. At this time, the user supplements are received and the content is classified into the user's request semantics according to the backfill position in the current resolution unit. When the resolution unit is executed by calling the status verification entry, the virtual digital human outputs the output content in the current resolution unit. At this time, the status verification return content is obtained through the status verification entry, and it is classified into the material status semantics according to the backfill position in the current resolution unit. When the resolution unit is executed by calling the processing entry, the virtual digital human outputs the output content in the current resolution unit. At this time, the processing return content is obtained through the processing entry and is classified into the material status semantics according to the backfill position in the current resolution unit. Iterative analysis is performed based on the updated material claim semantics, the material claim semantic editing fragment is re-acquired, and the editing actions to be resolved and the objects to be resolved in the current resolution unit are retrieved from the re-acquired material claim semantic editing fragment; If there is no current edit action to be resolved and no current object to be resolved, it means that the current resolution unit has passed the recalculation, and the next resolution unit will continue to be executed. If there are still current editing actions to be resolved and current objects to be resolved, it means that the current resolution unit has not passed the recalculation. At this time, the remaining resolution units will be stopped, and the virtual digital human will be controlled to output the continuation prompts according to the output content of the current resolution unit. When all the resolution units in the resolution unit sequence have passed the recalculation, an interactive completion prompt is generated based on the output content of each resolution unit, and the virtual digital human is controlled to output the interactive completion prompt.

[0036] In this embodiment, the subsequent processing path for the virtual digital human is first selected based on the fragment evaluation result. When the fragment evaluation result is an evidence response evaluation result, the intelligent customer service system verifies whether the editing action sequence in the semantic editing fragment of the material request only contains evidence acceptance actions, and verifies whether the evidence phrase sequence in the semantic editing fragment of the material request contains an evidence phrase in the same order as the edited object. When the above verifications pass, it indicates that the current virtual digital human can provide evidence explanations around the business material image and generate response content based on the edited object and evidence phrase in the semantic editing fragment of the material request. When the verification fails, or the fragment evaluation result is a continuation evaluation result, the system reads the editing actions and their edited objects in the semantic editing fragment of the material request that do not meet the round construction conditions, outputs the manual continuation evaluation conclusion, and generates a continuation summary based on the editing actions, edited objects, and evidence phrases in the semantic editing fragment of the material request for manual customer service or backend customer service processes to continue processing. For the resolution units formed when the fragment evaluation result is the interactive construction evaluation result, the output content and backfill position of each resolution unit are executed one by one according to the execution method of the resolution unit: when the execution method of the resolution unit is to receive user supplements, the virtual digital human outputs the output content of the current resolution unit, receives the user supplements, and is assigned to the user request semantics according to the backfill position of the current resolution unit; when the execution method of the resolution unit is to call the status verification entry, the virtual digital human outputs the output content of the current resolution unit, obtains the status verification return content through the status verification entry, and is assigned to the material status semantics according to the backfill position of the current resolution unit; when the execution method of the resolution unit is to call the processing entry, the virtual digital human outputs the output content of the current resolution unit, obtains the processing return content through the processing entry, and is assigned to the material status semantics according to the backfill position of the current resolution unit. Subsequently, iterative analysis is performed based on the updated material claim semantics. The material claim semantic editing fragment is re-acquired, and the editing actions to be resolved and the objects to be resolved in the current resolution unit are retrieved from the re-acquired material claim semantic editing fragment. If there are no current editing actions to be resolved and objects to be resolved, it means that the current resolution unit has passed the recalculation, and the next resolution unit is executed. If there are still current editing actions to be resolved and objects to be resolved, it means that the current resolution unit has not passed the recalculation, and the remaining resolution units are stopped. The virtual digital human is controlled to output the continuation prompt based on the output content of the current resolution unit. When all resolution units in the resolution unit sequence have passed the recalculation, interactive completion prompts are formed based on the output content of each resolution unit, and the virtual digital human is controlled to output the interactive completion prompts.Through the above implementation methods, the current solution incorporates evidence explanation, manual follow-up assessment conclusion, user-supplemented content, status verification return content, processing return content, and semantic editing fragment recalculation of material requests into the same closed-loop process. Compared to the current approach of directly outputting vague statements or directly transferring to human intervention when evidence is insufficient or the process changes, the current solution can first determine whether the current virtual digital human is suitable for providing evidence explanation based on business material images. Then, when user supplementation, status verification, or continued processing is required, the corresponding execution method is triggered. The recalculation result confirms whether the current resolution unit has been processed. This achieves the goal of ensuring that the subsequent interaction content of the virtual digital human has evidence sources, process basis, and follow-up basis, making the multimodal virtual digital human interaction have a clearer processing path between evidence response, process processing, and manual follow-up.

[0037] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended technical solutions and their equivalents.

Claims

1. A multimodal virtual digital human interaction method for intelligent customer service, characterized in that, include: Receive business material images and user inquiry content from the current session, parse the business material images to generate material status semantics, and perform word segmentation on the inquiry content to generate user request semantics; The semantics of material status and user demands are encapsulated into a sequence of semantic units. After semantic encoding and linear mapping, the attention weight of each semantic unit is calculated and registration items are organized to form a semantic registration sequence. The action type configuration is performed on the semantic registration sequence to obtain the editing action record. After sorting, the material demand semantic editing fragment is formed. Read the material requests semantically in sequence to generate round markers and form fragment evaluation results. When the fragment evaluation result is an interactive construction evaluation result, filter out the evidence acceptance action to form a resolution unit, and determine the backfilling position and execution method. When the fragment evaluation result is an evidence response evaluation result or a continuation evaluation result, verify the editing action and evidence phrase and output evidence description or manual continuation evaluation conclusion, then recalculate the semantic editing fragment of the material claim according to the execution method, and control the virtual digital human to output continuation prompts or interactive completion prompts.

2. The multimodal virtual digital human interaction method for intelligent customer service according to claim 1, characterized in that, The intelligent customer service system receives images of business materials uploaded by the user in the current session, as well as text inquiries entered by the user; if the user's input is voice, the voice input is converted into speech-to-text. Perform image text recognition on business material images to obtain text blocks in the images; Perform layout semantic analysis on image text blocks to obtain the layout position and reading order of each image text block in the business material images; Named entity recognition is performed on the text blocks in the image to obtain a set of business entities. Then, the business entities are normalized according to the business status template in the business fusion knowledge base component to form material status semantics. The business entity set includes business object entities, state entities, result entities, and evidence entities; the business state template includes business object slots, state word slots, state result slots, evidence phrase slots, state types, state values, and evidence phrases; and the material state semantics includes state types, state values, and evidence phrases.

3. The multimodal virtual digital human interaction method for intelligent customer service according to claim 2, characterized in that, The text consultation content or speech-to-text is segmented to obtain a set of slots to be matched. Then, the slots are matched with the request template recorded in the intelligent interactive knowledge base component. When the action segment, business object segment, question segment or status segment in the set of slots to be matched are received by the same request template, the request type and target status of the current request template are used as the semantics of the user request. The set of slots to be matched includes verb segments, business object segments, interrogative segments, and state segments; the request template includes word slots, business object slots, interrogative slots, target state slots, request type, and target state; and the user request semantics includes request type and target state.

4. The multimodal virtual digital human interaction method for intelligent customer service according to claim 3, characterized in that, All material state semantics and user request semantics are encapsulated in a unified format to obtain a semantic unit sequence including material semantic unit sequence and request semantic unit sequence. The same semantic encoder is used to encode the material semantic unit sequence and request semantic unit sequence respectively to obtain material encoding sequence and request encoding sequence. The request encoding sequence is linearly mapped to obtain query vector, and the material encoding sequence is linearly mapped to obtain key vector and value vector. The query vector and key vector corresponding to each claim semantic unit are subjected to dot product operation to obtain several initial matching values. After scaling the several initial matching values ​​according to the key vector dimension, softmax normalization is performed to obtain the attention weight of the current claim semantic unit relative to each material semantic unit. The material semantic units are sorted according to the attention weight, and the material semantic unit ranked first is selected. Evidence phrases that are homologous to the current material semantic unit are extracted to form registration items. The registration items are organized according to the arrangement order of the claim semantic unit sequence to obtain the semantic registration sequence.

5. A multimodal virtual digital human interaction method for intelligent customer service according to claim 4, characterized in that, Configure the action type for the semantic registration sequence to obtain the edit action record, including: If the claim semantic unit and the material semantic unit in the registration item have the same unit semantics, and the current material semantic unit has the same source evidence phrase, the claim semantic unit and the material semantic unit in the current registration item are the editing objects, and the same unit semantics and the same source evidence phrase are the evidence phrase identifiers. The action type is configured as evidence acceptance action. If the semantic unit of the claim in the registration item cannot be expressed by the semantic unit of the material, and the current semantic unit of the claim originates from the claim type or the target state, the semantic unit of the claim in the current registration item is the editing object, the source identifier of the semantic unit of the claim is the evidence phrase identifier, and the action type is configured as the claim supplement action. If the claim semantic unit and the material semantic unit in the registration item point to the same state type, and the state value in the material semantic unit is different from the target state in the claim semantic unit, the claim semantic unit and the material semantic unit in the current registration item are used as the editing objects, and the action type is configured as the state review action with the same state type, the state value of the material semantic unit and the target state evidence phrase of the claim semantic unit. If the semantic unit of a request originates from the type of a request, and the type of a request is to continue processing or to inquire about progress, and the status value in the semantic unit of materials indicates that the current business has not entered the end state, then the semantic unit of a request and the semantic unit of materials in the current registration item are the editing objects, and the status value of the type of a request and the semantic unit of materials are the evidence phrase identifiers, and the action type is configured as a process advancement action. Editing action logs include actions for receiving evidence, adding requests, reviewing status, and advancing processes. Based on the arrangement order of the semantic unit sequence of the appeal, the editing action record is organized to obtain the semantic editing fragment of the material appeal.

6. A multimodal virtual digital human interaction method for intelligent customer service according to claim 5, characterized in that, Based on the order of the editing action record, the editing actions, editing objects, and evidence phrases in the semantic editing segments of the material claims in the same order are read one by one, and a round mark is generated for each order; where the order indicates that after the sequence of editing actions, the sequence of editing objects, and the sequence of evidence phrases in the semantic editing segments of the material claims are set according to the same arrangement order, the data in the same arrangement position are the same order. If the current editing action is an evidence continuation action and the edited object in the same sequence is complete and the evidence phrase is not empty, the round will be marked as an evidence response round; If the current editing action is a request completion action and the editing object in the same order contains a request semantic unit, the round will be marked as a single-item completion round; If the current editing action is a status verification action and the editing object in the same order contains a claim semantic unit and a material semantic unit, the action basis contains a status value and a target status, and the evidence phrase is not empty, then the round will be marked as a status verification round. If the current editing action is a process advancement action and the editing object in the same sequence contains a demand semantic unit and a material semantic unit, the action basis contains a demand type and a status value, and the evidence phrase is not empty, then the round will be marked as a processing guidance round. If the current position does not meet the conditions for forming any turn mark, the turn is marked as a continuation round.

7. A multimodal virtual digital human interaction method for intelligent customer service according to claim 6, characterized in that, Based on the evaluation results of fragments formed by all round markers: When all rounds are marked as evidence response rounds, the fragment evaluation result is an evidence response; If there is a single supplementary question round, a status verification round, or a processing guidance round, and there is no follow-up sorting round, the fragment evaluation result is interactive construction; When there is a continuation round, the fragment evaluation result is continuation.

8. A multimodal virtual digital human interaction method for intelligent customer service according to claim 7, characterized in that, When the fragment evaluation result is an interactive construction evaluation result, the evidence acceptance action is screened out from the material appeal semantic editing fragment, and the appeal supplementation action, status review action and process advancement action are retained to obtain the sequence of editing actions to be resolved. The editing objects and evidence phrases of each editing action to be resolved in the material appeal semantic editing fragment are read to form a resolution unit. Then, the backfill position of each resolution unit is determined according to the current editing action. When the action to be resolved is the action of adding a request, the object to be edited is the request semantic unit that is not expressed by the material semantic unit, and the backfill position of the current resolution unit is determined as the user's request semantic. When the action to be resolved is a status review action or a process advancement action, the editing objects include claim semantic units and material semantic units, and the evidence phrases include status values ​​and target status. At this time, the backfill position of the current resolution unit is determined as the material status semantic. The execution method of the resolution unit is obtained based on the current editing action to be resolved and the current backfill position, including: When the action to be resolved is a request addition action and the backfill position is the semantics of the user's request, the execution method is to receive user additions; When the action to be resolved is a status verification action and the backfill location is a material status semantic, the execution method is to call the status verification entry. When the action to be resolved is a process advancement action and the backfill position is a material status semantic, the execution method is to call the processing entry.

9. A multimodal virtual digital human interaction method for intelligent customer service according to claim 8, characterized in that, When the fragment evaluation result is the evidence response evaluation result, verify whether the editing action sequence in the semantic editing fragment of the material claim only contains evidence receiving actions, and verify whether there is an evidence phrase sequence in the semantic editing fragment of the material claim that is in the same order as the edited object; If all verifications pass, it means that the current virtual digital human is suitable for providing evidence based on business materials and images; otherwise, it means that the current virtual digital human is not suitable for providing evidence, and a human follow-up evaluation conclusion will be output. When the fragment evaluation result is a continuation evaluation result, the editing actions and their editing objects that do not meet the turn construction conditions in the semantic editing fragment of the material request are read, and the manual continuation evaluation conclusion is output. Among them, the conclusion of the manual follow-up assessment indicates that the current virtual digital human has stopped independently generating subsequent interactive content. Based on the editing actions, editing objects and evidence phrases in the semantic editing fragments of the material requests, a follow-up summary is generated for human customer service or back-end customer service processes to continue processing.

10. A multimodal virtual digital human interaction method for intelligent customer service according to claim 9, characterized in that, The output content and backfill location of each digestion unit are executed one by one according to the execution method of the digestion unit; When the resolution unit is executed in the manner of receiving user supplements, the virtual digital human outputs the output content in the current resolution unit. At this time, the user supplements are received and the content is classified into the user's request semantics according to the backfill position in the current resolution unit. When the resolution unit is executed by calling the status verification entry, the virtual digital human outputs the output content of the current resolution unit. At this time, the status verification return content is obtained through the status verification entry, and it is classified into the material status semantics according to the backfill position in the current resolution unit. When the resolution unit is executed by calling the processing entry, the virtual digital human outputs the output content in the current resolution unit. At this time, the processing return content is obtained through the processing entry and is classified into the material status semantics according to the backfill position in the current resolution unit. Iterative analysis is performed based on the updated material claim semantics, the material claim semantic editing fragment is re-acquired, and the editing actions to be resolved and the objects to be resolved in the current resolution unit are retrieved from the re-acquired material claim semantic editing fragment; If there is no current edit action to be resolved and no current object to be resolved, it means that the current resolution unit has passed the recalculation, and the next resolution unit will continue to be executed. If there are still current editing actions to be resolved and current objects to be resolved, it means that the current resolution unit has not passed the recalculation. At this time, the remaining resolution units will be stopped, and the virtual digital human will be controlled to output the continuation prompts according to the output content of the current resolution unit. When all the resolution units in the resolution unit sequence have passed the recalculation, an interactive completion prompt is generated based on the output content of each resolution unit, and the virtual digital human is controlled to output the interactive completion prompt.

Images