Customer service omni-channel-oriented multi-modal arrangement system and customer service omni-channel-oriented multi-modal arrangement method

Abstract

The invention discloses a customer service omni-channel-oriented multi-modal arrangement system, and the system comprises an omni-channel access module which integrates ASR, OCR and MLLM technologies and is used for translating multi-modal data input into unified text semantic representation in real time; the logic arrangement module integrates NLP, LLM and IT interface components, supports extension and is used for logic arrangement based on business requirements, and the arranged service has multiple mechanisms and can realize a low-delay and high-availability service chain; the multi-modal generation module integrates TTS, T2I and MLLM technologies and is used for generating corresponding modal data according to channel requirements and supporting streaming and non-streaming output; and the full-process service monitoring module is used for monitoring the state, the time delay and the error rate of each module in real time and guaranteeing the service stability and the user experience. The cross-modal data unified processing is effectively realized, the system development and maintenance cost is reduced, and the customer service efficiency and quality are improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of artificial intelligence, in particular to a multi-modal arrangement system and method for customer service full channel. BACKGROUND

[0002] In today's customer service field, in order to provide more comprehensive services to customers, enterprises often interact with customers through multiple channels. However, the traditional customer service system has many problems when facing these situations.

[0003] Multi-modal adaptation is complex, and the traditional customer service system needs to develop an adaptation layer independently for different channels, which leads to redundant system architecture and low development efficiency. At the same time, the data of each channel is difficult to analyze and make decisions uniformly, and the value of data cannot be fully utilized.

[0004] High development and maintenance cost, the traditional system relies on manual coding, not only the development cycle is long, the iteration efficiency is also low. Moreover, the maintenance of multiple systems needs to consume a lot of manpower, material resources and financial resources, when new channels need to be added or optimized, it is easy to cause waste of resources.

[0005] Limited intelligent level, due to the lack of unified processing framework, cross-channel collaborative analysis and knowledge sharing cannot be realized, which makes the customer service response efficiency low, the user experience inconsistent, and the customer's demand cannot be well met. SUMMARY

[0006] In view of the above defects of the prior art, the present application aims to solve the problems of complex multi-modal adaptation, high development and maintenance cost, and limited intelligent level of the traditional customer service platform. An intelligent and highly integrated multi-modal customer service unified processing platform is provided. In order to achieve the above purpose, the present application provides a multi-modal arrangement system for customer service full channel, comprising: A full-channel access module integrates ASR, OCR, MLLM technology, and is used for real-time translation of voice, image, video and other inputs into unified text semantic representation; A logic arrangement module integrates NLP, LLM, IT interface components, supports extension, and is used for logic arrangement based on business requirements, and the arranged service has multiple mechanisms, which can realize low latency, high availability service chain; A multi-modal generation module integrates TTS, T2I, MLLM technology, and is used for generating corresponding modal data according to channel requirements, supporting streaming and non-streaming output; A full-process service monitoring module is used to monitor the state, delay and error rate of each module in real time, and to ensure service stability and user experience.

[0007] Further, the full-channel access module can access one or more input information of telephone, text, video, and APP message.

[0008] Further, the logic arrangement module has a visual process configuration interface, and logic arrangement is performed by means of dragging components and connecting logic nodes.

[0009] Further, the multi-modal generation module generates modal data including one or more of text, voice, image and video.

[0010] Further, the all-process service monitoring module monitors the states of each module, including the running state and the abnormal state.

[0011] Further, the service arranged in the logic arrangement module has mechanisms including load balancing, fusing and degradation.

[0012] The application further provides a multi-modal arrangement method for customer service full channels, including the following steps: Step one: through the ASR, OCR and MLLM technologies integrated by the full-channel access module, input voice, image and video are converted into unified text semantic representation in real time; Step two: based on business requirements, logic arrangement is performed by using the NLP, LLM and IT interface components integrated by the logic arrangement module, the arranged service has multiple mechanisms, and a service chain with low delay and high availability is realized; Step three: according to channel requirements, corresponding modal data is generated by means of the TTS, T2I and MLLM technologies integrated by the multi-modal generation module, and stream and non-stream outputs are supported; Step four: through the all-process service monitoring module, the states, time delay and error rate of each module are monitored in real time, and the service stability and user experience are ensured.

[0013] Further, the input in step one comes from one or more customer service channels of telephone, text, video and APP message.

[0014] Further, the logic arrangement in step two is performed by means of dragging components and connecting logic nodes through the visual process configuration interface.

[0015] Further, the modal data generated in step three includes one or more of text, voice, image and video.

[0016] The technical scheme can reduce development cost and shorten development time, adopts a configurable service framework and modular design, reduces the workload of manual coding, shortens the development cycle, and through practice verification, the development cycle can be shortened by 40%-60%. Reduce operation and maintenance cost, the whole process service monitoring module can monitor the state and data of each module in real time, quickly locate the problem module, and reduce the troubleshooting time and cost in the maintenance process. Enhance system scalability, the logical arrangement module supports component expansion, and the full-channel access module can adapt to multiple channels, support flexible access to new channels, and meet the needs of enterprise business expansion. Improve business agility, business departments can independently adjust service processes through a visual process configuration interface without relying on technical personnel for a large amount of coding work, improving the business response speed. It has wide applicability and can be applied to customer service systems in multiple fields and scenarios, providing efficient and high-quality customer service solutions for enterprises in different industries.

[0017] The concept, specific structure and technical effects of the present application will be further described below in conjunction with the drawings to fully understand the purpose, features and effects of the present application. BRIEF DESCRIPTION OF DRAWINGS

[0018] Figure 1 is a structural schematic diagram of a preferred embodiment of the present application; Figure 2 is a structural schematic diagram of another preferred embodiment of the present application; Figure 3 is a structural schematic diagram of another preferred embodiment of the present application. DETAILED DESCRIPTION

[0019] The following describes several preferred embodiments of the present application with reference to the drawings to make the technical content of the present application clearer and easier to understand. The present application can be embodied in many different forms, and the scope of protection of the present application is not limited to the embodiments described herein.

[0020] In the drawings, components of the same structure are denoted by the same reference numerals, and components having similar structures or functions are denoted by similar reference numerals. The size and thickness of each component shown in the drawings are arbitrarily shown, and the present application does not limit the size and thickness of each component. In order to make the drawing clearer, the thickness of some components is appropriately exaggerated in some places in the drawing.

[0021] Glossary: Full-channel customer service: Enterprises provide seamless and unified customer service experience for customers through multiple service channels (such as telephone customer service, text customer service, video customer service, APP message, etc.).

[0022] Multi-modal: usually refers to a service mode that integrates multiple interaction methods, such as the comprehensive application of multiple media such as text, voice, image, and video.

[0023] Orchestration system: A visual process configuration interface that allows for flexible configuration of service processes by dragging and dropping components and connecting logical nodes. Complex business processes can be built and adjusted simply by writing code.

[0024] ASR (Automatic Speech Recognition) technology: Automatic speech recognition, a technology that converts speech into text. OCR (Optical Character Recognition) technology: Optical character recognition, a technology for recognizing text content from images / scanned documents. MLLM (Multimodal Large Language Models) technology: A multimodal large language model that can simultaneously process multiple input formats such as text, images, and audio. NLP (Natural Language Processing): A general term for technologies that enable computers to understand, interpret, and generate human language. LLM (Large Language Models): Large language models, deep learning models trained on massive amounts of text data. TTS (Text-to-Speech): A technology that converts text content into natural speech output. T2I (TextToImage): A technique for generating images from text descriptions. Hot deployment technology: Replace old processes and enable new processes in real time without pausing or restarting services. Version management: Records historical versions of the process, allowing for rollback at any time.

[0025] Historical versions will be retained, and if there are any issues after the new version is released, you can roll back directly.

[0026] Delayed triggering: An internal interface of the process that allows for delayed node calls.

[0027] Timeout response: Set a maximum timeout for each component (e.g., a 3-second timeout limit for LLM calls), and automatically skip or return a degraded result if the timeout occurs.

[0028] Failure retry mechanism: Automatic retry for temporary errors (such as network jitter) using an exponential backoff strategy (1s / 3s / 9s).

[0029] Streaming output: content is generated segment by segment and returned in real time (e.g., LLM outputs answers word by word).

[0030] Non-streaming output: Returns the complete result at once (traditional request-response model).

[0031] Multimodal output: Supports mixed text / image / speech output (such as LLM to generate copy, image generation component to create posters, TTS to synthesize speech).

[0032] like Figure 1 As shown in Example 1, the omnichannel access module integrates ASR, OCR, and MLLM technologies. Based on user input, it translates voice, image, and video inputs into a unified text semantic representation in real time. For voice, image, and video inputs, ASR, OCR, and MLLM technologies are used respectively to translate them into text. A standardized semantic representation layer is designed for unified multimodal translation, converting different modal input data (voice, video, images, etc.) into an intermediate semantic representation, enabling multiple channels to connect to a single system.

[0033] The logic orchestration module integrates components such as small-model NLP, large-model LLM, and IT interfaces. It supports the addition of extended components, such as knowledge base components, real-time monitoring components, caching components, and business rule components. Based on business needs, it performs component logic orchestration and logical processing on the text translated by the omnichannel access module, ensuring business consistency and allowing all channels to share the same logic, avoiding redundant development. The new business launch cycle is shortened from weeks to days. The orchestrated services support hot deployment technology, version management, delayed triggering, timeout response, and failure retry mechanisms, enabling seamless business upgrades, second-level rollback from failures, shifting computational pressure to off-peak periods, preventing failures from blocking processes, and increasing service response success rates, thereby achieving a low-latency, highly available service chain. Multimodal Generation Module: Based on channel requirements, the multimodal module integrates TTS, T2I, and MLLM technologies. It generates corresponding modal data from the text output by the logic orchestration module according to user needs, supporting both streaming and non-streaming multimodal output. It can display data in real time to improve user experience or obtain full data for data processing. Through a visual, low-code orchestration engine, it enables rapid service configuration and deployment. Through version management and hot deployment technology, it can achieve second-level updates and rollbacks of business logic, greatly improving the system's flexibility and reliability.

[0034] End-to-end service monitoring module: Monitors the status, latency, and error rate of each module in real time to ensure service stability and user experience. Adopts a modular design, significantly reducing backend code development; enables end-to-end streaming processing capabilities.

[0035] Example 2: A multimodal orchestration method for customer service across all channels, comprising the following steps: Step 1: Through the ASR, OCR, and MLLM technologies integrated in the omnichannel access module, voice, image, and video inputs are translated into a unified text semantic representation in real time; Step 2: Utilize the NLP, LLM, and IT interface components integrated in the logic orchestration module to perform logic orchestration based on business requirements. The orchestrated services have multiple mechanisms to achieve a low-latency, highly available service chain. Step 3: Utilize the TTS, T2I, and MLLM technologies integrated into the multimodal generation module to generate corresponding modal data based on channel requirements, supporting both streaming and non-streaming output; Step 4: Monitor the status, latency, and error rate of each module in real time through the end-to-end service monitoring module to ensure service stability and user experience.

[0036] Preferably, the input in step one comes from one or more customer service channels, such as telephone, text, video, or in-app messages.

[0037] Preferably, the logic orchestration in step two is carried out through a visual process configuration interface by dragging and dropping components and connecting logic nodes.

[0038] Preferably, the modal data generated in step three includes one or more of text, speech, images, and videos. This invention introduces a pigeon-inspired bionic algorithm into clustering calculations to avoid the original clustering methods getting trapped in local optima during log information analysis. The advantages of this technical solution are fast convergence speed and high accuracy, improving analysis efficiency.

[0039] like Figure 2 As shown in Example 3: A user makes a phone call to inquire about phone charges. When a user calls to inquire about call charges, the workflow of this system is as follows: The omnichannel access module uses integrated ASR technology to translate users' voice input into a unified text semantic representation in real time.

[0040] The logic orchestration module integrates LLM and related IT interface components. Staff can pre-arrange the logical flow for handling call charge inquiries through a visual process configuration interface. Upon receiving the translated text information, the logic orchestration module calls the LLM to obtain the user's specific needs, then calls the corresponding IT interface to obtain the user's call charge data, and generates a script to inform the user of the call charge details.

[0041] The multimodal generation module integrates TTS technology, which converts the generated scripts into speech according to the needs of the telephone channel.

[0042] The end-to-end service monitoring module monitors in real time the voice translation latency of the omnichannel access module, the interface call error rate of the logic orchestration module, and the voice generation status of the multimodal generation module, ensuring the stable operation of the entire service process. Finally, the voice message is delivered to the user via telephone to complete the call charge consultation service.

[0043] like Figure 3As shown in Example 4: Users upload screenshots of bills in the app to appeal abnormal charges. When a user uploads a screenshot of their bill to the app to appeal an abnormal charge, the system's workflow is as follows: The omnichannel access module uses integrated OCR technology to process user-uploaded bill screenshots, translating the information in the images into a unified text semantic representation.

[0044] The logic orchestration module has a pre-arranged logic flow for handling abnormal deduction appeals. After receiving the translated text information, it calls the LLM to analyze the user's appeal content, obtain key information, and then calls the corresponding IT interface to obtain the user's detailed billing data and generate a script to explain the abnormal deduction situation.

[0045] Based on the characteristics of the APP channel, the multimodal generation module directly outputs the generated scripts in text form.

[0046] The end-to-end service monitoring module monitors the accuracy of OCR translation, the latency of API calls, and the operating status of each module in real time to ensure the normal operation of the service. Finally, the text message is returned to the user within the APP to complete the abnormal deduction appeal service.

[0047] As can be seen from the above specific implementation scenarios, the present invention can effectively achieve unified processing of cross-modal data, reduce system development and maintenance costs, improve the efficiency and quality of customer service, and has good practical value.

[0048] The preferred embodiments of the present invention have been described in detail above. It should be understood that those skilled in the art can make numerous modifications and variations based on the concept of the present invention without creative effort. Therefore, all technical solutions that can be obtained by those skilled in the art based on the concept of the present invention through logical analysis, reasoning, or limited experimentation on the basis of existing technology should be within the scope of protection defined by the claims.

Claims

1. A multimodal orchestration system for customer service across all channels, characterized in that, include: The omnichannel access module integrates ASR, OCR, and MLLM technologies to translate multimodal data input into a unified text semantic representation in real time. The logic orchestration module integrates NLP, LLM, and IT interface components, supports expansion, and is used for logical orchestration based on business needs. The orchestrated services have multiple mechanisms to achieve a low-latency, highly available service chain. The multimodal generation module integrates TTS, T2I, and MLLM technologies to generate corresponding modal data according to channel requirements, supporting both streaming and non-streaming output. The end-to-end service monitoring module is used to monitor the status, latency, and error rate of each module in real time, ensuring service stability and user experience.

2. The system according to claim 1, characterized in that, The omnichannel access module can access one or more types of input information, including phone calls, text messages, video messages, and in-app messages.

3. The system according to claim 1, characterized in that, The logic orchestration module has a visual process configuration interface, which allows for logic orchestration by dragging and dropping components and connecting logic nodes.

4. The system according to claim 1, characterized in that, The modal data generated by the multimodal generation module includes one or more of the following: text, speech, image, and video.

5. The system according to claim 1, characterized in that, The status of each module monitored by the full-process service monitoring module includes running status and abnormal status.

6. The system according to claim 1, characterized in that, The orchestrated services in the logical orchestration module have mechanisms including load balancing, circuit breaking, and degradation.

7. A multimodal orchestration method for customer service across all channels, characterized in that, Includes the following steps: Step 1: Through the ASR, OCR, and MLLM technologies integrated in the omnichannel access module, voice, image, and video inputs are translated into a unified text semantic representation in real time; Step 2: Utilize the NLP, LLM, and IT interface components integrated in the logic orchestration module to perform logic orchestration based on business requirements. The orchestrated services have multiple mechanisms to achieve a low-latency, highly available service chain. Step 3: Utilize the TTS, T2I, and MLLM technologies integrated into the multimodal generation module to generate corresponding modal data based on channel requirements, supporting both streaming and non-streaming output; Step 4: Monitor the status, latency, and error rate of each module in real time through the end-to-end service monitoring module to ensure service stability and user experience.

8. The method according to claim 7, characterized in that, The input in step one comes from one or more customer service channels, such as phone, text, video, and in-app messages.

9. The method according to claim 7, characterized in that, The logic orchestration in step two is carried out through a visual process configuration interface by dragging and dropping components and connecting logic nodes.

10. The method according to claim 7, characterized in that, The modal data generated in step three includes one or more of the following: text, speech, image, and video.

Images