A method and system for intelligent digital human interaction based on a web-based data dashboard
By employing multi-turn voice dialogue technology, intelligent data retrieval, and knowledge base search, the system addresses the issues of low interaction efficiency and difficulty in data understanding within large data dashboards, achieving efficient and reliable data acquisition and management decision support.
Patent Information
- Application Number
- CN202611376905.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2026-09-07
- Publication Date
- 2026-11-17
AI Technical Summary
Existing data dashboard systems suffer from low interaction efficiency, making it difficult for non-professional users to understand the meaning of the data. The voice function lacks semantic understanding and multi-turn dialogue capabilities, making it unable to accurately retrieve unstructured knowledge and resulting in unreliable output content.
Employing multi-turn voice dialogue interaction, the system uses automatic speech recognition, natural language processing, and speech synthesis technologies to convert users' spoken questions into data query commands. Combined with intelligent question counting and private knowledge base retrieval, it generates visual charts and natural language conclusions, and supports multi-level data drill-down and original text tracing.
It achieves efficient human-computer interaction, enabling non-professional users to quickly understand the meaning of data, and the results are traceable, ensuring the credibility of the output and meeting the data security and compliance requirements of enterprises.
Smart Images

Figure CN123115339A_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the field of artificial intelligence and data visualization technology, specifically relating to an intelligent digital human interaction method and system based on a web data dashboard. Background Technology
[0002] Data dashboards, data platform dashboards, and other visualization systems are widely used in enterprise management, government supervision, and industry operation monitoring to centrally display and monitor massive amounts of business data. Existing technologies primarily rely on pre-designed charts and dashboards, requiring manual clicking and drill-down to browse data. These systems typically consist of a front-end visualization engine, a back-end data service, and a database. Their working principle involves rendering structured data into static or semi-interactive graphics according to fixed templates. Some large-screen systems incorporate voice broadcasting or simple keyword search functions to assist users in obtaining information.
[0003] The existing technology has the following drawbacks: First, large screens often display stacked charts, requiring managers to switch between multiple pages and click repeatedly to locate target metrics, resulting in low efficiency in interactive data retrieval.
[0004] Second, the data is presented only in graphical form, lacking natural language interpretation of its meaning, making it difficult for non-professional users to quickly understand the business conclusions behind the data.
[0005] Third, most existing voice functions are one-way pre-set scripts that are mechanically broadcast and lack semantic understanding and multi-turn dialogue capabilities. They cannot query data and output business conclusions in real time based on users' colloquial questions.
[0006] Fourth, for unstructured knowledge such as policy documents and large-screen instructions, there is a lack of precise retrieval and original source tracing methods; directly calling large models to answer questions can easily create an AI illusion and output unreliable answers.
[0007] The root cause of the above defects is that the existing system does not deeply integrate speech recognition, natural language processing, speech synthesis, intelligent questioning, knowledge retrieval and visualization screen. It lacks an integrated technical solution that can both drive the virtual digital human image in real time and automatically convert natural language questions into data query commands. Summary of the Invention
[0008] To address the aforementioned problems in existing technologies, this invention provides an intelligent digital human interaction method and system based on a web-based data dashboard. This system enables multi-turn voice dialogue interaction with users, automatically converting spoken questions into data query commands, and generating visual charts and business conclusions. Simultaneously, it supports private unstructured knowledge retrieval and original text tracing, lowering the data acquisition threshold, ensuring data access security and compliance, improving management decision-making efficiency, and guaranteeing the security and compliance of data access.
[0009] To achieve the above objectives, the present invention provides the following technical solution: an intelligent digital human interaction method based on a web data dashboard, the method comprising the following steps: S1: The front end of the web data dashboard loads a virtual digital human image, collects user voice questions, and converts the user's voice into question text through the automatic speech recognition (ASR) module; S2. The question text is sent to the Natural Language Processing (NLP) module to perform multi-turn dialogue management with intent recognition and context tracking, and to determine whether the user's question is a data query or a knowledge question and answer type. S3. If the query is determined to be a data query, the ChatBI module is called. Through semantic modeling of natural language to structured query language (NL2SQL) / domain specific language (DSL), the natural language query is converted into a database query instruction. The query is executed to obtain business data, generate corresponding visualization charts and natural language business conclusions, and support multi-level data drill-down and root cause analysis. S4. If the question is identified as a knowledge-based question and answer type, the private knowledge base retrieval enhancement generation (RAG) module is invoked to perform vector retrieval, multi-way recall and reordering on the pre-stored unstructured document, match the answer fragment and trace back to the original document; S5. Send the output of step S3 or step S4 into the Natural Language Generation (NLG) module to generate the response text; S6, the text-to-speech (TTS) module performs text-to-speech synthesis with emotional parameters on the reply text, and outputs the audio and lip-sync parameters (Viseme). S7. Push the voice audio, lip-sync parameters, and visualization charts to the Web data dashboard to drive the virtual digital human to perform corresponding lip movements and display the visualization charts; adopt a duplex interaction mode to continuously detect user voice interruption signals during the digital human's broadcast; when a voice interruption signal is received, terminate the current broadcast and return to step S1 to receive a new round of user voice questions; if no interruption signal is detected, wait for the next question after the broadcast ends; S8's backend adopts a Spring Boot microservice architecture, using WebSockets to push real-time interactive data, employing asynchronous message queues to smooth out and decouple high-concurrency requests, and implementing row-level and column-level data permission control based on the JSON Web Token (JWT) authentication system.
[0010] As a preferred technical solution of the intelligent digital human interaction method based on Web data screen of the present invention, in step S2, the natural language processing (NLP) module calls the large language model to complete intent recognition and context dialogue management; in step S4, the private knowledge base retrieval enhancement generation (RAG) module pre-processes the uploaded unstructured documents by text splitting and vectorization and then stores them in the vector database.
[0011] As a preferred technical solution of the intelligent digital human interaction method based on Web data dashboard of the present invention, the vector database is Milvus or PgVector; the asynchronous message queue is RabbitMQ or Kafka.
[0012] As a preferred technical solution of the intelligent digital human interaction method based on Web data dashboard of the present invention, the backend simultaneously adopts a RESTful interface to handle non-real-time business requests; in step S8, a compliance audit log is generated to record user access and data call behavior.
[0013] The present invention also provides an intelligent digital human interaction system based on a web data dashboard, used to implement the intelligent digital human interaction method based on a web data dashboard described in any of the above technical solutions; The system is divided into three layers: AI brain and interaction layer, intelligent data query and knowledge retrieval layer, and Java backend architecture and full-stack integration layer; The AI brain and interaction layer include an Automatic Speech Recognition (ASR) module, a Natural Language Processing (NLP) module, a Natural Language Generation (NLG) module, and a Text-to-Speech (TTS) module. The text output from the ASR module is fed into the NLP module, the NLP module output is processed by the NLG module, and the NLG module output is input into the TTS module. The TTS module outputs audio and lip-sync parameters, supporting a duplex interaction mode. In duplex mode, the system continuously listens to and collects the user's voice signal while the digital human is speaking, and the speaking process can be interrupted by the user's voice. The intelligent query and knowledge retrieval layer includes the intelligent query (ChatBI) module and the private knowledge base retrieval enhancement generation (RAG) module; the intelligent query (ChatBI) module and the private knowledge base retrieval enhancement generation (RAG) module respectively connect to the business database and the private document data source through API interfaces; The Java backend architecture and full-stack integration layer are built on Spring Boot microservices, including WebSocket communication components, RESTful interface components, asynchronous message queue components, vector database integration units, and enterprise-level security and access control units. The enterprise-level security and access control unit implements row-level and column-level data access control based on JSON network tokens (JWT). The three-tier architecture completes data transmission and interface calls according to the preset data flow.
[0014] As a preferred technical solution of the intelligent digital human interaction system based on Web data screen of the present invention, the Natural Language Processing (NLP) module connects to the external large language model PaaS interface to realize intent recognition and multi-turn contextual dialogue management; the private knowledge base retrieval enhancement generation (RAG) module is configured with text splitting unit, vector encoding unit, multi-way recall and re-sorting unit for unstructured document storage, retrieval, and original text tracing.
[0015] As a preferred technical solution of the intelligent digital human interaction system based on Web data dashboard of the present invention, the asynchronous message queue component adopts RabbitMQ or Kafka for high-concurrency request peak shaving and service decoupling; the enterprise-level security and access control unit also includes a compliance audit log module for recording user access behavior and data call records.
[0016] As a preferred technical solution of the intelligent digital human interaction system based on Web data screen of the present invention, the system front end is deployed on Web data screen, the Web data screen renders virtual digital human image, receives and displays audio, lip-syncing (Viseme) parameters and visualization charts pushed by the back end.
[0017] Compared with the prior art, the beneficial effects of the present invention are: 1. Natural and efficient interaction: The system uses multi-turn voice dialogue to replace the traditional step-by-step click operation. Users can directly obtain data results by asking questions, which reduces the operation threshold for data acquisition and improves the efficiency of human-computer interaction.
[0018] 2. Possesses intelligent data interpretation capabilities: It can automatically generate visual charts from structured business data and output natural language business conclusions, making it easier for non-professionals to understand the business meaning behind the data and assisting management decision-making.
[0019] 3. Accurate and traceable knowledge answers: The retrieval enhancement generation scheme adopts vector retrieval, multi-way recall, and re-ranking, so that the answer results can be traced back to the original document, suppressing the AI illusion generated by the large model and ensuring the credibility of the output content.
[0020] 4. Stable and easy-to-integrate architecture: Based on Java microservice architecture, relying on asynchronous message queues to ensure high-concurrency operation, it can be smoothly integrated with existing business databases, vector libraries and AI service interfaces without the need to reconstruct the original business system.
[0021] 5. Secure, compliant and controllable: Implements fine-grained data access control at the row and column levels based on JSON network token authentication, retains audit logs, and enables sensitive data to be opened on demand, meeting enterprise data security and compliance audit requirements. Attached Figure Description
[0022] The accompanying drawings are provided to further illustrate the invention and form part of the specification. They are used in conjunction with embodiments of the invention to explain the invention and do not constitute a limitation thereof. In the drawings: Figure 1 This is a schematic diagram of the overall architecture of the intelligent digital human interaction system based on a web data dashboard according to the present invention; Figure 2 This is a schematic diagram of the digital human voice interaction and intelligent questioning process of the present invention; Detailed Implementation
[0023] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.
[0024] Example Please see Figure 1-2 The present invention provides the following technical solution: an intelligent digital human interaction method based on a web data dashboard, the method comprising the following steps: S1: The front end of the web data dashboard loads a virtual digital human image, collects user voice questions, and converts the user's voice into question text through the automatic speech recognition (ASR) module; S2. The question text is sent to the Natural Language Processing (NLP) module to perform multi-turn dialogue management with intent recognition and context tracking, and to determine whether the user's question is a data query or a knowledge question and answer type. S3. If the query is determined to be a data query, the ChatBI module is called. Through semantic modeling of natural language to structured query language (NL2SQL) / domain specific language (DSL), the natural language query is converted into a database query instruction. The query is executed to obtain business data, generate corresponding visualization charts and natural language business conclusions, and support multi-level data drill-down and root cause analysis. S4. If the question is identified as a knowledge-based question and answer type, the private knowledge base retrieval enhancement generation (RAG) module is invoked to perform vector retrieval, multi-way recall and reordering on the pre-stored unstructured document, match the answer fragment and trace back to the original document; S5. Send the output of step S3 or step S4 into the Natural Language Generation (NLG) module to generate the response text; S6, the text-to-speech (TTS) module performs text-to-speech synthesis with emotional parameters on the reply text, and outputs the audio and lip-sync parameters (Viseme). S7. Push the voice audio, lip-sync parameters, and visualization charts to the Web data dashboard to drive the virtual digital human to perform corresponding lip movements and display the visualization charts; adopt a duplex interaction mode to continuously detect user voice interruption signals during the digital human's broadcast; when a voice interruption signal is received, terminate the current broadcast and return to step S1 to receive a new round of user voice questions; if no interruption signal is detected, wait for the next question after the broadcast ends; S8's backend adopts a Spring Boot microservice architecture, using WebSockets to push real-time interactive data, employing asynchronous message queues to smooth out and decouple high-concurrency requests, and implementing row-level and column-level data permission control based on the JSON Web Token (JWT) authentication system.
[0025] Furthermore, in step S2, the Natural Language Processing (NLP) module calls the large language model to complete intent recognition and contextual dialogue management; In step S4, the private knowledge base retrieval enhancement generation (RAG) module pre-processes the uploaded unstructured documents by text splitting and vectorization before storing them in the vector database; The vector database is Milvus or PgVector; The asynchronous message queue is RabbitMQ or Kafka; The backend also uses a RESTful interface to handle non-real-time business requests; In step S8, a compliance audit log is generated simultaneously to record user access and data retrieval behavior.
[0026] The present invention also provides an intelligent digital human interaction system based on a web data dashboard, used to implement the intelligent digital human interaction method based on a web data dashboard described in the above technical solution; The system is divided into three layers: AI brain and interaction layer, intelligent data query and knowledge retrieval layer, and Java backend architecture and full-stack integration layer; AI Brain and Interaction Layer: Infusing digital humans with a "soul" and achieving a complete closed loop of voice communication and intelligent question answering. This layer includes an Automatic Speech Recognition (ASR) module, a Natural Language Processing (NLP) module, a Natural Language Generation (NLG) module, and a Text-to-Speech (TTS) module. The text output from the ASR module is fed into the NLP module, the NLP module's output is processed by the NLG module, and the NLG module's output is input into the TTS module. The TTS module outputs voice audio and lip-sync parameters (Viseme), supporting duplex interaction mode. Intelligent Data Query and Knowledge Retrieval Layer: The core of empowering business operations. Addressing the pain points of "difficult data retrieval and tedious data viewing" in the leadership dashboard, this layer introduces the Intelligent Data Query (ChatBI) module and the Private Knowledge Base Retrieval Enhanced Generation (RAG) module. The Intelligent Data Query (ChatBI) module and the Private Knowledge Base Retrieval Enhanced Generation (RAG) module respectively connect to the business database and the private document data source through API interfaces. Java backend architecture and full-stack integration layer: built on Spring Boot microservices, including WebSocket communication components, RESTful interface components, asynchronous message queue components, vector database integration units, and enterprise-level security and access control units; the enterprise-level security and access control unit implements row-level and column-level data access control based on JSON network tokens (JWT); The three-tier architecture completes data transmission and interface calls according to the preset data flow (System architecture description: This architecture adopts a layered decoupling design to ensure high availability and scalability, and each layer interacts through standardized interfaces).
[0027] Furthermore, the Natural Language Processing (NLP) module connects to the external large language model PaaS interface to achieve intent recognition and multi-turn contextual dialogue management; The Private Knowledge Base Retrieval Enhancement Generation (RAG) module is configured with text splitting units, vector encoding units, and multi-way recall and re-ranking units for unstructured document import, retrieval, and original text tracing. The asynchronous message queue component uses RabbitMQ or Kafka for peak shaving of high-concurrency requests and service decoupling; The enterprise-level security and access control unit also includes a compliance audit log module, which records user access behavior and data call records; The system front-end is deployed on a web data dashboard, which renders a virtual digital human image and receives and displays audio, lip-sync parameters (Viseme), and visual charts pushed from the back-end.
[0028] Through actual deployment and verification, the system can respond to verbal questions from managers in real time, automatically complete data queries, business interpretations, and large-screen explanations; the interaction is smooth, achieving "what you ask, you get," and it is suitable for use in enterprise and government data dashboard web screen scenarios.
[0029] In summary, this invention offers a natural and efficient interactive experience: it replaces traditional step-by-step click operations with multi-turn voice dialogue, allowing users to directly obtain data results simply by asking questions, thus lowering the operational threshold for data acquisition and improving the efficiency of human-computer interaction.
[0030] This invention possesses intelligent data interpretation capabilities: it can automatically generate visual charts from structured business data and output natural language business conclusions, making it easier for non-professionals to understand the business meaning behind the data and assisting in management decision-making.
[0031] This invention provides accurate and traceable knowledge answers: it employs a retrieval enhancement generation scheme that utilizes vector retrieval, multi-path recall, and re-ranking to ensure that the answer results are traceable to the original document, suppressing the AI illusion generated by large models and guaranteeing the credibility of the output content.
[0032] The architecture of this invention is stable and easy to integrate: based on Java microservice architecture, relying on asynchronous message queues to ensure high-concurrency operation, it can be smoothly integrated with existing business databases, vector libraries and AI service interfaces without the need to reconstruct the original business system.
[0033] This invention is secure, compliant, and controllable: it implements fine-grained data access control at the row and column levels based on JSON network token authentication, retains audit logs, and enables sensitive data to be opened on demand, meeting enterprise data security and compliance audit requirements.
[0034] In addition, module calls, network communications, and other aspects not described in detail in this embodiment are all within the scope of existing technology and common knowledge in this field, and will not be described in further detail here.
[0035] Finally, it should be noted that the above are merely preferred embodiments of the present invention and are not intended to limit the present invention. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art can still modify the technical solutions described in the foregoing embodiments or make equivalent substitutions for some of the technical features. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.
Claims
1. A method for intelligent digital human interaction based on a Web data dashboard, characterized in that: The method includes the following steps: S1: The front end of the web data dashboard loads a virtual digital human image, collects user voice questions, and converts the user's voice into question text through the automatic speech recognition (ASR) module; S2. The question text is sent to the Natural Language Processing (NLP) module to perform multi-turn dialogue management with intent recognition and context tracking, and to determine whether the user's question is a data query or a knowledge question and answer type. S3. If the query is determined to be a data query, the ChatBI module is called. Through semantic modeling of natural language to structured query language (NL2SQL) / domain specific language (DSL), the natural language query is converted into a database query instruction, the query is executed to obtain business data, and corresponding visualization charts and natural language business conclusions are generated. Multi-level data drill-down and root cause analysis are also supported. S4. If the question is identified as a knowledge question and answer type, the private knowledge base retrieval enhancement generation (RAG) module is called to perform vector retrieval, multi-way recall and reordering on the pre-stored unstructured document, match the answer fragment and trace back to the original document; S5. Send the output of step S3 or step S4 into the Natural Language Generation (NLG) module to generate the response text; S6, the text-to-speech (TTS) module performs text-to-speech synthesis with emotional parameters on the reply text, and outputs the audio and lip-sync parameters (Viseme). S7. Push the voice audio, lip-sync parameters, and visualization charts to the Web data dashboard to drive the virtual digital human to perform corresponding lip movements and display the visualization charts; adopt a duplex interaction mode to continuously detect user voice interruption signals during the digital human's broadcast; when a voice interruption signal is received, terminate the current broadcast and return to step S1 to receive a new round of user voice questions; if no interruption signal is detected, wait for the next question after the broadcast ends; S8's backend adopts a Spring Boot microservice architecture, using WebSockets to push real-time interactive data, employing asynchronous message queues to smooth out and decouple high-concurrency requests, and implementing row-level and column-level data permission control based on the JSON Web Token (JWT) authentication system.
2. The intelligent digital human interaction method based on a Web data dashboard according to claim 1, characterized in that: In step S2, the Natural Language Processing (NLP) module calls the large language model to complete intent recognition and contextual dialogue management; in step S4, the Private Knowledge Base Retrieval Enhanced Generation (RAG) module pre-processes the uploaded unstructured documents by text splitting and vectorization before storing them in the vector database.
3. The intelligent digital human interaction method based on a Web data dashboard according to claim 1, characterized in that: The vector database is Milvus or PgVector; the asynchronous message queue is RabbitMQ or Kafka.
4. The intelligent digital human interaction method based on a Web data dashboard according to claim 1, characterized in that: The backend also uses a RESTful interface to handle non-real-time business requests; in step S8, compliance audit logs are generated to record user access and data call behavior.
5. An intelligent digital human interaction system based on a web-based data dashboard, characterized in that: Used to implement the intelligent digital human interaction method based on a web data dashboard as described in any one of claims 1-4; The system is divided into three layers: AI brain and interaction layer, intelligent data query and knowledge retrieval layer, and Java backend architecture and full-stack integration layer; The AI brain and interaction layer include an Automatic Speech Recognition (ASR) module, a Natural Language Processing (NLP) module, a Natural Language Generation (NLG) module, and a Text-to-Speech (TTS) module. The text output from the ASR module is fed into the NLP module, the output from the NLP module is processed by the NLG module, and the text output from the NLG module is input into the TTS module. The TTS module outputs speech audio and lip-sync parameters, supporting a duplex interaction mode. The intelligent question counting and knowledge retrieval layer includes the intelligent question counting (ChatBI) module and the private knowledge base retrieval enhancement generation (RAG) module; The ChatBI module and the RAG module for enhanced private knowledge base retrieval connect to the business database and the private document data source via API interfaces, respectively. The Java backend architecture and full-stack integration layer are built on Spring Boot microservices, including WebSocket communication components, RESTful interface components, asynchronous message queue components, vector database integration units, and enterprise-level security and access control units. The enterprise-level security and access control unit implements row-level and column-level data access control based on JSON network tokens (JWT). The three-tier architecture completes data transmission and interface calls according to the preset data flow.
6. The intelligent digital human interaction system based on a Web data dashboard according to claim 5, characterized in that: The Natural Language Processing (NLP) module connects to the external large language model PaaS interface to realize intent recognition and multi-turn contextual dialogue management; the private knowledge base retrieval enhancement generation (RAG) module is configured with text splitting unit, vector encoding unit, multi-way recall and re-ranking unit for unstructured document storage, retrieval, and original text tracing.
7. The intelligent digital human interaction system based on a Web data dashboard according to claim 5, characterized in that: The asynchronous message queue component uses RabbitMQ or Kafka to handle high-concurrency request peak shaving and service decoupling; the enterprise-level security and access control unit also includes a compliance audit log module to record user access behavior and data call records.
8. The intelligent digital human interaction system based on a Web data dashboard according to claim 5, characterized in that: The system front-end is deployed on a web data dashboard, which renders a virtual digital human image and receives and displays audio, lip-sync parameters (Viseme), and visual charts pushed from the back-end.
