Intelligent cockpit interaction method and system based on multi-modal artificial intelligence

Abstract

The invention discloses an intelligent cockpit interaction method and system based on multi-modal artificial intelligence, and relates to the technical field of data interaction. In order to solve the problems of poor interactivity, low flexibility, poor user experience and the like of a traditional intelligent cockpit, the adopted scheme comprises the steps that user voice data are acquired, voice instruction recognition is achieved based on TensorFlow.js, and recognized voice is converted into a text; analyzing the text by using the large model, calling a data analysis service, generating a data set, and recommending a data presentation mode; the large model generates different types of statistical charts and interpretation texts of the statistical charts according to the data set and the data presentation mode; the digital human video generation service obtains the interpretation text and converts the interpretation text into a video which contains a digital human image and has an information broadcasting function; and sending the data set and the video to a front-end interface of the intelligent cockpit for displaying and broadcasting. The method is used for improving the interaction efficiency and the experience effect.

Description

Technical Field

[0001] The present invention relates to the technical field of cockpit interaction, and in particular to an intelligent cockpit interaction method and system based on multimodal artificial intelligence. Background Art

[0002] In the field of enterprise management and decision support, traditional intelligent cockpits have long played an important role. However, they have many limitations in data presentation and interaction, which seriously restrict managers from efficiently obtaining and using key information.

[0003] Traditional smart cockpits mainly rely on static data reports and chart displays. These reports and charts are usually generated according to pre-set formats and cycles. Once the data is determined, it is difficult to flexibly change it during the display process. If users want to view information on different topics, they often need to manually switch large screens. This process is not only cumbersome, but may also waste valuable time in emergency decision-making moments. If the existing display content cannot meet specific needs, it may even need to be redesigned and developed, which involves complex technical processes and a long development cycle, consuming a lot of manpower, material resources and time costs.

[0004] The disadvantages of this traditional model are reflected in many aspects: 1. Poor interactivity: The interaction between users and the system is extremely limited, and there is a lack of natural language interaction functions. For example, managers cannot directly ask the system "the comparison of sales in each region last quarter" through voice as in daily conversations, but must go through complex operation processes, search and switch between different menus and pages to obtain the required information. This undoubtedly increases the difficulty and time cost of obtaining information for managers who work at a fast pace and have great decision-making pressure, and greatly reduces work efficiency.

[0005] 2. Low flexibility: The data display method is highly fixed and difficult to customize according to the personalized needs of different managers. Different managers have different focuses and decision-making styles. Some managers may focus more on the real-time changes in sales data, while others pay more attention to the details of the cost structure. However, the traditional smart cockpit cannot easily meet these diverse needs. All users see a unified data display mode, which cannot highlight the key information of personal concern, seriously affecting the pertinence and effectiveness of decision-making.

[0006] 3. Single data source: Mainly relying on traditional data dimension modeling and statistical results, usually limited to some business system data within the enterprise. For example, only financial statement data is obtained from the financial system, and sales performance data is obtained from the sales system, etc., lacking the integration of multi-source information such as external market data, industry dynamics data, and social media data. In today's highly competitive and rapidly changing market environment, a single data source makes it difficult for managers to fully understand the internal and external environment of the enterprise, which may lead to insufficient decision-making basis and inability to respond to market changes and competitive challenges in a timely manner.

[0007] 4. Poor user experience: Interface design often lacks intuitiveness and friendliness. A large amount of data is piled up on the screen, the information layout is chaotic, and there is a lack of effective information stratification and guidance. It is difficult for managers to quickly locate key information in a short period of time. At the same time, the way of information transmission is limited to static charts and texts, lacking dynamic demonstrations, visual interactions and other means, making the whole experience boring and difficult to attract users to explore the meaning behind the data in depth, reducing managers' willingness and reliance on the smart cockpit.

[0008] In recent years, with the gradual maturity of large models (LLM), data multi-dimensional analysis technology, digital human technology and voice recognition technology, a new solution has been provided for the innovation of intelligent cockpits. With its powerful natural language processing capabilities, large models can understand users' complex language instructions and convert them into precise data query and analysis tasks. For example, if a manager says "analyze the sales growth trends of different product lines in different regions in the past year and compare them with the data of competitors in the same period", the large model can quickly parse the instructions, extract relevant information from multi-source data and conduct in-depth analysis.

[0009] Data multi-dimensional analysis technology breaks the limitations of traditional data dimensions and can slice, dice, drill, and drill data from multiple angles. Managers can freely choose data dimensions of interest, such as time, region, product category, customer group, etc., flexibly combine and view the results of cross-analysis of different dimensions in real time, and deeply explore the rules and trends hidden behind the data to provide more comprehensive and in-depth insights for decision-making.

[0010] Digital human technology adds a vivid and intuitive interactive experience to the smart cockpit. Digital humans can act as virtual assistants, appearing in the interface in an anthropomorphic image, and interact with managers through voice and action. For example, digital humans can use vivid voice to broadcast key data indicators, guide managers to view important information through body movements, and even simulate and demonstrate scenarios based on data analysis results, making information transmission more vivid, interesting, and easy to understand.

[0011] The combination of voice recognition technology and the above technologies enables natural voice interaction between managers and the intelligent cockpit. Managers do not need to manually operate the keyboard or mouse, but can complete operations such as data query, report generation, and interface switching through voice commands. This not only greatly improves the convenience of operation, but also enables managers to obtain the required information anytime and anywhere in busy work scenarios, such as in meetings or on the way out, truly realizing seamless information acquisition.

[0012] The integrated application of these emerging technologies has significantly improved the data processing and analysis capabilities of the intelligent cockpit, raising the intelligence level of the system to a new level. They provide managers with more efficient, convenient and personalized decision support tools, helping companies seize opportunities in a complex and changing market environment and make more scientific and accurate decisions. Summary of the invention

[0013] In order to solve the problems of poor interactivity, low flexibility, single data source and poor user experience in traditional smart cockpits, the present invention provides a smart cockpit interaction method and system based on multimodal artificial intelligence. Through the application of multimodal artificial intelligence technology, the interactive experience, flexibility, real-time and safety of the smart cockpit are improved, providing enterprises with a more efficient and intelligent data analysis tool.

[0014] In the first aspect, the present invention provides an intelligent cockpit interaction method based on multimodal artificial intelligence, and the technical solution adopted to solve the above technical problems is as follows: An intelligent cockpit interaction method based on multimodal artificial intelligence comprises the following steps: S1. Obtain user voice data, implement voice command recognition based on TensorFlow.js, and convert the recognized user voice into text; S2. Use the large model LLM to parse the text, call the API interface of the data analysis service, generate a data set corresponding to the analysis result, and recommend the data presentation method based on the data structure of the data set; S3, the large model LLM generates different types of statistical charts based on the data set and the recommended data presentation method, and at the same time, generates interpretation texts of the statistical charts; S4. Digital Human Video Generation Service obtains the interpretation text generated by the large model LLM, and uses the preset algorithm to convert the interpretation text into a video containing the digital human image and with information broadcasting function; S5. Send the data set generated by the large model LLM and the video converted by the digital human video generation service to the front-end interface of the intelligent cockpit for display and broadcast.

[0015] Optionally, the step S1 involved specifically includes: S1.1. Obtain audio data stream segments by monitoring the microphone on the browser side; S1.2. Load the pre-trained voice command model based on TensorFlow.js and perform voice command recognition on the audio data stream segment; S1.3. After completing speech recognition through the voice command model, call the audio-to-text service to convert the recognized audio data stream into text.

[0016] Optionally, the step S2 involved specifically includes: S2.1. Use the natural language understanding capability of the large model LLM to extract user needs or query requests from the text; S2.2. Call the API interface of the data analysis service and search for the statistical interface matching the text in the indicator knowledge base. (a) If a statistical interface matching the text is found in the indicator knowledge base, the request path and required interface parameters of the statistical interface are obtained, and step S2.3 is executed; (b) If no statistical interface matching the text word is found in the indicator knowledge base, the text is used as a prompt word, and the data in the industry knowledge base is used to generate a data set based on the retrieval enhancement generation technology of the large model LLM; S2.3. parse the text using the large model LLM, extract the specific value corresponding to the interface parameter, use the specific value to call the request path, and obtain statistical data; S2.4. Use the large model LLM to further analyze and process the statistical data to generate a data set.

[0017] Further optionally, when executing step S2, the large model LLM uses retrieval enhancement generation technology to retrieve the query interface of data indicators from the knowledge base and assist in generating a data set; wherein the knowledge base includes an industry knowledge base and an indicator knowledge base, the industry knowledge base contains the statistical annual report of the specified industry, and the indicator knowledge base is constructed based on the statistical interface request path, statistical indicators and interface description information.

[0018] Further optionally, step S2 is performed, and the data set includes the following fields: Figure title; Data labels; Presentation methods, including bar charts, line charts, and pie charts; A dataset containing actual data values, used to generate the chart; The interpretation text of the dataset is used to provide a detailed interpretation of the dataset.

[0019] In the second aspect, the present invention is an intelligent cockpit interaction system based on multimodal artificial intelligence, and the technical solution adopted to solve the above technical problems is as follows: An intelligent cockpit interaction system based on multimodal artificial intelligence, characterized in that it includes: Audio collection module, users obtain user voice data; The speech recognition module is used to implement voice command recognition based on TensorFlow.js and convert the recognized user speech into text; The text processing module is used to parse text using the large model LLM, call the API interface of the data analysis service, generate a data set corresponding to the parsing result, and recommend a data presentation method based on the data structure of the data set; The chart generation module is used to generate different types of statistical charts using the large model LLM according to the data set and the recommended data presentation method, and at the same time, generate the interpretation text of the statistical charts; The video generation module is used to obtain the interpretation text generated by the large model LLM using the digital human video generation service, and use the preset algorithm to convert the interpretation text into a video containing the digital human image and having the information broadcasting function; The transmission and display module is used to send the data set generated by the large model LLM and the video converted by the digital human video generation service to the front-end interface of the intelligent cockpit for display and broadcast.

[0020] Optionally, the audio acquisition module involved obtains audio data stream segments by monitoring a microphone on the browser side; The speech recognition module first loads the pre-trained speech command model based on TensorFlow.js, performs speech command recognition on the audio data stream segment, and then calls the audio-to-text service to convert the recognized audio data stream into text.

[0021] Optionally, the specific execution process of the text processing module involved includes: Leverage the natural language understanding capabilities of the large model LLM to extract user needs or query requests from the text; Call the API interface of the data analysis service and search for the statistical interface matching the text in the indicator knowledge base. (a) If a statistical interface matching the text is found in the indicator knowledge base, the request path and required interface parameters of the statistical interface are obtained, and the subsequent steps are continued; (b) If no statistical interface matching the text word is found in the indicator knowledge base, the text is used as a prompt word, and the data in the industry knowledge base is used to generate a data set based on the retrieval enhancement generation technology of the large model LLM; Use the large model LLM to parse the text, extract the specific value corresponding to the interface parameter, use the specific value to call the request path, and obtain statistical data; The statistical data are further analyzed and processed using the large model LLM to generate a data set.

[0022] Further optionally, when executing step (b), the large model LLM uses retrieval enhancement generation technology to retrieve the query interface of data indicators from the knowledge base and assist in generating a data set; wherein the knowledge base includes an industry knowledge base and an indicator knowledge base, the industry knowledge base contains the statistical annual report of the specified industry, and the indicator knowledge base is constructed based on the statistical interface request path, statistical indicators and interface description information.

[0023] Preferably, the data set involved contains the following fields: Figure title; Data labels; Presentation methods, including bar charts, line charts, and pie charts; A dataset containing actual data values, used to generate the chart; The interpretation text of the dataset is used to provide a detailed interpretation of the dataset.

[0024] The intelligent cockpit interaction method and system based on multimodal artificial intelligence of the present invention has the following beneficial effects compared with the prior art: 1. The present invention can realize multimodal interaction through technologies such as voice command recognition, voice-to-text, large model LLM analysis, multidimensional data analysis, and digital human broadcasting, which significantly improves the interactive experience, flexibility, real-time performance, and safety of the intelligent cockpit; 2. The present invention can realize human-computer dialogue-style interaction; can dynamically generate statistical charts according to user needs, and support multiple display forms; supports querying data from existing databases, and can also parse data from large model knowledge bases; uses digital human broadcasting and voice command functions to improve user convenience and satisfaction. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] Attached Figure 1 is a flow chart of a method according to Embodiment 1 of the present invention; Attached Figure 2 This is a module connection block diagram of the second embodiment of the present invention. DETAILED DESCRIPTION

[0026] In order to make the technical solution, the technical problem solved and the technical effect of the present invention more clearly understood, the technical solution of the present invention is clearly and completely described below in conjunction with specific embodiments.

[0027] Embodiment 1: Refer to the attached Figure 1 , the embodiment proposes an intelligent cockpit interaction method based on multimodal artificial intelligence, which includes the following steps: S1. Voice acquisition, recognition and conversion: Monitor the microphone on the browser side and obtain the audio data stream fragment; Load the pre-trained voice command model based on TensorFlow.js to perform voice command recognition on the audio data stream segment; After completing speech recognition through the voice command model, call the audio-to-text service to convert the recognized audio data stream into text.

[0028] S2. Use the large model LLM to parse the text, call the API interface of the data analysis service, generate a data set corresponding to the analysis result, and recommend data presentation methods (such as bar charts, line charts or bar graphs) based on the data structure of the data set. Use the large model LLM to parse the text, call the API interface of the data analysis service, and generate a data set corresponding to the analysis result. This process specifically includes: S2.1. Use the natural language understanding capability of the large model LLM to extract user needs or query requests from the text; S2.2. Call the API interface of the data analysis service and search for the statistical interface matching the text in the indicator knowledge base. (a) If a statistical interface matching the text is found in the indicator knowledge base, the request path and required interface parameters of the statistical interface are obtained, and step S2.3 is executed; (b) If no statistical interface matching the text word is found in the indicator knowledge base, the text is used as a prompt word, and based on the retrieval enhancement generation technology of the large model LLM, the query interface of the data indicator is retrieved from the knowledge base, and the data set is assisted in generating; wherein the knowledge base includes an industry knowledge base and an indicator knowledge base, the industry knowledge base contains the statistical annual report of the specified industry, and the indicator knowledge base is constructed based on the statistical interface request path, statistical indicators and interface description information; S2.3. parse the text using the large model LLM, extract the specific value corresponding to the interface parameter, use the specific value to call the request path, and obtain statistical data; S2.4. Use the large model LLM to further analyze and process the statistical data to generate a data set.

[0029] The data set generated by the process of step S2.2 (b) and step S2.4 contains the following fields: Figure title; Data labels; Presentation methods, including bar charts, line charts, and pie charts; A dataset containing actual data values, used to generate the chart; The interpretation text of the dataset is used to provide a detailed interpretation of the dataset.

[0030] S3. The large model LLM generates different types of statistical charts based on the data set and the recommended data presentation method, and at the same time, generates text to interpret the statistical charts.

[0031] S4. The digital human video generation service obtains the interpretation text generated by the large model LLM, and uses a preset algorithm to convert the interpretation text into a video containing a digital human image and having information broadcasting function.

[0032] S5. Send the data set generated by the large model LLM and the video converted by the digital human video generation service to the front-end interface of the intelligent cockpit for display and broadcast.

[0033] Based on the above steps, suppose an e-commerce company manager wants to know the recent sales data. The specific execution process is as follows: 1. The manager opens the smart cockpit front-end interface of the internal system through the browser, clicks the voice input button, and says to the microphone: "I want to know the sales and sales volume of each category of goods in different regions in the past three months." At this time, the front-end interface begins to capture the manager's voice and converts it into audio data stream fragments. TensorFlow.js uses the pre-trained voice command model to recognize these fragments and determine that this is a voice command for data query.

[0034] 2. The recognized voice command is further converted into text: "I want to know the sales and sales volume of each category of goods in different regions in the past three months." The text is passed to the subsequent data processing flow.

[0035] 3. The converted text is input into the big model LLM as a prompt word. The big model LLM uses natural language understanding to conduct an in-depth analysis of the text and successfully parses out the statistical indicators as "sales" and "sales volume", and the dimensions are "the last three months (time dimension)" and "different regions" and "various categories of goods (category dimension)". Then, based on these analysis results and its own industry knowledge base and chart display experience, the big model recommends using a bar chart to display the sales of various categories of goods in different regions. For example, a line chart can be used to show the trend of sales volume changes in the last three months, because a bar chart is easy to compare the numerical differences of different categories in different regions, and a line chart can clearly show the changes in sales volume over time.

[0036] 4. According to the statistical indicators and dimensions parsed, when the statistical interface matching the text word is found in the indicator knowledge base, the API interface of the company's data analysis service is called. After receiving the request, the API interface performs data query tasks in the indicator knowledge base, screens out the sales and sales volume data of various categories of goods in different regions in the past three months from relevant data tables such as sales record tables, and generates data sets in a specific format; or, when the statistical interface matching the text word cannot be found in the indicator knowledge base, the text is used as a prompt word, and the query interface of the data indicator is retrieved from the knowledge base based on the retrieval enhancement generation technology of the large model LLM, and the data set is assisted in generating; wherein, the knowledge base includes the industry knowledge base and the indicator knowledge base, the industry knowledge base contains the statistical annual report of the specified industry, and the indicator knowledge base is constructed based on the statistical interface request path, statistical indicators and interface description information. This data set and the chart display method recommended by the large model are transmitted to the front-end page of the intelligent cockpit together. The front-end page draws the corresponding bar chart and line chart on the page according to the received data set and display method, and intuitively presents the data to the manager. For example, the different colored columns in the bar chart represent different categories of goods, and the height of the columns corresponds to the sales values ​​in different regions; the line chart uses time as the horizontal axis and sales volume as the vertical axis, clearly showing the fluctuations in sales volume in the past three months.

[0037] 5. Organize the generated analysis results into easy-to-understand text, such as: "In the past three months, the sales of electronic products in the eastern region were the highest, reaching [X] million yuan; the sales volume of clothing products showed a steady upward trend in the past three months, from [X] pieces in the first month to [X] pieces in the third month..." Then, the digital human broadcasts these analysis results through voice. The digital human can simulate natural and fluent voice, with appropriate expressions and body movements, to increase the vividness and fun of information transmission, allowing managers to obtain key information more easily.

[0038] Embodiment 2: Refer to the attached Figure 2 This embodiment proposes an intelligent cockpit interaction system based on multimodal artificial intelligence, which is characterized by comprising: Audio acquisition module, users monitor the microphone through the browser side and obtain audio data stream segments; The speech recognition module is used to first load the pre-trained speech command model based on TensorFlow.js, perform speech command recognition on the audio data stream segment, and then call the audio-to-text service to convert the recognized audio data stream into text; The text processing module is used to parse text using the large model LLM, call the API interface of the data analysis service, generate a data set corresponding to the parsing result, and recommend a data presentation method based on the data structure of the data set; The chart generation module is used to generate different types of statistical charts using the large model LLM according to the data set and the recommended data presentation method, and at the same time, generate the interpretation text of the statistical charts; The video generation module is used to obtain the interpretation text generated by the large model LLM using the digital human video generation service, and use the preset algorithm to convert the interpretation text into a video containing the digital human image and having the information broadcasting function; The transmission and display module is used to send the data set generated by the large model LLM and the video converted by the digital human video generation service to the front-end interface of the intelligent cockpit for display and broadcast.

[0039] In this embodiment, the text processing module involved uses the large model LLM to parse the text, calls the API interface of the data analysis service, and generates a data set corresponding to the parsing result. This process specifically includes: 1) Leverage the natural language understanding capabilities of the large model LLM to extract user needs or query requests from the text; 2) Call the API interface of the data analysis service and search for the statistical interface that matches the text in the indicator knowledge base. (a) If a statistical interface matching the text is found in the indicator knowledge base, the request path and required interface parameters of the statistical interface are obtained, and the subsequent steps are continued; (b) If no statistical interface matching the text word is found in the indicator knowledge base, the text is used as a prompt word, and based on the retrieval enhancement generation technology of the large model LLM, the query interface of the data indicator is retrieved from the knowledge base, and the data set is assisted in generating; wherein the knowledge base includes an industry knowledge base and an indicator knowledge base, the industry knowledge base contains the statistical annual report of the specified industry, and the indicator knowledge base is constructed based on the statistical interface request path, statistical indicators and interface description information; 3) Use the large model LLM to parse the text, extract the specific value corresponding to the interface parameter, use the specific value to call the request path, and obtain statistical data; 4) Use the large model LLM to further analyze and process the statistical data to generate a data set.

[0040] The dataset generated by the process of step 2) (b) and step 4) contains the following fields: Figure title; Data labels; Presentation methods, including bar charts, line charts, and pie charts; A dataset containing actual data values, used to generate the chart; The interpretation text of the dataset is used to provide a detailed interpretation of the dataset.

[0041] In summary, it can be seen that the intelligent cockpit interaction method and system based on multimodal artificial intelligence of the present invention can realize multimodal interaction through technologies such as voice command recognition, voice-to-text, large model LLM analysis, multidimensional data analysis, and digital human broadcasting.

[0042] The above specific examples are used to explain the principles and implementation methods of the present invention in detail. These embodiments are only used to help understand the core technical content of the present invention. Based on the above specific embodiments of the present invention, any improvements and modifications made to the present invention by those skilled in the art without departing from the principles of the present invention shall fall within the scope of protection of the present invention.

Claims

1. An intelligent cockpit interaction method based on multimodal artificial intelligence, characterized in that: The steps include: S1. Obtain user voice data, implement voice command recognition based on TensorFlow.js, and convert the recognized user voice into text; S2. Use the large model LLM to parse the text, call the API interface of the data analysis service, generate a data set corresponding to the analysis result, and recommend the data presentation method based on the data structure of the data set; S3, the large model LLM generates different types of statistical charts based on the data set and the recommended data presentation method, and at the same time, generates interpretation texts of the statistical charts; S4. Digital Human Video Generation Service obtains the interpretation text generated by the large model LLM, and uses the preset algorithm to convert the interpretation text into a video containing the digital human image and with information broadcasting function; S5. Send the data set generated by the large model LLM and the video converted by the digital human video generation service to the front-end interface of the intelligent cockpit for display and broadcast.

2. The intelligent cockpit interaction method based on multimodal artificial intelligence according to claim 1 is characterized in that: The step S1 specifically includes: S1.

1. Obtain audio data stream segments by monitoring the microphone on the browser side; S1.

2. Load the pre-trained voice command model based on TensorFlow.js and perform voice command recognition on the audio data stream segment; S1.

3. After completing speech recognition through the voice command model, call the audio-to-text service to convert the recognized audio data stream into text.

3. The intelligent cockpit interaction method based on multimodal artificial intelligence according to claim 1 is characterized in that: The step S2 specifically includes: S2.

1. Use the natural language understanding capability of the large model LLM to extract user needs or query requests from the text; S2.

2. Call the API interface of the data analysis service and search for the statistical interface matching the text in the indicator knowledge base. (a) If a statistical interface matching the text is found in the indicator knowledge base, the request path and required interface parameters of the statistical interface are obtained, and step S2.3 is executed; (b) If no statistical interface matching the text word is found in the indicator knowledge base, the text is used as a prompt word, and the data in the industry knowledge base is used to generate a data set based on the retrieval enhancement generation technology of the large model LLM; S2.

3. parse the text using the large model LLM, extract the specific value corresponding to the interface parameter, use the specific value to call the request path, and obtain statistical data; S2.

4. Use the large model LLM to further analyze and process the statistical data to generate a data set.

4. The intelligent cockpit interaction method based on multimodal artificial intelligence according to claim 3 is characterized in that: When executing step (b), the large model LLM uses retrieval enhancement generation technology to retrieve the query interface of data indicators from the knowledge base and assist in generating a data set; wherein the knowledge base includes an industry knowledge base and an indicator knowledge base, the industry knowledge base contains the statistical annual report of the specified industry, and the indicator knowledge base is constructed based on the statistical interface request path, statistical indicators and interface description information.

5. The intelligent cockpit interaction method based on multimodal artificial intelligence according to claim 3 is characterized in that: Execute step S2, the data set includes the following fields: Figure title; Data labels; Presentation methods, including bar charts, line charts, and pie charts; A dataset containing actual data values, used to generate the chart; The interpretation text of the dataset is used to provide a detailed interpretation of the dataset.

6. An intelligent cockpit interaction system based on multimodal artificial intelligence, characterized in that: It includes: Audio collection module, users obtain user voice data; The speech recognition module is used to implement voice command recognition based on TensorFlow.js and convert the recognized user speech into text; The text processing module is used to parse text using the large model LLM, call the API interface of the data analysis service, generate a data set corresponding to the parsing result, and recommend a data presentation method based on the data structure of the data set; The chart generation module is used to generate different types of statistical charts using the large model LLM according to the data set and the recommended data presentation method, and at the same time, generate the interpretation text of the statistical charts; The video generation module is used to obtain the interpretation text generated by the large model LLM using the digital human video generation service, and use the preset algorithm to convert the interpretation text into a video containing the digital human image and having the information broadcasting function; The transmission and display module is used to send the data set generated by the large model LLM and the video converted by the digital human video generation service to the front-end interface of the intelligent cockpit for display and broadcast.

7. The intelligent cockpit interaction system based on multimodal artificial intelligence according to claim 6 is characterized in that: The audio acquisition module obtains audio data stream segments by monitoring the microphone on the browser side; The speech recognition module first loads the pre-trained speech command model based on TensorFlow.js, performs speech command recognition on the audio data stream segment, and then calls the audio-to-text service to convert the recognized audio data stream into text.

8. The intelligent cockpit interaction system based on multimodal artificial intelligence according to claim 6 is characterized in that: The specific execution process of the text processing module includes: Leverage the natural language understanding capabilities of the large model LLM to extract user needs or query requests from the text; Call the API interface of the data analysis service and search for the statistical interface matching the text in the indicator knowledge base. (a) If a statistical interface matching the text is found in the indicator knowledge base, the request path and required interface parameters of the statistical interface are obtained, and the subsequent steps are continued; (b) If no statistical interface matching the text word is found in the indicator knowledge base, the text is used as a prompt word, and the data in the industry knowledge base is used to generate a data set based on the retrieval enhancement generation technology of the large model LLM; Use the large model LLM to parse the text, extract the specific value corresponding to the interface parameter, use the specific value to call the request path, and obtain statistical data; The statistical data are further analyzed and processed using the large model LLM to generate a data set.

9. The intelligent cockpit interaction system based on multimodal artificial intelligence according to claim 8 is characterized in that: When executing step (b), the large model LLM uses retrieval enhancement generation technology to retrieve the query interface of data indicators from the knowledge base and assist in generating a data set; wherein the knowledge base includes an industry knowledge base and an indicator knowledge base, the industry knowledge base contains the statistical annual report of the specified industry, and the indicator knowledge base is constructed based on the statistical interface request path, statistical indicators and interface description information.

10. The intelligent cockpit interaction system based on multimodal artificial intelligence according to claim 8, characterized in that: The dataset contains the following fields: Figure title; Data labels; Presentation methods, including bar charts, line charts, and pie charts; A dataset containing actual data values, used to generate the chart; The interpretation text of the dataset is used to provide a detailed interpretation of the dataset.

Images