Intelligent question number large-screen display system and method based on multi-modal large model

Abstract

The invention discloses an intelligent number-asking large-screen display system and method based on a multi-modal large model, belongs to the technical field of artificial intelligence and big data processing, and aims to solve the technical problem of how to flexibly adapt to diversified application scenes, perform optimization adjustment according to specific scene requirements and improve applicability and pertinence. According to the technical scheme, a multi-mode input processing module is used for receiving input of a user and conducting semantic conversion by means of the natural language processing technology; the multi-modal large model analysis engine module is used for performing analysis and aggregation by means of a trained multi-modal and cross-modal deep learning model; the data processing and integrating module is used for processing and integrating data from different sources in real time according to the output requirements of the analysis engine; and the large-screen visualization display module is used for visually displaying the processed data on a large screen in the forms of charts, thermodynamic diagrams and 3D models by utilizing an advanced data visualization technology and interactive design.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence and big data processing technology, and in particular to an intelligent large-screen display system and method based on a multimodal large model. Background Art

[0002] With the influx of big data, we have ushered in an era of unprecedentedly rich but also extremely complex information. The geometric growth of massive data is not only astonishing in scale, but also poses unprecedented challenges to existing data processing capabilities and display methods in terms of diversity and depth. Although traditional data large-screen display systems meet basic data visualization needs to a certain extent, most of them remain at the level of "displaying" rather than "narrating" data, lacking dynamic interactivity and revealing the deep connotation of data, which makes them unable to cope with increasingly complex data analysis tasks.

[0003] Especially in the vast ocean of big data, data sources are diverse (such as social media, sensor networks, public databases, etc.), and the types are complex (structured, semi-structured and even unstructured data coexist). A single analysis perspective and display mode can no longer fully capture the value and significance of data. For example, relying only on charts and numbers to reflect the emotional tendencies of social media may miss a lot of emotional fluctuations hidden in the details of speech; only showing the increase or decrease of sales data without combining market trends, customer feedback image and voice information analysis may miss the key signals of market changes.

[0004] Therefore, how to flexibly adapt to diverse application scenarios, optimize and adjust according to specific scenario requirements, and improve applicability and pertinence is a technical problem that needs to be solved urgently. Summary of the invention

[0005] The technical task of the present invention is to provide an intelligent large-screen display system and method based on a multimodal large model to solve the problem of how to flexibly adapt to diversified application scenarios, and optimize and adjust according to specific scenario requirements to improve applicability and pertinence.

[0006] The technical task of the present invention is achieved in the following manner: an intelligent large-screen display system for questioning numbers based on a multimodal large model, the system comprising:

[0007] A multimodal input processing module, which is used to receive user input and perform semantic conversion with the help of natural language processing technology;

[0008] Multimodal large model analysis engine module, used for analysis and aggregation with the help of trained multimodal and cross-modal deep learning models;

[0009] The data processing and integration module is used to process and integrate data from different sources in real time according to the output requirements of the analysis engine, ensure the accuracy and timeliness of the data, and optimize the data structure to adapt to the large-screen display requirements;

[0010] The large-screen visualization display module is used to use advanced data visualization technology and interactive design to intuitively display the processed data on the large screen in the form of charts, heat maps and 3D models. It supports dynamic updates and user-customized views to enhance data insights.

[0011] Preferably, the multimodal input processing module is also used for information bridging between the user and the system, integrating multiple perception and understanding technologies to ensure comprehensive and accurate capture of user intent;

[0012] The multimodal input processing module includes:

[0013] The text instruction parsing and semantic understanding submodule is used to receive text input using natural language processing (NLP) technology, deeply analyze the multi-dimensional features of syntactic structure, emotional tendency and keyword extraction, and convert the text content into structured data that can be understood by machines, so as to perform complex instruction parsing and semantic understanding;

[0014] The speech recognition submodule is used to achieve high accuracy and fast response capabilities through the speech recognition (ASR) engine when processing speech information; and through complex audio signal processing, acoustic model matching and language model analysis, combined with model training and tuning, it can achieve the reception and processing of natural languages ​​in different dialects, different speech speeds, and different background noises, ensuring the fluency and naturalness of communication;

[0015] The image recognition submodule is used to realize the input of pictures and other visuals with the help of computer vision (CV) algorithms and technologies: object recognition, scene understanding, face recognition and optical character recognition (OCR) are used to extract rich information from images; the information extracted from the image includes recognizing the text content in the image, analyzing the image subject and background, and even understanding the emotions and meanings conveyed by the image, and then converting them into unified and meaningful semantic expressions;

[0016] The integration sub-module is used to integrate the diverse inputs of text, voice and images into a unified high-level semantic model after their respective processing flows, so as to form recognition of user intent and ensure that no matter how the user communicates, the intent can be captured accurately, providing a solid foundation for subsequent intelligent decision-making and services.

[0017] Preferably, the multimodal large model analysis engine module adopts a pre-trained cross-modal deep learning model, which performs cross-domain semantic understanding and generation tasks by virtue of its powerful parallel processing capability and effective capture of long-distance dependencies;

[0018] The multimodal large model analysis engine module includes:

[0019] The deep semantic analysis and intent recognition submodule is used to use multimodal and cross-modal deep learning models to deeply analyze unified semantic representations, accurately identify and process complex query logic, specifically: compound queries that contain "and", "or", and "not" Boolean logical relationships, and fine-grained screening of multiple conditions of time series and spatial locations; at the same time, it is also used to track conversation history and user preferences with the help of context-aware mechanisms, understand the implicit intent behind requests, and thus provide more personalized and context-adaptive information services;

[0020] The sub-module for accurate and rapid screening and aggregation of massive and heterogeneous data is used to optimize information screening and aggregation strategies, efficiently cope with information retrieval challenges in a big data environment, and use high-level semantic features learned by the model to quickly locate relevant data fragments, achieving accurate matching even in massive and heterogeneous data sets. At the same time, it is also used to automatically summarize and summarize information through intelligent aggregation algorithms, extract key points, and provide users with structured and easy-to-digest feedback, greatly improving the efficiency of information processing and the user's experience in obtaining information.

[0021] Preferably, the data processing and integration module constructs an efficient and flexible data processing and integration engine, closely connects the output results of the analysis engine, and responds to each data request triggered by the user's complex needs in real time;

[0022] Data processing and integration modules include:

[0023] The data fusion submodule is used to seamlessly connect and extract data across multiple heterogeneous data sources, while performing data cleaning and verification to effectively eliminate redundant information and correct inconsistencies; heterogeneous data sources include databases, API interfaces, and cloud storage services;

[0024] The real-time data stream processing submodule is used to build in real-time data stream processing technology, and uses the streaming computing framework to perform low-latency processing and analysis on continuously updated data streams; the streaming computing framework includes Apache Kafka and Flink;

[0025] The data optimization and reconstruction logic submodule is used to perform intelligent structural optimization on the integrated data and use efficient data compression algorithms to reduce the transmission burden while retaining data details, so that large amounts of data can be smoothly loaded and displayed on the large screen; among them, the intelligent structural optimization adopts a layered data model to improve the hierarchy and readability of information display;

[0026] The adaptive layout submodule is used to use an adaptive layout algorithm to dynamically adjust the data view layout according to the actual size and resolution of the large screen, so that charts, tables and map information can obtain the best display effect on large screens of different sizes.

[0027] Preferably, the large-screen visual display module includes:

[0028] The visualization library integration submodule is used to integrate the top visualization libraries of D3.js, ECharts, and Three.js. It supports classic chart forms such as bar charts, line charts, and pie charts. It also implements advanced chart types such as heat maps, scatter plot matrices, and Sankey diagrams, as well as 3D models and animation effects, to reveal the trends, patterns, and correlations behind the data in multiple dimensions and levels.

[0029] The real-time data processing submodule is used to automatically refresh the visual content on the large screen as the data changes based on the real-time data processing engine. At the same time, SadTalker is used to empower the IP image formed by the modeling, effectively introduce the recognized content, and achieve a better interactive experience.

[0030] The view options customization module allows users to easily configure the display content according to their own focus, such as selecting a specific time range, data dimension, comparison indicator or perspective; it also supports drag-and-drop operations, allowing users to freely adjust the chart layout, or generate personalized data dashboards through simple settings.

[0031] A method for displaying intelligent data on a large screen based on a multimodal large model, the method is as follows:

[0032] Receive user input and perform semantic conversion with the help of natural language processing technology;

[0033] Analysis and aggregation with the help of trained multi-modal and cross-modal deep learning models;

[0034] According to the requirements of the analysis engine output, data from different sources are processed and integrated in real time to ensure the accuracy and timeliness of the data, and the data structure is optimized to adapt to the large-screen display requirements;

[0035] Utilizing advanced data visualization technology and interactive design, the processed data is intuitively displayed on the large screen in the form of charts, heat maps and 3D models, supporting dynamic updates and user-customized views to enhance data insights.

[0036] Preferably, receiving the user's input and performing semantic conversion with the help of natural language processing technology is as follows:

[0037] It uses natural language processing (NLP) technology to receive text input, deeply analyzes the multi-dimensional features of syntactic structure, emotional tendency and keyword extraction, and converts text content into machine-understandable structured data, so as to perform complex instruction parsing and semantic understanding;

[0038] When processing voice information, the voice recognition (ASR) engine achieves high accuracy and fast response capabilities; and through complex audio signal processing, acoustic model matching and language model analysis, combined with model training and tuning, it can achieve the reception and processing of natural languages ​​in different dialects, different speech speeds, and different background noises, ensuring the fluency and naturalness of communication;

[0039] Using computer vision (CV) algorithms and technologies to realize image and other visual input: object recognition, scene understanding, face recognition and optical character recognition (OCR) are used to extract rich information from images; the information extracted from images includes identifying the text content in the image, analyzing the image subject and background, and even understanding the emotions and meanings conveyed by the image, and then converting them into unified and meaningful semantic expressions;

[0040] After various inputs such as text, voice and images have gone through their respective processing flows, they are integrated into a unified high-level semantic model to form recognition of user intent. This ensures that no matter how users communicate, intent can be captured accurately, providing a solid foundation for subsequent intelligent decision-making and services.

[0041] The analysis and aggregation with the help of trained multimodal and cross-modal deep learning models are as follows:

[0042] Utilize multimodal and cross-modal deep learning models to deeply analyze unified semantic representations, accurately identify and process complex query logic, specifically: compound queries that simultaneously contain "and", "or", and "not" Boolean logical relationships, as well as fine screening of multiple conditions of time series and spatial locations; at the same time, it is also used to track conversation history and user preferences with the help of context-aware mechanisms, understand the implicit intentions behind requests, and thus provide more personalized and context-adaptive information services;

[0043] Optimize information screening and aggregation strategies, efficiently cope with information retrieval challenges in big data environments, and use high-level semantic features learned by the model to quickly locate relevant data fragments, achieving accurate matching even in massive, heterogeneous data sets; at the same time, it is also used to automatically summarize and summarize information through intelligent aggregation algorithms, extract key points, and provide users with structured and easy-to-digest feedback, greatly improving the efficiency of information processing and the user experience of obtaining information.

[0044] As a priority, according to the requirements of the analysis engine output, data from different sources are processed and integrated in real time to ensure the accuracy and timeliness of the data, and the data structure is optimized to adapt to the large-screen display requirements as follows:

[0045] Seamlessly connect and extract data across multiple heterogeneous data sources, while performing data cleaning and verification to effectively eliminate redundant information and correct inconsistencies; heterogeneous data sources include databases, API interfaces, and cloud storage services;

[0046] Adopt real-time data stream processing technology and use streaming computing frameworks to perform low-latency processing and analysis on continuously updated data streams; streaming computing frameworks include Apache Kafka and Flink;

[0047] Intelligent structural optimization is performed on the integrated data and efficient data compression algorithms are used to reduce the transmission burden while retaining data details, so that large amounts of data can be smoothly loaded and displayed on the large screen. Among them, intelligent structural optimization uses a layered data model to improve the hierarchy and readability of information display;

[0048] Adopting adaptive layout algorithm, the data view layout is dynamically adjusted according to the actual size and resolution of the large screen, so that charts, tables and map information can obtain the best display effect on large screens of different sizes;

[0049] Using advanced data visualization technology and interactive design, the processed data is intuitively displayed on the large screen in the form of charts, heat maps and 3D models, supporting dynamic updates and user-customized views to enhance data insights as follows:

[0050] Integrate D3.js, ECharts, Three.js top visualization libraries, support classic chart forms such as bar charts, line charts, and pie charts, and realize advanced chart types such as heat maps, scatter plot matrices, and Sankey diagrams, as well as 3D models and animation effects, to reveal the trends, patterns, and correlations behind the data in multiple dimensions and levels;

[0051] Relying on the real-time data processing engine, the visual content on the large screen is automatically refreshed as the data changes. At the same time, SadTalker is used to empower the IP image formed by the modeling, effectively introduce the recognized content, and achieve a better interactive experience;

[0052] Users can easily configure the display content according to their own focus, such as selecting a specific time range, data dimension, comparison indicator or perspective; it also supports drag-and-drop operations, allowing users to freely adjust the chart layout, or generate personalized data dashboards through simple settings.

[0053] An electronic device comprising: a memory and at least one processor;

[0054] Wherein, the memory stores computer-executable instructions;

[0055] The at least one processor executes the computer-executable instructions stored in the memory, so that the at least one processor executes the above-mentioned intelligent question-and-answer large-screen display method based on the multimodal large model.

[0056] A computer-readable storage medium stores computer-executable instructions. When a processor executes the computer-executable instructions, the above-mentioned intelligent large-screen display method for questioning and answering based on a multimodal large model is implemented.

[0057] The intelligent large-screen display system and method based on a multimodal large model of the present invention has the following advantages:

[0058] (I) The present invention is deeply rooted in the forefront of artificial intelligence, big data processing and visualization technology, and has innovatively built a highly integrated intelligent large-screen display system for question and answer, deeply integrating natural language processing (NLP) technology, speech recognition, speech recognition (ASR) engine and multimodal large model, realizing user intention recognition and data retrieval and aggregation, relying on dynamic visualization technology to realize the visualization and presentation of user data, accurately identifying user intentions, effectively responding to user questions, and improving the interactive experience, so as to better present data to customers;

[0059] (ii) The present invention can autonomously learn and adapt to the ever-changing operation and maintenance scenarios, and utilize the powerful semantic understanding, intent recognition and pattern recognition capabilities of the multimodal large model to effectively receive the user's text, voice, picture and other multi-mode inputs, effectively deal with complex and changing problem expressions, effectively respond to user needs, and continuously evolve itself, so that the accuracy and personalization level of the service are increasingly improved;

[0060] (III) The dynamic visualization technology used in the present invention can adaptively adjust the display interface according to the user's query needs and data analysis results, and vividly display the stories and trends behind the data in various forms such as charts, three-dimensional models, and dynamic video storyboards. This highly interactive design allows users to intuitively feel the flow and changes of data while enjoying immersive data exploration, thus promoting the collision of sparks of inspiration in the decision-making process;

[0061] (IV) The present invention is also equipped with natural language processing technology. Users can freely raise complex query requirements through voice or text input, and the system will respond immediately, providing instant feedback in a way that is closest to human communication habits, greatly improving the fluency and convenience of user experience. Whether it is used in the fields of corporate strategic planning, market research, or scientific research, the present invention can significantly improve the efficiency and accuracy of data analysis and promote the intelligent transformation of the decision-making process;

[0062] (V) The present invention constructs an intelligent large-screen display system that can overcome data modality barriers and integrate advanced artificial intelligence algorithms, big data processing technology and innovative visualization methods, which is particularly important and urgent; such a system should have three core characteristics: First, it needs to be a master of multimodal information processing, able to integrate the delicacy of text, the emotion of voice, the intuitiveness of images, and more dimensional data to achieve a comprehensive analysis of the complex data ecosystem; second, it must have strong interactive capabilities, allowing users to ask questions in natural language and have in-depth conversations with the system through voice or text, so as to obtain customized data insights; finally, the system must also be good at telling data stories, using dynamic and contextualized visualization technology to vividly display the logic, trends and potential stories behind the data, so that decision makers can quickly capture key information and gain insight into the deep meaning behind the data in a rich interactive experience, providing strong support for scientific decision-making;

[0063] (VI) The invention develops a smart large-screen display system for questioning numbers, which is not only a technological innovation, but also a key step in promoting the transformation of data from a silent pile of numbers into a vivid, interactive and dynamic information treasure house. It is of great significance to improve the decision-making efficiency and quality of organizations and promote the intelligent transformation of various industries;

[0064] (VII) The present invention focuses on building an unprecedented intelligent data query and large-screen display system. It is not only a data exploration tool that integrates technological advancement and user experience, but also a landmark innovation in the era of big data, entering a new era of efficient and intelligent decision-making. It aims to completely subvert the traditional data processing and display mode, and provide users with unprecedented data query and analysis experience through deep integration of multimodal interaction technology, large model algorithms and large-scale data processing capabilities;

[0065] (VIII) The present invention cleverly integrates text analysis, voice recognition, and even the latest human motion recognition technology to ensure that the system can fully cover and accurately understand various query instructions of users, whether these instructions are transmitted through typed text, verbal commands, or even intuitive gestures. It greatly broadens the communication channel between users and data and lowers the threshold for use, allowing both technical experts and business personnel to easily control and freely explore the mysteries of data;

[0066] (IX) The present invention introduces advanced multimodal fusion algorithms combined with the latest large-scale pre-trained models. These models are deeply learned based on extensive data on the Internet and have powerful contextual understanding and cross-domain knowledge reasoning capabilities. When a user makes a query, the system starts immediately, like an experienced data detective, shuttling through the vast ocean of data to quickly and accurately extract relevant data information. The present invention uses cutting-edge visualization technology to transform complex data into intuitive and easy-to-understand charts, animations, and even three-dimensional interactive scenes, vividly interpreting the stories and trends behind the data on a large display screen, making data insights clear at a glance.

[0067] (10) The present invention supports instant feedback and dynamic adjustment during the display process. Users can modify query conditions in real time based on the preliminary display results, such as adjusting the time range, the data dimension of interest, or filtering specific information. The system can respond in real time and quickly update the display content. This high degree of interactivity not only greatly improves the flexibility of data exploration, but also makes decision support more agile and accurate.

[0068] (XI) The present invention perfectly integrates cutting-edge artificial intelligence technology with intuitive interactive design to create an intelligent large-screen display system that can adapt to diverse needs and promote deep data interaction and intelligent decision-making. It not only greatly pushes the boundaries of big data analysis and application, but also lays a solid technical foundation for opening a new era of intelligent decision-making in the future;

[0069] (XII) The present invention is committed to creating a barrier-free and intuitive operating environment for users through innovative multimodal interaction design. It not only supports traditional keyboard input, but also integrates natural interaction methods such as voice command recognition, gesture control and facial expression analysis, so that users of different ages and skill levels can easily get started, significantly lowering the threshold for use; during the query process, the addition of intelligent prompts and automatic completion functions makes information retrieval faster, reduces user operation steps, and improves the efficiency and convenience of queries; at the same time, the learning mechanism based on user behavior continuously optimizes the interaction process, so that each use is more in line with personal habits, enhancing the overall user experience satisfaction;

[0070] (XII) The present invention adopts a comprehensive data integration and analysis strategy, deeply integrating multi-source data such as detailed descriptions of text, emotional colors of voice, and intuitive information of images, to construct a multi-dimensional data observation perspective. The comprehensive processing method transcends the limitations of a single data type, revealing the relationships, trends, and anomalies hidden behind the data for decision makers, and promoting deeper data understanding and insight discovery. Through complex machine learning models, the system can automatically generate summary reports, highlight key information and potential impacts, and bring users an intuitive data insight experience;

[0071] (XIII) The instant response technology integrated in the present invention ensures real-time feedback of query results. Users can see the changes in data visualization at the moment of asking questions. This dynamic interaction greatly shortens the decision-making waiting time. At the same time, through the intelligent early warning system and key indicator tracking function, users can be notified of important events or abnormal situations of concern in the first time, so that decision makers can quickly focus on core issues and make fast and accurate judgments based on real-time updated data. Combined with flexible data slicing and drilling capabilities, it helps users analyze layer by layer, dig deep into data details, and assist in making more intelligent data-driven decisions.

[0072] (XIV) The highly modular and configurable features of the present invention enable it to be flexibly adapted to a variety of application scenarios, from efficient management of smart cities and real-time traffic flow control to internal enterprise operational data analysis and market trend forecasting, as well as risk assessment and fraud monitoring in the financial industry, etc., all of which can provide customized solutions; each module can be optimized and adjusted according to the needs of specific scenarios, ensuring the applicability and pertinence of the system, demonstrating strong flexibility and universal value, and truly realizing cross-industry and cross-field data empowerment;

[0073] (XV) The multimodal large model analysis engine module of the present invention adopts a pre-trained cross-modal deep learning model. With its powerful parallel processing capability and effective capture of long-distance dependencies, it can achieve deep fusion of data from multiple modalities such as text, voice, and image. It not only achieves deep analysis within a single modality, but also promotes semantic association and complementarity between modalities, and constructs a more comprehensive and in-depth user intent analysis and data aggregation. The multimodal large model analysis engine module, with its multimodal and cross-modal deep learning capabilities, can not only deeply understand complex and multi-dimensional user requests, but also navigate agilely in the ocean of big data, providing users with information services that are both accurate and insightful. BRIEF DESCRIPTION OF THE DRAWINGS

[0074] The present invention is further described below in conjunction with the accompanying drawings.

[0075] Attached Figure 1This is a structural diagram of the intelligent large-screen display system based on a multimodal large model. DETAILED DESCRIPTION

[0076] The intelligent large-screen display system and method for multimodal large models of the present invention are described in detail below with reference to the accompanying drawings and specific embodiments of the specification.

[0077] Embodiment 1:

[0078] As attached Figure 1 As shown, this embodiment provides an intelligent large-screen display system for questioning numbers based on a multimodal large model, the system comprising:

[0079] A multimodal input processing module, which is used to receive user input and perform semantic conversion with the help of natural language processing technology;

[0080] Multimodal large model analysis engine module, used for analysis and aggregation with the help of trained multimodal and cross-modal deep learning models;

[0081] The data processing and integration module is used to process and integrate data from different sources in real time according to the output requirements of the analysis engine, ensure the accuracy and timeliness of the data, and optimize the data structure to adapt to the large-screen display requirements;

[0082] The large-screen visualization display module is used to use advanced data visualization technology and interactive design to intuitively display the processed data on the large screen in the form of charts, heat maps and 3D models. It supports dynamic updates and user-customized views to enhance data insights.

[0083] The multimodal input processing module in this embodiment is also used to bridge information between the user and the system, integrating multiple perception and understanding technologies to ensure comprehensive and accurate capture of user intent;

[0084] The multimodal input processing module includes:

[0085] The text instruction parsing and semantic understanding submodule is used to receive text input using natural language processing (NLP) technology, deeply analyze the multi-dimensional features of syntactic structure, emotional tendency and keyword extraction, and convert the text content into structured data that can be understood by machines, so as to perform complex instruction parsing and semantic understanding;

[0086] The speech recognition submodule is used to achieve high accuracy and fast response capabilities through the speech recognition (ASR) engine when processing speech information; and through complex audio signal processing, acoustic model matching and language model analysis, combined with model training and tuning, it can achieve the reception and processing of natural languages ​​in different dialects, different speech speeds, and different background noises, ensuring the fluency and naturalness of communication;

[0087] The image recognition submodule is used to realize the input of pictures and other visuals with the help of computer vision (CV) algorithms and technologies: object recognition, scene understanding, face recognition and optical character recognition (OCR) are used to extract rich information from images; the information extracted from the image includes recognizing the text content in the image, analyzing the image subject and background, and even understanding the emotions and meanings conveyed by the image, and then converting them into unified and meaningful semantic expressions;

[0088] The integration sub-module is used to integrate the diverse inputs of text, voice and images into a unified high-level semantic model after their respective processing flows, so as to form recognition of user intent and ensure that no matter how the user communicates, the intent can be captured accurately, providing a solid foundation for subsequent intelligent decision-making and services.

[0089] The multimodal large model analysis engine module in this embodiment uses a pre-trained cross-modal deep learning model, which performs cross-domain semantic understanding and generation tasks with its powerful parallel processing capabilities and effective capture of long-distance dependencies;

[0090] The multimodal large model analysis engine module includes:

[0091] The deep semantic analysis and intent recognition submodule is used to use multimodal and cross-modal deep learning models to deeply analyze unified semantic representations, accurately identify and process complex query logic, specifically: compound queries that contain "and", "or", and "not" Boolean logical relationships, and fine-grained screening of multiple conditions of time series and spatial locations; at the same time, it is also used to track conversation history and user preferences with the help of context-aware mechanisms, understand the implicit intent behind requests, and thus provide more personalized and context-adaptive information services;

[0092] The sub-module for accurate and rapid screening and aggregation of massive and heterogeneous data is used to optimize information screening and aggregation strategies, efficiently cope with information retrieval challenges in a big data environment, and use high-level semantic features learned by the model to quickly locate relevant data fragments, achieving accurate matching even in massive and heterogeneous data sets. At the same time, it is also used to automatically summarize and summarize information through intelligent aggregation algorithms, extract key points, and provide users with structured and easy-to-digest feedback, greatly improving the efficiency of information processing and the user's experience in obtaining information.

[0093] The data processing and integration module in this embodiment builds an efficient and flexible data processing and integration engine, closely connects the output results of the analysis engine, and responds to every data request triggered by the user's complex needs in real time;

[0094] Data processing and integration modules include:

[0095] The data fusion submodule is used to seamlessly connect and extract data across multiple heterogeneous data sources, while performing data cleaning and verification to effectively eliminate redundant information and correct inconsistencies; heterogeneous data sources include databases, API interfaces, and cloud storage services;

[0096] The real-time data stream processing submodule is used to build in real-time data stream processing technology, and uses the streaming computing framework to perform low-latency processing and analysis on continuously updated data streams; the streaming computing framework includes Apache Kafka and Flink;

[0097] The data optimization and reconstruction logic submodule is used to perform intelligent structural optimization on the integrated data and use efficient data compression algorithms to reduce the transmission burden while retaining data details, so that large amounts of data can be smoothly loaded and displayed on the large screen; among them, the intelligent structural optimization adopts a layered data model to improve the hierarchy and readability of information display;

[0098] The adaptive layout submodule is used to use an adaptive layout algorithm to dynamically adjust the data view layout according to the actual size and resolution of the large screen, so that charts, tables and map information can obtain the best display effect on large screens of different sizes.

[0099] The large-screen visual display module in this embodiment includes:

[0100] The visualization library integration submodule is used to integrate the top visualization libraries of D3.js, ECharts, and Three.js. It supports classic chart forms such as bar charts, line charts, and pie charts. It also implements advanced chart types such as heat maps, scatter plot matrices, and Sankey diagrams, as well as 3D models and animation effects, to reveal the trends, patterns, and correlations behind the data in multiple dimensions and levels.

[0101] The real-time data processing submodule is used to automatically refresh the visual content on the large screen as the data changes based on the real-time data processing engine. At the same time, SadTalker is used to empower the IP image formed by the modeling, effectively introduce the recognized content, and achieve a better interactive experience.

[0102] The view options customization module allows users to easily configure the display content according to their own focus, such as selecting a specific time range, data dimension, comparison indicator or perspective; it also supports drag-and-drop operations, allowing users to freely adjust the chart layout, or generate personalized data dashboards through simple settings.

[0103] Embodiment 2:

[0104] This embodiment provides a method for displaying intelligent data on a large screen based on a multimodal large model, and the method is specifically as follows:

[0105] S1, receiving user input and performing semantic conversion with the help of natural language processing technology;

[0106] S2, analysis and aggregation with the help of trained multimodal and cross-modal deep learning models;

[0107] S3. According to the output requirements of the analysis engine, real-time processing and integration of data from different sources to ensure the accuracy and timeliness of the data, and optimize the data structure to adapt to the large-screen display requirements;

[0108] S4. Using advanced data visualization technology and interactive design, the processed data is intuitively displayed on the large screen in the form of charts, heat maps and 3D models, supporting dynamic updates and user-customized views to enhance data insights.

[0109] In step S1 of this embodiment, receiving the user's input and performing semantic conversion by means of natural language processing technology are specifically as follows:

[0110] S101, using natural language processing (NLP) technology to receive text input, deeply analyze the multi-dimensional features of syntactic structure, emotional tendency and keyword extraction, and convert the text content into machine-understandable structured data, so as to perform complex instruction parsing and semantic understanding;

[0111] S102: When processing voice information, the voice recognition (ASR) engine is used to achieve high accuracy and fast response capabilities; and through complex audio signal processing, acoustic model matching and language model analysis, combined with model training and tuning, it is possible to receive and process natural languages ​​in different dialects, different speech speeds, and different background noises, ensuring the fluency and naturalness of communication;

[0112] S103. Use computer vision (CV) algorithms and technologies to realize image and other visual input: extract rich information from images through object recognition, scene understanding, face recognition and optical character recognition (OCR) technologies; the information extracted from images includes identifying the text content in the image, analyzing the image subject and background, and even understanding the emotions and meanings conveyed by the image, and then converting them into unified and meaningful semantic expressions;

[0113] S104. After the diverse inputs of text, voice and images have gone through their respective processing flows, they are integrated into a unified high-level semantic model to form an identification of user intent, ensuring that no matter what form the user communicates in, the intent can be captured accurately, providing a solid foundation for subsequent intelligent decision-making and services.

[0114] The analysis and aggregation by means of the trained multimodal and cross-modal deep learning model in step S2 of this embodiment are specifically as follows:

[0115] S201. Use multimodal and cross-modal deep learning models to conduct in-depth analysis of unified semantic representations, accurately identify and process complex query logic, specifically: compound queries that contain "and", "or", and "not" Boolean logical relationships, and fine-tune the screening of multiple conditions of time series and spatial locations; at the same time, it is also used to track conversation history and user preferences with the help of context-aware mechanisms, understand the implicit intentions behind requests, and thus provide more personalized and context-adaptive information services;

[0116] S202. Optimize information screening and aggregation strategies to efficiently cope with information retrieval challenges in a big data environment, and use high-level semantic features learned by the model to quickly locate relevant data fragments, achieving accurate matching even in massive, heterogeneous data sets; at the same time, it is also used to automatically summarize and summarize information through intelligent aggregation algorithms, extract key points, and provide users with structured and easy-to-digest feedback, greatly improving the efficiency of information processing and the user's experience in obtaining information.

[0117] In step S3 of this embodiment, according to the requirements of the analysis engine output, data from different sources are processed and integrated in real time to ensure the accuracy and timeliness of the data, and the data structure is optimized to adapt to the large-screen display requirements as follows:

[0118] S301. Seamlessly connect and extract data across multiple heterogeneous data sources, and perform data cleaning and verification to effectively eliminate redundant information and correct inconsistencies; heterogeneous data sources include databases, API interfaces, and cloud storage services;

[0119] S302, using real-time data stream processing technology, and using a streaming computing framework to perform low-latency processing and analysis on continuously updated data streams; wherein the streaming computing framework includes Apache Kafka and Flink;

[0120] S303. Intelligently optimize the structure of the integrated data and use efficient data compression algorithms to reduce the transmission burden while retaining data details, so that large amounts of data can be smoothly loaded and displayed on the large screen; among them, the intelligent structure optimization uses a layered data model to improve the hierarchy and readability of information display;

[0121] S304. Adopt an adaptive layout algorithm to dynamically adjust the data view layout according to the actual size and resolution of the large screen, so that the charts, tables and map information can obtain the best display effect on large screens of different sizes.

[0122] In step S4 of this embodiment, advanced data visualization technology and interactive design are used to intuitively display the processed data on a large screen in the form of charts, heat maps and 3D models, support dynamic updates and user-customized views, and enhance data insights as follows:

[0123] S401, integrated with D3.js, ECharts, Three.js top visualization libraries, supports classic chart forms such as bar charts, line charts, and pie charts, and also implements advanced chart types such as heat maps, scatter plot matrices, and Sankey diagrams, as well as 3D models and animation effects, to reveal the trends, patterns, and correlations behind the data in multiple dimensions and levels;

[0124] S402. Relying on the real-time data processing engine, the visual content on the large screen is automatically refreshed as the data changes. At the same time, SadTalker is used to empower the IP image formed by the modeling, and the recognized content is effectively introduced to achieve a better interactive experience;

[0125] S403. Users can easily configure the display content according to their own focus, such as selecting a specific time range, data dimension, comparison indicator or perspective; it is also used to support drag-and-drop operations, allowing users to freely adjust the chart layout, or generate personalized data dashboards through simple settings.

[0126] Embodiment 3:

[0127] This embodiment also provides an electronic device, including: a memory and at least one processor;

[0128] Wherein, the memory stores computer-executable instructions;

[0129] The at least one processor executes the computer-executable instructions stored in the memory, so that the at least one processor executes the large-screen display method for intelligent questioning and number based on a multimodal large model as described in any one of the present inventions.

[0130] The processor may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The processor may be a microprocessor or any conventional processor, etc.

[0131] The memory can be used to store computer programs and / or modules. The processor realizes various functions of the electronic device by running or executing the computer programs and / or modules stored in the memory, and calling the data stored in the memory. The memory can mainly include a program storage area and a data storage area, wherein the program storage area can store an operating system, at least one application required for a function, etc.; the data storage area can store data created according to the use of the terminal, etc. In addition, the memory can also include a high-speed random access memory, and can also include a non-volatile memory, such as a hard disk, a memory, a plug-in hard disk, a smart memory card (SMC), a secure digital (SD) card, a flash memory card, at least one disk storage period, a flash memory device, or other volatile solid-state storage devices.

[0132] Embodiment 4:

[0133] This embodiment also provides a computer-readable storage medium, which stores a plurality of instructions, which are loaded by a processor to enable the processor to execute the large-screen display method for intelligent questioning based on a multimodal large model in any embodiment of the present invention. Specifically, a system or device equipped with a storage medium can be provided, on which a software program code that implements the functions of any of the above embodiments is stored, and a computer (or CPU or MPU) of the system or device reads and executes the program code stored in the storage medium.

[0134] In this case, the program code itself read from the storage medium can realize the function of any one of the above-mentioned embodiments, and thus the program code and the storage medium storing the program code constitute a part of the present invention.

[0135] The storage medium embodiments for providing the program code include a floppy disk, a hard disk, a magneto-optical disk, an optical disk (such as CD-ROM, CD-R, CD-RW, DVD-ROM, DVD-RAM, DVD-RW, DVD+RW), a magnetic tape, a non-volatile memory card, and a ROM. Alternatively, the program code can be downloaded from a server computer by a communication network.

[0136] In addition, it should be clear that the functions of any of the above embodiments can be implemented not only by executing the program code read by the computer, but also by enabling an operating system operating on the computer to complete part or all of the actual operations based on instructions from the program code.

[0137] In addition, it can be understood that the program code read from the storage medium is written to a memory provided in an expansion board inserted into the computer or written to a memory provided in an expansion unit connected to the computer, and then based on the instructions of the program code, a CPU installed on the expansion board or the expansion unit is enabled to perform part or all of the actual operations, thereby realizing the functions of any of the above-mentioned embodiments.

[0138] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. An intelligent large-screen display system based on a multi-modal large model, characterized in that: The system includes: A multimodal input processing module, which is used to receive user input and perform semantic conversion with the help of natural language processing technology; Multimodal large model analysis engine module, used for analysis and aggregation with the help of trained multimodal and cross-modal deep learning models; The data processing and integration module is used to process and integrate data from different sources in real time according to the output requirements of the analysis engine, ensure the accuracy and timeliness of the data, and optimize the data structure to adapt to the large-screen display requirements; The large-screen visualization display module is used to use advanced data visualization technology and interactive design to intuitively display the processed data on the large screen in the form of charts, heat maps and 3D models. It supports dynamic updates and user-customized views to enhance data insights.

2. According to claim 1, the intelligent large-screen display system based on multimodal large model is characterized in that: The multimodal input processing module is also used to bridge information between users and systems, integrating multiple perception and understanding technologies to ensure comprehensive and accurate capture of user intentions; The multimodal input processing module includes: The text instruction parsing and semantic understanding submodule is used to receive text input using natural language processing technology, deeply analyze the multi-dimensional features of syntactic structure, emotional tendency and keyword extraction, and convert the text content into structured data that can be understood by the machine, so as to perform complex instruction parsing and semantic understanding; The speech recognition submodule is used to achieve high accuracy and fast response capabilities through the speech recognition engine when processing speech information; and through audio signal processing, acoustic model matching and language model analysis, combined with model training and tuning, it can achieve the reception and processing of natural languages ​​in different dialects, different speech speeds, and different background noises, ensuring the fluency and naturalness of communication; The image recognition submodule is used to realize the input of pictures and other visuals with the help of computer vision algorithms and technologies: object recognition, scene understanding, face recognition and optical character recognition are used to extract rich information from images; the information extracted from the image includes recognizing the text content in the image, analyzing the image subject and background, and even understanding the emotions and meanings conveyed by the image, and then converting them into unified and meaningful semantic expressions; The integration sub-module is used to integrate the diverse inputs of text, voice and images into a unified high-level semantic model after their respective processing flows, so as to form recognition of user intent and ensure that no matter how the user communicates, the intent can be captured accurately, providing a solid foundation for subsequent intelligent decision-making and services.

3. According to claim 1, the intelligent large-screen display system based on multimodal large model is characterized in that: The multimodal large model analysis engine module uses a pre-trained cross-modal deep learning model to perform cross-domain semantic understanding and generation tasks; The multimodal large model analysis engine module includes: The deep semantic analysis and intent recognition submodule is used to use multimodal and cross-modal deep learning models to deeply analyze unified semantic representations, accurately identify and process complex query logic; at the same time, it is also used to track conversation history and user preferences with the help of context-aware mechanisms, understand the implicit intent behind requests, and thus provide more personalized and context-adaptive information services; The sub-module for accurate and rapid screening and aggregation of massive and heterogeneous data is used to optimize information screening and aggregation strategies, efficiently cope with information retrieval challenges in a big data environment, and use high-level semantic features learned by the model to quickly locate relevant data fragments, achieving accurate matching even in massive and heterogeneous data sets. At the same time, it is also used to automatically summarize and summarize information through intelligent aggregation algorithms, extract key points, and provide users with structured and easy-to-digest feedback.

4. The intelligent large-screen display system based on multimodal large model according to claim 1 is characterized in that: The data processing and integration module builds a data processing and integration engine, closely connects the output results of the analysis engine, and responds to every data request triggered by the user's complex needs in real time; Data processing and integration modules include: The data fusion submodule is used to seamlessly connect and extract data across multiple heterogeneous data sources, while performing data cleaning and verification to effectively eliminate redundant information and correct inconsistencies; heterogeneous data sources include databases, API interfaces, and cloud storage services; The real-time data stream processing submodule is used to build in real-time data stream processing technology, and uses the streaming computing framework to perform low-latency processing and analysis on continuously updated data streams; the streaming computing framework includes Apache Kafka and Flink; The data optimization and reconstruction logic submodule is used to perform intelligent structural optimization on the integrated data and use efficient data compression algorithms to reduce the transmission burden while retaining data details, so that large amounts of data can be smoothly loaded and displayed on the large screen; among them, the intelligent structural optimization adopts a layered data model to improve the hierarchy and readability of information display; The adaptive layout submodule is used to use an adaptive layout algorithm to dynamically adjust the data view layout according to the actual size and resolution of the large screen, so that charts, tables and map information can obtain the best display effect on large screens of different sizes.

5. The intelligent large-screen display system for questioning numbers based on a multimodal large model according to any one of claims 1 to 4, characterized in that: The large-screen visual display module includes: The visualization library integration submodule is used to integrate the top visualization libraries of D3.js, ECharts, and Three.js. It supports classic chart forms such as bar charts, line charts, and pie charts. It also implements advanced chart types such as heat maps, scatter plot matrices, and Sankey diagrams, as well as 3D models and animation effects, to reveal the trends, patterns, and correlations behind the data in multiple dimensions and levels. The real-time data processing submodule is used to automatically refresh the visual content on the large screen as the data changes based on the real-time data processing engine. At the same time, SadTalker is used to empower the IP image formed by the modeling, effectively introduce the recognized content, and achieve a better interactive experience. The view option customization module allows users to easily configure display content according to their own focus; it also supports drag-and-drop operations, allowing users to freely adjust the chart layout or generate personalized data dashboards through simple settings.

6. A method for displaying intelligent data on a large screen based on a multimodal large model, characterized in that: The method is as follows: Receive user input and perform semantic conversion with the help of natural language processing technology; Analysis and aggregation with the help of trained multi-modal and cross-modal deep learning models; According to the requirements of the analysis engine output, data from different sources are processed and integrated in real time to ensure the accuracy and timeliness of the data, and the data structure is optimized to adapt to the large-screen display requirements; Utilizing advanced data visualization technology and interactive design, the processed data is intuitively displayed on the large screen in the form of charts, heat maps and 3D models, supporting dynamic updates and user-customized views to enhance data insights.

7. The intelligent large-screen display method based on a multimodal large model according to claim 6 is characterized in that: Receive user input and use natural language processing technology to perform semantic conversion as follows: It uses natural language processing technology to receive text input, deeply analyzes the multi-dimensional features of syntactic structure, emotional tendency and keyword extraction, and converts text content into machine-understandable structured data, so as to perform complex instruction parsing and semantic understanding; When processing voice information, the voice recognition engine achieves high accuracy and fast response capabilities; and through audio signal processing, acoustic model matching and language model analysis, combined with model training and tuning, it can receive and process natural languages ​​in different dialects, different speech speeds, and different background noises, ensuring the fluency and naturalness of communication; Using computer vision algorithms and technologies to realize image and other visual input: object recognition, scene understanding, face recognition and optical character recognition are used to extract rich information from images; the information extracted from images includes identifying the text content in the image, analyzing the image subject and background, and even understanding the emotions and meanings conveyed by the image, and then converting them into unified and meaningful semantic expressions; After various inputs such as text, voice and images have gone through their respective processing flows, they are integrated into a unified high-level semantic model to form recognition of user intent. This ensures that no matter how users communicate, intent can be captured accurately, providing a solid foundation for subsequent intelligent decision-making and services. The analysis and aggregation with the help of trained multimodal and cross-modal deep learning models are as follows: It uses multimodal and cross-modal deep learning models to deeply analyze unified semantic representations, accurately identify and process complex query logic; at the same time, it is also used to track conversation history and user preferences with the help of context-aware mechanisms, understand the implicit intentions behind requests, and thus provide more personalized and context-adaptive information services; Optimize information screening and aggregation strategies to efficiently cope with information retrieval challenges in big data environments, and use high-level semantic features learned by the model to quickly locate relevant data fragments, achieving accurate matching even in massive, heterogeneous data sets; at the same time, it is also used to automatically summarize and summarize information through intelligent aggregation algorithms, extract key points, and provide users with structured and easy-to-digest feedback.

8. The intelligent large-screen display method based on a multimodal large model according to claim 6 is characterized in that: According to the requirements of the analysis engine output, data from different sources are processed and integrated in real time to ensure the accuracy and timeliness of the data, and the data structure is optimized to adapt to the large-screen display requirements as follows: Seamlessly connect and extract data across multiple heterogeneous data sources, while performing data cleaning and verification to effectively eliminate redundant information and correct inconsistencies; heterogeneous data sources include databases, API interfaces, and cloud storage services; Adopt real-time data stream processing technology and use streaming computing frameworks to perform low-latency processing and analysis on continuously updated data streams; streaming computing frameworks include Apache Kafka and Flink; Intelligent structural optimization is performed on the integrated data and efficient data compression algorithms are used to reduce the transmission burden while retaining data details, so that large amounts of data can be smoothly loaded and displayed on the large screen. Among them, intelligent structural optimization uses a layered data model to improve the hierarchy and readability of information display; Adopting adaptive layout algorithm, the data view layout is dynamically adjusted according to the actual size and resolution of the large screen, so that charts, tables and map information can obtain the best display effect on large screens of different sizes; Using advanced data visualization technology and interactive design, the processed data is intuitively displayed on the large screen in the form of charts, heat maps and 3D models, supporting dynamic updates and user-customized views to enhance data insights as follows: Integrate D3.js, ECharts, Three.js top visualization libraries, support classic chart forms such as bar charts, line charts, and pie charts, and realize advanced chart types such as heat maps, scatter plot matrices, and Sankey diagrams, as well as 3D models and animation effects, to reveal the trends, patterns, and correlations behind the data in multiple dimensions and levels; Relying on the real-time data processing engine, the visual content on the large screen is automatically refreshed as the data changes. At the same time, SadTalker is used to empower the IP image formed by the modeling, effectively introduce the recognized content, and achieve a better interactive experience; Users can easily configure the display content according to their own focus; it also supports drag-and-drop operations, allowing users to freely adjust the chart layout or generate personalized data dashboards through simple settings.

9. An electronic device, characterized in that: include: memory and at least one processor; Wherein, the memory stores computer-executable instructions; The at least one processor executes the computer-executable instructions stored in the memory, so that the at least one processor executes the large-screen display method for intelligent questions and numbers based on a multimodal large model as described in any one of claims 6 to 8.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-executable instructions. When the processor executes the computer-executable instructions, the large-screen display method of intelligent number query based on a multimodal large model as described in any one of claims 6 to 8 is implemented.

Images