Intelligent large-screen multi-round dialogue linkage method based on large model

Abstract

The invention relates to the technical field of artificial intelligence, in particular to an intelligent large-screen multi-round dialogue linkage method based on a large model. A user can control the intelligent large screen anytime and anywhere through the mobile phone without being limited by time and equipment operation. Compared with a traditional interaction mode, the user experience is more natural and convenient. A two-dimensional code generated by an intelligent large screen is scanned through a mobile phone program, a safe interaction channel is established through a WSS protocol of Websocket, and multi-round dialogue control is performed by using a large model.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and specifically to a method for multi-round dialogue linkage of a smart large screen based on a large model; the method is applied to a method for realizing multi-round dialogue linkage of a smart large screen based on a large model by using a mobile phone program. Background Art

[0002] In today's digital and intelligent era, people's demand for intelligent interactive experience is growing. Traditional interactive methods often have certain limitations and cannot meet the increasingly complex and diverse application scenarios.

[0003] In the field of smart devices, smart large screens have been widely used as an important carrier for information display and interaction, but there are still some shortcomings in multi-round dialogue interaction. At the same time, mobile phones, as devices that people carry with them every day, have convenient operation and powerful functions.

[0004] However, how to organically combine mobile phones with smart large screens to achieve efficient, convenient and intelligent multi-round dialogue linkage has always been a technical problem to be solved.

[0005] With the continuous development and maturity of big model technology, new possibilities are provided for realizing smarter and more accurate multi-round dialogues. However, how to effectively apply big model technology to the multi-round dialogue linkage between mobile phone programs and smart large screens still requires further exploration and innovation to adapt to the ever-changing market demands and user expectations. Against this background, the present invention comes into being, aiming to provide a new method for multi-round dialogue linkage between mobile phones and smart large screens based on big models, so as to improve the deficiencies of the prior art. Summary of the invention

[0006] The present invention solves the deficiencies of the prior art and provides the following technical solutions: The method of multi-round dialogue linkage of a smart large screen based on a large model includes the following steps: S1. After the user logs in, the system automatically connects to the business management system based on WebSocket technology After completing the login operation, the user can smoothly enter the smart screen; at this time, the system will automatically realize efficient connection with the business management system based on Websocket technology. The background will create WebSocketServer according to the sreeenId identifier and add it to the thread-safe ConcurrentHashMap, where sreeenId is the key and WebSocketServer is the value; When the user establishes a WebSocket connection, the system will generate a QR code. The process is as follows: (1) The business management end encodes the user information into JSON format; (2) Combine the encrypted information with the server URL to generate a URL containing the account information; (3) The front end passes the URL containing the account information to the business management back end to generate a QR code interface. The management back end returns the generated QR code image address, and the front end displays and renders it. The business back end uses Python's qrcode library to generate the QR code. S2. Scan the smart screen voice interactive QR code with your mobile phone to establish an interactive channel When there is a need to use a mobile phone to carry out linkage activities with a smart screen, the user needs to open the relevant program on the mobile terminal, and then use the program to scan the linkage dialogue QR code displayed on the smart screen. After the scan is completed, multiple rounds of dialogue processes can be started. In this way, an interactive channel between the mobile phone and the smart screen is conveniently established; When the user scans the QR code, the business management backend decrypts and verifies the user logo. After verification, the status code and Token are returned. The value of ConcurrentHashMap in step S1 is obtained according to screenId, and a connection is established with the big screen, so that multiple rounds of conversations can be carried out; S3. User asks a question and submits it The user enters relevant questions on the mobile interface, and then these questions are transmitted to the business management system through a specific transmission mechanism. The business management system can receive the question information entered by the user from the mobile terminal; when the voice or text input by the user is transmitted to the business management backend, the business management system verifies the legitimacy of the Token. After the verification is passed, the input information is immediately transmitted to the large screen through Websocket to achieve real-time information synchronization; S4. The system performs a series of operations such as multi-modal decoding scheduling and reasoning and provides feedback The business management system will perform decoding and scheduling operations on the input content based on the multimodal alignment module, and distribute text and voice to different processing modules. If the input content is in voice form, the system will first decode the natural voice and successfully convert it into text format. The voice-to-text model uses Qwen-Audio-Chat; if the input is in text form, the text will be directly submitted to the big model; the big model calls the intention recognition prompt project to identify the intent and obtain the SQL condition field and SQL template in the database, replace the SQL condition with the subject recognition template, and obtain the actual value of the SQL condition. The actual value of the SQL condition is combined with the SQL template for data retrieval. After obtaining the data, the data is supplemented to the data summary and data analysis prompt word template, and the big model is called. After the big model completes these reasoning tasks, the generated content will be accurately fed back to the business management system; S5. Realize synchronous data transmission and rendering between mobile phone and smart large screen based on data channel After receiving the relevant data content, the business management system will feed back the data content to the mobile terminal, and at the same time, it will synchronously transmit the data to the smart large screen through WebSocket. After receiving the data, the mobile terminal and the smart large screen will synchronously perform data visualization rendering, through this synchronous visualization processing.

[0007] The visualization engine of the smart large screen in the S5 uses Echars to create dynamic interactive charts to achieve real-time visualization and interaction of data. The mobile phone receives and updates data in real time by using RESTful API to display the optimized dynamic interface.

[0008] The process steps are shown in S1-S5. S2 and S4 are important innovations of this application. The specific process is as follows: After the S1 user logs in, the system automatically and efficiently connects with the business management system based on WebSocket technology; S2 mobile phone scans the intelligent large screen voice interactive QR code to establish an interactive channel; S3 users raise questions and submit them; The S4 system performs a series of operations such as multimodal decoding scheduling and reasoning and provides feedback; S5 realizes synchronous transmission and rendering of data between mobile phone and smart large screen based on data channel.

[0009] Traditional smart large screens are mainly controlled by remote controls, touch screens, keyboards and mice, and the control methods are not flexible enough. Mobile phones, as devices that people carry with them daily, have convenient operations and powerful functions. How to organically combine mobile phones with smart large screens to achieve efficient, convenient and intelligent multi-round dialogue linkage has always been a technical problem to be solved. Through the multi-round dialogue linkage between mobile phones and smart large screens, the interaction between users and smart devices becomes more natural, smooth and convenient. Users can enjoy more personalized services and accurate information. At the same time, this interactive method greatly enhances flexibility. Users can use their mobile phones to control the multi-round dialogues on the smart large screen at any time and any place, completely free from the constraints of time and space, thereby greatly improving the freedom of interaction. In addition, in important scenarios such as business and office, the present invention can help users quickly obtain the required information, thereby significantly improving work efficiency and decision-making effects.

[0010] When traditional Wensheng SQL technology is directly output to users, it is difficult to ensure the reliability of the results due to insufficient accuracy and stability, which in turn affects user experience and decision quality. We have cleverly integrated the Wensheng SQL function into the front end of the business management system, and introduced a manual correction and result preview mechanism by closely binding with user intent. This innovation not only directly addresses the two major pain points of Wensheng SQL accuracy and stability, but also opens up a new operating process, significantly improving the quality and reliability of system output. In the process of binding intent, Wensheng SQL templates and DDL information are fully utilized to automatically generate the required SQL instructions. This process greatly reduces the burden on operation and maintenance personnel, significantly improves work efficiency, achieves real cost reduction and efficiency improvement, and injects strong impetus into the digital transformation of enterprises.

[0011] Beneficial effects of the present invention Flexible control of multiple rounds of dialogue between mobile phone and smart large screen The present invention significantly improves the user's freedom and flexibility, allowing users to control the smart large screen through mobile phones anytime and anywhere, without being restricted by time and device operation. Compared with traditional interaction methods, the user experience is more natural and convenient. Scan the QR code generated by the smart large screen through the mobile phone program, establish a secure interaction channel through the WSS protocol of Websocket, and use the large model for multi-round dialogue control. In Examples 1 and 2, the large screen can be controlled by the mobile phone, realizing the linkage between the large screen and the mobile phone, and increasing the diversity of large screen operations.

[0012] Efficient data transmission and synchronization mechanism Compared with the traditional HTTP request method, the WebSocket protocol provides real-time, two-way communication, significantly reduces latency, improves the efficiency and accuracy of data transmission, ensures the consistency of data between the mobile phone and the smart screen, and avoids problems caused by inconsistent data. The WebSocket protocol and high-speed data transmission channel are used to achieve real-time data transmission and synchronization between the mobile phone and the smart screen. In Examples 1 and 2, the use of WebSocket can achieve millisecond-level data synchronization and synchronization.

[0013] Wensheng SQL is separated from Q&A, which improves the accuracy of the smart screen; SQL is converted into SQL template to improve the versatility of SQL.

[0014] Compared with the traditional SQL generation and execution, prepending SQL generation and manually correcting the SQL generation results can not only improve the system response time, but also improve the accuracy of the data generated by the smart screen. Converting SQL statements into SQL templates can reduce the writing of SQL statements and improve SQL utilization. Introducing a large language model to generate SQL-related information based on prompt words can improve the work efficiency of operation and maintenance personnel. Define intent classification in the background management system, create detailed intents, and bind data elements to the detailed intents. The data elements are generated based on the prompt words, with the help of the language large model, SQL templates, SQL variables, SQL fields and other information in the above voice input scenario. When the user on the smart screen asks a question, with the help of the large language model, the intent in the background management system is hit, and the relevant SQL template, SQL variable and SQL field information are obtained. Then, the subject of the user question is identified, and the prompt word template is identified according to the user subject. The user question and SQL condition information are passed in to generate the detailed value of the SQL variable, and then the SQL statement is executed in combination with the SQL template to output the result. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] Figure 1 This is a flowchart for the dialogue linkage process between the mobile terminal and the smart large screen. DETAILED DESCRIPTION

[0016] In order to overcome the problems existing in the above-mentioned prior art, the purpose of the present invention is to provide an innovative method for realizing multi-round dialogue linkage of a smart large screen based on a large model by using a mobile phone program, while ensuring the secure transmission of data between the smart large screen and the mobile phone program. The language large model used in this patent is Qwen2-72B-Instruct, and the language large models mentioned below are all this model. This model is mainly used to complete Wensheng SQL, intent recognition, subject recognition and multi-round dialogue. Multi-round dialogue can realize the ability to understand the context of user questions, and through continuous and natural communication, users can feel more understood and respected, and enhance the friendliness and satisfaction of the interaction.

[0017] On the mobile phone side, we have developed a powerful application that implements multi-round dialogue capabilities based on a large model. The application includes a module for stable communication and interaction with the smart large screen. Through this application, users can easily operate and control the process of multi-round dialogue.

[0018] The smart large screen has a built-in dialogue system based on a large model, which can deeply understand and analyze the input information, accurately identify user intentions, and generate accurate and personalized responses.

[0019] When the smart large screen starts the multi-round dialogue mode and requires voice interaction, the mobile phone will scan the QR code generated by the smart large screen, establish a secure data transmission connection, and realize linkage with the smart large screen.

[0020] In order to ensure the security of data during transmission, we use advanced encryption technology to encrypt the data between the mobile phone program and the smart screen. This ensures that the data will not be stolen or tampered with during transmission, protecting the privacy and data security of users.

[0021] At the same time, we have established a strict identity authentication mechanism, and only authorized users can perform data transmission and dialogue linkage. This effectively prevents unauthorized access and malicious attacks.

[0022] In terms of technical implementation planning, the mobile app and the smart screen are connected through a high-speed data transmission channel. When the user starts a conversation operation on the mobile phone, the command will be transmitted to the backend service at a very fast speed. The big model will accurately identify and efficiently process the received information, and then generate multiple rounds of conversation content. At the same time, it will be synchronized to the mobile app and the smart screen in real time through a safe and stable transmission channel, so as to ensure that the content on both ends can be displayed synchronously and accurately.

[0023] In order to improve the accuracy of the conversation and the personalized experience, the big model will continue to learn and optimize. It will adjust according to the user's historical conversation data and preferences to provide responses that better meet the user's needs. The overall idea is shown in the figure below, and the methods include S1-S5.

[0024] After the user logs in, the system automatically connects efficiently with the business management system based on WebSocket technology. After completing the login operation, the user can smoothly enter the smart screen. At this time, the system will automatically achieve efficient connection with the business management system based on Websocket technology. The background will create WebSocketServer according to the sreeenId identifier and add it to the thread-safe ConcurrentHashMap, where sreeenId is the key and WebSocketServer is the value. In this process, the user does not need to perform additional manual operations. The system will intelligently complete this connection step, thus laying a solid foundation for subsequent interactive activities.

[0025] WebSocket is a protocol for full-duplex communication over a single TCP connection with the following main features: Real-time: Realize real-time data interaction between client and server, and reduce communication delay compared to the traditional HTTP request / response mode.

[0026] Two-way communication: allows the server to actively push data to the client, not just respond to client requests.

[0027] Efficiency: Reduce the overhead of connection establishment and closing.

[0028] WebSocket enables web applications to implement functions such as real-time chatting and real-time data updating. It is widely used in online games, real-time monitoring, and instant messaging, greatly enriching the functions and user experience of Web applications.

[0029] To ensure the security of WebSocket, we adopt the following three measures: (1) Using the WSS protocol, WSS transmits data through SSL / TLS encryption to prevent data from being eavesdropped and tampered with.

[0030] (2) Perform user authentication and authorization before establishing a WebSocket connection. You can use standard authentication methods such as JWT (JSON Web Tokens), OAuth, etc., and attach tokens or credentials in the handshake request.

[0031] (3) Deploy a Web Application Firewall (WAF) to monitor and filter Websocket traffic. Implement rate limiting and connection limiting to prevent malicious clients from abusing connection resources.

[0032] When the user establishes a WebSocket connection, the system will generate a QR code. The process is as follows: (1) The business management end encodes user information into JSON format, such as {"userIdCode":"user identification code","screenId":"large screen id"}, and uses SM2 encryption to protect user privacy and security. SM2 is an asymmetric encryption algorithm. In asymmetric encryption, different key pairs are used for encryption and decryption, namely public key and private key. The SM2 algorithm is an elliptic curve asymmetric encryption algorithm developed by the China National Cryptography Administration for use in secure communication scenarios such as digital signatures and key agreement. The security of the SM2 algorithm is based on the difficulty of the elliptic curve discrete logarithm problem. Compared with the traditional RSA algorithm, the SM2 algorithm has higher security and efficiency. The SM2 algorithm has a shorter key length, so at the same security level, the SM2 algorithm has a faster calculation speed and occupies less storage space.

[0033] (2) Combine the encrypted information with the server URL to generate a URL containing the account information. The format is http:domain name:port / qrCodeAuth paras=JSON string encrypted using SM2.

[0034] (3) The front end passes the URL containing the account information to the business management back end to generate a QR code interface. The management back end returns the generated QR code image address, and the front end displays and renders it. The business back end uses Python's qrcode library to generate the QR code.

[0035] The mobile phone scans the voice interaction QR code of the smart screen to establish an interactive channel. When there is a need to use a mobile phone to carry out linkage activities with the smart screen, the user needs to open the relevant program on the mobile terminal, and then use the program to scan the linkage dialogue QR code displayed on the smart screen. After the scan is completed, multiple rounds of dialogue processes can be started. In this way, an interactive channel between the mobile phone and the smart screen is conveniently established.

[0036] When the user scans the QR code, the business management backend decrypts and verifies the user logo. After verification, the status code and Token are returned. The value of ConcurrentHashMap in step S1 is obtained according to screenId, and a connection is established with the large screen, so that multiple rounds of conversations can be carried out.

[0037] The user asks a question and submits it. The user enters the relevant question on the mobile interface, and then the question is transmitted to the business management system through a specific transmission mechanism. In this way, the business management system can receive the question information entered by the user from the mobile terminal and prepare for subsequent processing and analysis.

[0038] When the voice or text input by the user is transmitted to the business management backend, the business management system verifies the legitimacy of the Token. Once the verification is passed, the input information is immediately transmitted to the large screen through Websocket to achieve real-time synchronization of information.

[0039] The system performs a series of operations such as multimodal decoding scheduling and reasoning and provides feedback. The business management system will implement decoding scheduling operations on the input content based on the multimodal alignment module. Among them, the multimodal decoding scheduling system is independently developed and can distribute text and voice to different processing modules. It has the key technical requirements of real-time processing, fault recovery, privacy protection, resource optimization, and intelligent scheduling to achieve the best resource utilization and the best decoding effect. Specifically, if the input content is in the form of voice, the system will first decode the natural voice and successfully convert it into text format. The voice-to-text model uses Qwen-Audio-Chat; if the input is in the form of text, the text will be directly submitted to the big model. After the big model calls the intention recognition prompt project to recognize the intention, it obtains the SQL condition field and SQL template in the database, replaces the SQL condition with the subject recognition template, and obtains the actual value of the SQL condition. The actual value of the SQL condition is combined with the SQL template for data retrieval. After obtaining the data, the data is supplemented to the data summary and data analysis prompt word template, and the big model is called. After the big model completes these reasoning tasks, the generated content will be accurately fed back to the business management system.

[0040] Specific example: Voice input of "Gender percentage of tourists in Tianshui City in 2024".

[0041] Prerequisites: (1) Add a new intent: Click the Add Intent button in the backend business system, select the intent category, fill in the intent name as "Tourist Gender", then select the rendered chart as a pie chart, and finally fill in the prompt word "Gender ratio of tourists in Tianshui during Labor Day in 2024".

[0042] (2) SQL information generation: According to the prompt words filled in above, click the Generate Script button. The backend classifies the intent and finds the related DDL statements. Then, through the Wensheng SQL prompt word template, combined with the DDL and prompt words, the big model can infer the relevant SQL information (SQL template script, SQL field information, SQL condition information, preview SQL script), and return the generated SQL information to the front end. The specific information generated is as follows: SQL field: [{"field description":"male","field English name":"male_ratio"},{"field description":"female","field English name":"female_ratio"}] SQL conditions: [{"Field English name":"startTime","Field description":"Start time","Field type":"varchar"},{"Field English name":"endTime","Field description":"End time","Field type":"varchar"},{"Field English name":"cname","Field description":"Statistical city","Field type":"varchar","data":"Tianshui city"}] SQL template script: select avg(male_ratio) male_ratio,avg(female_ratio)female_ratio from tpfm_tourist_label_day WHEREscope ='2' and cname like #{cname} and stime>= #{startTime} and stime<= #{endTime} SQL preview script: select male_ratio,female_ratio from tpfm_tourist_label_day WHERE scope ='2' and ccode = 620500 and stime>= '2024-05-01'and stime<= '2024-05-05' (3) Data preview and save: Click Preview, and the backend will return the data format required for the pie chart according to the chart type. The frontend will render the pie chart. Click Save after the rendering is correct. Otherwise, modify the generated SQL information until it is rendered correctly. The saved data will be used in the intent recognition and subject recognition below.

[0043] Process analysis: (1) User input voice content: The user inputs "Gender ratio of tourists in Tianshui City in 2024" through the microphone of the program on the mobile phone and transmits it to the backend system.

[0044] (2) Speech-to-text: After receiving the speech, the background passes it to Qwen-Audio-Chat, which then performs inference and generates text information.

[0045] (3) Intent recognition: Based on the intent recognition template, the user question converted from speech to text is replaced with the corresponding variables in the template, and the language model Qwen2-72B-Instruct is called to infer the corresponding intent. In this example, the intent recognized is the tourist's gender. After the intent is recognized, it can be associated with the big model to generate SQL information in advance.

[0046] (4) Subject identification: Replace the variable of the subject identification template prompt word with the SQL condition of the above SQL information to identify the subject information. The subject information in this example is: {"came":"Tianshui City","startTime":"2024-01-01","endTime":"2024-12-31"}.

[0047] (5) Database query: The system queries the database and returns the data results based on the SQL information associated with the subject's identified variables and intended data elements - SQL template.

[0048] (6) Chart data processing: Convert the data into pie chart data according to the chart type associated with the intent.

[0049] (7) Data summary: Based on the data summary prompt word template and the queried data, the language model inference results are obtained.

[0050] (8) Data analysis: Based on the prompt words and the queried data of the data analysis system, the language model inference results are obtained.

[0051] (9) Output results: Return chart data, data summary, and data analysis results to the large screen and mobile phone program for rendering.

[0052] The text input case is similar to the speech case, except that steps (1) and (2) are omitted.

[0053] Based on the data channel, the data on the mobile phone and the smart screen are synchronously transmitted and rendered. After receiving the relevant data content, the business management system will feed back the data content to the mobile terminal, and at the same time, it will synchronously transmit the data to the smart screen through WebSocket. After receiving the data, the mobile terminal and the smart screen will synchronously carry out data visualization rendering. Through this synchronous visualization processing, the linkage interaction effect between the two ends is successfully achieved. This linkage interaction allows users to have a coherent and consistent experience on different terminal devices. For example, when the user enters "check today's weather" on the mobile terminal, after a series of processing, the smart screen will synchronously display weather-related data visualization information, allowing users to clearly and intuitively understand the required information on both terminals at the same time.

[0054] The smart large screen uses advanced visualization engines and visualization components. The visualization engine uses Echars. The visualization components are independently developed based on the visualization engine. They are compatible with mainstream Web browsers and mobile terminals, and the components are easy to expand. They can support the key technical requirements of multiple charts and create dynamic interactive charts. Dynamic interactive charts have high-performance rendering and respond to dynamic data. The key technical requirements significantly improve rendering efficiency and graphics performance, realize real-time visualization and interaction of data, and provide users with an intuitive and smooth visual experience. The mobile terminal uses RESTful API to receive and update data in real time, and displays an optimized dynamic interface, thereby improving the smoothness and efficiency of the user experience.

[0055] Through the method of the present invention, we have achieved efficient, convenient, safe and intelligent multi-round dialogue linkage between mobile phones and smart large screens. This brings users a new interactive experience and higher application value. At the same time, through technical means such as voice interaction and QR code scanning, the dialogue mode and application scenarios are further enriched.

[0056] Embodiment 1: User 1 successfully logs in on the smart screen, and the system displays a QR code.

[0057] User 1 uses a mobile phone to scan the QR code, establishes a secure interactive channel with the smart screen, and starts a multi-round dialogue system.

[0058] User 1 inputs via voice on his mobile phone: "How many tourists did Maiji Mountain receive today?" The system returns and displays on the mobile phone: "The number of visitors received by Maiji Mountain today is 1,128." It is displayed synchronously on the smart large screen and quickly rendered as a block diagram.

[0059] User 1 continues voice input: "What is the saturation of Maiji Mountain today?" The system returns and displays on the mobile phone: "Today's passenger flow saturation of Maiji Mountain is 17%." It is displayed synchronously on the smart large screen and quickly rendered as a dashboard.

[0060] Embodiment 2: User 2 successfully logs in on the smart screen, and the system displays a QR code.

[0061] User 2 uses a mobile phone to scan the QR code, establishes a secure interactive channel with the smart screen, and starts a multi-round dialogue system.

[0062] User 2 inputs via voice on the phone: "What is the current tourist density of Maiji Mountain Visitor Center?" The system returns and displays on the mobile phone: "The current tourist density of Maijishan Visitor Center is 1.42㎡ / person, which is sparsely populated with tourists and no safety hazards." It is displayed synchronously on the smart large screen and quickly rendered as a block diagram.

[0063] User 2 continues voice input: "Summary of the tourist profile of scenic spots during the National Day in 2023?" The system returns and displays on the mobile phone: "The gender structure of Maiji Mountain Scenic Area during the National Day in 2023: males account for 47%, females account for 53%, age structure: young people account for 17%, middle-aged people account for 43%, and the elderly account for 30%" It is displayed synchronously on the smart large screen and quickly rendered into two fan-shaped graphs.

[0064] Figure 1 Flowchart for the dialogue linkage process between the mobile terminal and the smart large screen After the user completes the login, he can enter the smart big screen. The system will automatically connect to the business management system efficiently based on WebSocket technology. When there is a need to link the mobile phone with the smart big screen, the user opens the relevant program on the mobile terminal and scans the linkage dialogue QR code on the smart big screen to start the voice dialogue. The user inputs the question on the mobile terminal and transmits it to the business management system through the transmission mechanism. The business management system decodes and schedules the input content according to the multimodal alignment module. The voice form is first decoded into text and then submitted to the big model, and the text form is directly submitted. The big model performs user intent recognition, reasoning, data retrieval and other operations and then feeds back to the business management system. After receiving the data, the business management system feeds back the data content to the mobile terminal and transmits it to the smart big screen simultaneously. The mobile terminal and the smart big screen simultaneously perform data visualization rendering to achieve linkage interaction at both ends, allowing users to get a consistent experience on different terminals.

[0065] (I) DDL statements CREATE TABLE `tpfm_tourist_label_day` ( `id` varchar(21) COMMENT 'Record number', `stime` varchar(20) COMMENT 'Statistical time', `scope` int(11) COMMENT 'Statistical scope 1: province; 2: city; 3: county; 4: place (e.g. scenic area, street, museum, airport, etc.)', `pname` varchar(32) COMMENT 'Statistical province name', `cname` varchar(40) COMMENT 'Statistical city name', `aname` varchar(50) COMMENT 'Statistical county name', `sname` varchar(100) COMMENT 'Statistical location name', `male_ratio` float COMMENT 'male', `female_ratio` float COMMENT 'Female', ); -- Reception tourist portrait-age, gender (day) (II) Prompt Word Template Wensheng SQL: You are a database expert, familiar with writing MySQL database query statements. Based on the user question: ${question}, combined with the database schema, complete the following tasks Task: 1. Generate a query SQL statement and ensure that the generated query is executable on the given 'database schema' below.

[0066] 2. Identify the query conditions, which must be in the SQL statement; 3. Identify the query fields.

[0067] 4. Identify the SQL template and generate the query SQL statement based on the generated query and filter conditions, such as select tvalue from tpfm_reception_pt_day where snamelike #{sname} and stime = #{stime}.

[0068] 5. Please only output JSON data, do not output the analysis process content, to ensure that JSON can be parsed correctly.

[0069] Database Schema: ${ddl} Example: User question: Gender ratio of tourists during the May Day holiday in 2024 Output: { "sqlCode": "SELECT male_ratio AS male, female_ratio AS femaleFROM tpfm_tourist_label_day WHEREstime BETWEEN '2024-05-01' AND '2024-05-05'", "sqlTemplate": "SELECT male_ratio AS male, female_ratio ASfemale FROM tpfm_tourist_label_day WHEREstime BETWEEN #{startTime} AND #{endTime}", "fields": [ { "Field Description": "Male Percentage", "Field English name": "male" }, { "Field Description": "Percentage of women", "Field English Name": "female" } ], "filters": [ { "Field English name": "startTime", "Field Description": "Start Time", "Field Type": "varchar" }, { "Field English Name": "endTime", "Field Description": "End Time", "Field Type": "varchar" } ] } Output requirements: Output format example: {"sqlCode": "SQL statement","sqlTemplate":"SQL template","fields": "query fields", "filters": "query conditions"} Query condition output format: [{"Field English name":"stime","Field description":"Statistical time","Field type":"varchar"},{"Field English name":"sname","Field description":"Scenic area","Field type":"varchar"}] Query field output format: [{"field description":"statistical time":,"field English name":"stime"},{"name":passenger flow":,"field English name":"num"}] Intent classification and intent recommendation: You are an intent classification intelligent assistant that has the ability to accurately analyze and classify user questions, identify intents based on user questions and intent descriptions, and recommend relevant intent classifications: User question: ${question}, Statement of Intent: The user's questions cover multiple aspects, and the intent classification and intent information are as follows.

[0070] ${IntentionInfo} Mission Statement: 1. Classify user intents, ensure that the classification is accurate and rigorous, and recommend relevant intent classifications. 2. It is not necessary to answer specific user questions, but only to classify intents according to questions and output intents in a fixed output format; 3. Make sure that the output intents are all intents in the intent description; 4. One question can hit multiple intents. 5. The other_intention_class in the output format is in addition to the classification in the data. Please ensure that the recommended intent classification is related to the user's question. 6. Make sure that the recommended intent classifications are not repeated.

[0071] Output JSON example: { "data": [ { "intention_class":"Passenger flow monitoring", "intention":"Customer flow summary", "intentionId":"101" } ], "other_intention_class": [ { "intention_class":"risk warning", "intention_class_id":"10002"} ] }.

[0072] User principal identification: Identify query conditions based on user questions and given SQL query conditions User question: ${question}.

[0073] SQL query conditions: ${filtersInfo} Date and time identification instructions: 1. Analyze the given text carefully and find all possible parts that represent dates or times.

[0074] 2. Recognize date and time expressions in various formats, including but not limited to: - Standard date format (e.g. YYYY-MM-DD, DD / MM / YYYY) - A date described in words (such as "tomorrow", "next Wednesday", "May 1st") - Time expressions (e.g. 14:30, 2:30 PM) - Relative time expressions (such as "3 hours later", "last month") - Time period (e.g. "January to March 2023") 3. Standardize the recognized date and time information into the format of YYYY-MM-DD HH:MM:ss.

[0075] 4. If there is no date or time information in the text, please clearly state "No date or time information found".

[0076] 5. Do not add any additional explanation or commentary, just list the identified date and time information.

[0077] 6. Each week starts on Monday and ends on Sunday.

[0078] Example: User question: Transportation options in Tianshui during the May Day holiday in 2024 Output: { "startTime": "2024-05-01 00:00:00", "endTime": "2024-05-05 23:59:59", "cname": "Tianshui City" } Output requirements: Output json format Data summary: You are a data summary intelligent assistant with the ability to present data information. Your task is to analyze and summarize the given questions and JSON data and finally output a statement text to explain them one by one. Follow the following rules: 1. Read the question and JSON data carefully word by word, and report to the leader in an appropriate manner; 2. Output in JSON format: the keys are status (status, 200 if analysis view is returned, 500 otherwise) and dataSummary (data summary). Do not output other irrelevant content. \n Question: [QUESTION]\n JSON data: [JSON_DATA] Analysis point of view: You are a data analysis assistant with the ability to analyze and summarize the key information mentioned by the user. Your task is to analyze and summarize the given questions and data summary and finally output a statement. Follow the following rules: 1. Read the questions and data summary carefully word by word, and express your analysis to the leader in an appropriate manner.

[0079] 2. Output in JSON format: the keys are status (status, 200 if the analysis view is returned, 500 otherwise), viewpoint (analysis view). Do not output other irrelevant content.

[0080] Question: [QUESTION] Data summary: [DATA_SUMMARY] Glossary WebSocket: A full-duplex communication protocol for two-way communication over a single TCP connection. The WebSocket protocol enables the server to actively push data to the client, thereby achieving real-time communication. It is often used in application scenarios that require fast response, such as online games, real-time chat, and stock quote push.

[0081] WSS protocol (WebSocket Secure): is an encrypted version of the WebSocket protocol, using TLS (Transport Layer Security) to ensure the security of communications and prevent data from being stolen or tampered with during transmission. The connection using the WSS protocol begins with "wss: / / ".

[0082] SM2 (Public Key Cryptography Algorithm Standard): It is a public key cryptography algorithm based on elliptic curves defined by the China National Cryptography Administration and is a national cryptography standard (GM / T 0003-2012). The SM2 algorithm is efficient and secure and is widely used in identity authentication and data encryption in e-government, finance and other fields.

[0083] Prompt: In the field of computer interaction, prompt refers to the signal that the system sends to the user to prompt or request input. For example, in a command line interface, prompt is usually a symbol or string that prompts the user to enter a command. In natural language processing (NLP), prompt can also refer to the input prompt that guides the language model to generate a response.

[0084] RESTful API (Representational State Transfer Application Programming Interface): is an API design style based on HTTP protocol, which uses standard HTTP verbs (such as GET, POST, PUT, DELETE) to create, read, update and delete resources. RESTful API is simple, stateless and scalable, and is widely used in Web service development.

[0085] SQL (Structured Query Language): is a standardized programming language for managing and operating relational databases. SQL provides a variety of functions, including querying data, updating data, deleting data, creating and modifying database structures, etc. SQL is a universal and easy-to-learn language, and is widely used in various database management systems (such as MySQL, PostgreSQL, Oracle, etc.).

[0086] ECharts: A pure JavaScript chart library developed by Baidu that provides a variety of chart types, supports data-driven dynamic and smooth animation effects, and is suitable for Web applications.

[0087] Qwen2-72B-Instruct: Alibaba's open-source language model, which has 72 billion parameters and powerful expressiveness, can usually complete a variety of natural language processing tasks such as text generation, question answering, translation, summary generation, sentiment analysis, etc.

[0088] Multi-turn conversation: refers to the continuous, contextualized multiple exchanges between users and machines during human-computer interaction. Compared with single-turn conversation, multi-turn conversation can provide a more natural and coherent interactive experience, which is closer to the way humans communicate.

[0089] Qwen-Audio: A large-scale audio language model developed by Alibaba Cloud that can convert speech to text.

Claims

1. A method for multi-round dialogue linkage of a large-scale intelligent screen based on a large model, characterized in that The steps include: S1: After the user logs in, the system automatically and efficiently connects with the business management system based on WebSocket technology After completing the login operation, the user can smoothly enter the smart screen; at this time, the system will automatically realize efficient connection with the business management system based on Websocket technology. The background will create WebSocketServer according to the sreeenId identifier and add it to the thread-safe ConcurrentHashMap, where sreeenId is the key and WebSocketServer is the value; When the user establishes a WebSocket connection, the system will generate a QR code. The process is as follows: (1) The business management end encodes the user information into JSON format; (2) Combine the encrypted information with the server URL to generate a URL containing the account information; (3) The front end passes the URL containing the account information to the business management back end to generate a QR code interface. The management back end returns the generated QR code image address, and the front end displays and renders it. The business back end uses Python's qrcode library to generate the QR code. S2: Scan the smart screen voice interactive QR code with your mobile phone to establish an interactive channel When there is a need to use a mobile phone to carry out linkage activities with the smart large screen, the user needs to open the relevant program on the mobile terminal, and then use the program to scan the linkage dialogue QR code displayed on the smart large screen; After the scan is completed, you can start a multi-round dialogue process; in this way, an interactive channel between the mobile phone and the smart screen is conveniently established; When the user scans the QR code, the business management backend decrypts and verifies the user logo. After verification, the status code and Token are returned. The value of ConcurrentHashMap in step S1 is obtained according to screenId, and a connection is established with the big screen, so that multiple rounds of conversations can be carried out; S3: User asks a question and submits it The user enters relevant questions on the mobile interface, and then these questions are transmitted to the business management system through a specific transmission mechanism. The business management system can receive the question information entered by the user from the mobile terminal; when the voice or text input by the user is transmitted to the business management backend, the business management system verifies the legitimacy of the Token. After the verification is passed, the input information is immediately transmitted to the large screen through Websocket to achieve real-time information synchronization; S4: The system performs a series of operations such as multi-modal decoding scheduling and reasoning and provides feedback The business management system will perform decoding and scheduling operations on the input content based on the multimodal alignment module, and distribute text and voice to different processing modules. If the input content is in voice form, the system will first decode the natural voice and successfully convert it into text format. The voice-to-text model uses Qwen-Audio-Chat; if the input is in text form, the text will be directly submitted to the big model; the big model calls the intention recognition prompt project to identify the intent and obtain the SQL condition field and SQL template in the database, replace the SQL condition with the subject recognition template, and obtain the actual value of the SQL condition. The actual value of the SQL condition is combined with the SQL template for data retrieval. After obtaining the data, the data is supplemented to the data summary and data analysis prompt word template, and the big model is called. After the big model completes these reasoning tasks, the generated content will be accurately fed back to the business management system; S5: Synchronous data transmission and rendering between mobile phone and smart large screen based on data channel After receiving the relevant data content, the business management system will feed back the data content to the mobile terminal, and at the same time, it will synchronously transmit the data to the smart large screen through WebSocket; After receiving the data, the mobile terminal and the smart large screen will simultaneously carry out data visualization rendering work through this synchronized visualization processing.

2. According to the method of multi-round dialogue linkage of intelligent large screen based on large model according to claim 1, it is characterized in that The visualization engine of the smart large screen in the S5 uses Echars to create dynamic interactive charts to achieve real-time visualization and interaction of data. The mobile phone receives and updates data in real time by using RESTful API to display the optimized dynamic interface.

Images