HMI interface generation methods, cloud computing, mobile devices, HMI terminals, and readable storage media

By collecting voice or text commands from mobile devices and generating HMI interfaces using large language models, the high barriers to entry and high costs of traditional HMI configuration solutions are solved, enabling rapid and intelligent interface development and updates.

CN123114468APending Publication Date: 2026-11-17XIAN HUICHUAN TECHNOLOGY R&D CENTER CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202611440890.7
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2026-09-16
Publication Date
2026-11-17

AI Technical Summary

Technical Problem

Traditional HMI configuration solutions are highly specialized, have high technical barriers, and incur high manpower and time costs, making them unable to meet the needs for rapid commissioning and lightweight design.

Method used

By collecting voice or text commands from mobile devices, a structured instruction set and a visual requirement description are generated using a large language model. Combined with a standardized dataset, an HMI interface is automatically generated and supports dynamic updates and adjustments.

Benefits of technology

It lowers the barrier to HMI interface development and iteration, significantly reduces labor and equipment maintenance costs, and improves the intelligence and convenience of industrial HMI.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN123114468A_ABST
    Figure CN123114468A_ABST
Patent Text Reader

Abstract

This application discloses an HMI interface generation method, a cloud platform, a mobile terminal, an HMI terminal, and a readable storage medium. This application relates to the field of industrial control technology. The cloud-based HMI interface generation method includes: receiving interface generation instruction text sent by a mobile terminal; inputting the interface generation instruction text into a pre-trained large language model to obtain the corresponding structured instruction set and visual requirement description text; querying the corresponding standardized dataset in the HMI terminal based on each structured instruction in the structured instruction set; and generating a first HMI interface based on the standardized dataset, the visual requirement description text, and preset interface generation rules, and pushing it to the HMI terminal for loading and display, and then pushing it to the mobile terminal for remote visualization. This application solves the problems of high technical barriers, high labor costs, and high time costs associated with traditional HMI configuration solutions.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] This application relates to the field of industrial control technology, and in particular to an HMI interface generation method, cloud platform, mobile terminal, HMI terminal, and computer-readable storage medium. Background Technology

[0002] HMI (Human-Machine Interface) is a commonly used control tool in the industrial control field. However, traditional HMI configuration solutions are highly specialized, involving tasks such as screen setup, point binding, and logic configuration that require professional industrial control engineers. This high technical threshold makes it difficult for general maintenance personnel to perform the operations. Furthermore, traditional HMI configuration processes are rigid, relying on specialized engineering software for offline configuration and requiring complex manual operations such as dragging and dropping controls, configuring parameters, and binding device points. Even minor functional adjustments to the HMI interface require a complete development process, resulting in high redundancy, slow iteration, heavy workload, and excessive human and time costs, failing to meet the needs for rapid on-site debugging and lightweight temporary modifications. Summary of the Invention

[0003] The main purpose of this application is to provide an HMI interface generation method, cloud platform, mobile terminal, HMI terminal and computer-readable storage medium, which aims to solve the problems of high configuration threshold, high manpower cost and high time cost of traditional HMI configuration solutions.

[0004] To achieve the above objectives, this application provides an HMI interface generation method for cloud applications, the HMI interface generation method comprising: Receive interface generation instruction text sent by the mobile terminal, wherein the interface generation instruction text is generated by the mobile terminal based on the collected first voice signal or first original text; The interface generates instruction text which is then input into a pre-trained large language model. The large language model then outputs the corresponding structured instruction set and a visual description of the requirements. Based on each structured instruction in the structured instruction set, query the corresponding standardized dataset in the HMI terminal; Based on the standardized dataset, the visual requirement description text, and the preset interface generation rules, a first HMI interface is generated; The first HMI interface is pushed to the HMI terminal for loading and display, and the first HMI interface is simultaneously pushed to the mobile terminal for remote visualization display.

[0005] In one embodiment, the interface generation rules include at least the mapping relationship between data types and chart types; The step of generating the first HMI interface based on the standardized dataset, the visual requirement description text, and the preset interface generation rules includes: Identify the data types corresponding to each frame in the standardized dataset; Based on the data types of each of the aforementioned screen data, determine the corresponding chart type for each of the aforementioned screen data; Based on the chart types, the screen data, and the visualization requirement description text, a first HMI interface is generated.

[0006] In one embodiment, the step of determining the chart type corresponding to each of the screen data according to the data type of each screen data includes: In response to the data type being a time-series continuous type, the chart type is determined to be a line chart; Since the data type is classified comparison, the chart type is determined to be a bar chart. In response to the data type being multi-dimensional percentage type, the chart type is determined to be a pie chart.

[0007] In one embodiment, after the step of generating the first HMI interface based on the standardized dataset, the visual requirement description text, and preset interface generation rules, the method further includes: Based on the runtime configuration reload mechanism, the changed production data, data mapping relationships and / or page display information are dynamically replaced to obtain updated configuration data; In response to meeting preset update conditions or receiving a manual update command, the first HMI interface is updated according to the update configuration data to obtain the second HMI interface; The second HMI interface is pushed to the HMI terminal for loading and display, and then pushed to the mobile terminal for remote visualization.

[0008] In one embodiment, the HMI interface generation method further includes: In response to receiving the interface adjustment instruction text sent by the mobile terminal, the interface adjustment instruction text is input into the large language model, and the large language model outputs the corresponding structured adjustment instruction; The first HMI interface is adjusted based on the structured adjustment instructions to obtain the third HMI interface; The third HMI interface is pushed to the mobile device for visual display on the mobile device; The third HMI interface is pushed to the HMI terminal for loading and display, and then pushed to the mobile terminal for remote visualization.

[0009] This application also provides an HMI interface generation method for mobile devices, the HMI interface generation method comprising: In response to the acquisition of a first speech signal or a first original text, the first speech signal or the first original text is preprocessed to obtain a second speech signal or a second original text, wherein the preprocessing includes at least one of noise reduction processing, filtering processing and sentence segmentation processing; Convert the second voice signal or the second original text into interface generation instruction text; The interface-generated instruction text is validated based on preset validation rules; After the interface generation instruction text passes verification, the interface generation instruction text is sent to the cloud so that the cloud can generate the first HMI interface based on the interface generation instruction text.

[0010] In one embodiment, the HMI interface generation method further includes: In response to the acquisition of a third speech signal or a third original text, the third speech signal or the third original text is preprocessed to obtain a fourth speech signal or a fourth original text. Convert the fourth voice signal or the fourth original text into interface adjustment instruction text; The interface adjustment instruction text is validated based on preset validation rules; After the interface adjustment instruction text passes verification, the interface adjustment instruction text is sent to the cloud so that the cloud can adjust the first HMI interface according to the interface adjustment instruction text.

[0011] This application also provides a cloud platform, which is a physical device, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, which are executed by the at least one processor to enable the at least one processor to perform the steps of the HMI interface generation method as described above.

[0012] This application also provides a mobile terminal, which is a physical device, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, which are executed by the at least one processor to enable the at least one processor to perform the steps of the HMI interface generation method as described above.

[0013] This application also provides an HMI terminal, which is a physical device. The HMI terminal includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, which, when executed by the at least one processor, enable the at least one processor to perform: providing a standardized dataset to the cloud in response to a query request from the cloud, so that the cloud can generate an HMI interface; receiving and loading the HMI interface pushed by the cloud for visualization on the HMI terminal; and dynamically loading the runtime configuration data in response to receiving updated configuration data to obtain an updated HMI interface.

[0014] This application also provides a readable storage medium, which is a computer-readable storage medium, storing a program that implements the HMI interface generation method. The program that implements the HMI interface generation method is executed by a processor to implement the steps of the HMI interface generation method as described above.

[0015] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the steps of the HMI interface generation method described above.

[0016] This application provides an HMI interface generation method. The cloud-based HMI interface generation method includes: receiving interface generation instruction text sent by a mobile terminal, wherein the interface generation instruction text is generated by the mobile terminal based on a first collected voice signal or first original text; inputting the interface generation instruction text into a pre-trained large language model, and having the large language model output a corresponding structured instruction set and a visual requirement description text; querying a corresponding standardized dataset in an HMI terminal based on each structured instruction in the structured instruction set; generating a first HMI interface based on the standardized dataset, the visual requirement description text, and preset interface generation rules; pushing the first HMI interface to the HMI terminal for loading and display, and simultaneously pushing the first HMI interface to the mobile terminal for remote visualization display.

[0017] In this way, users can generate corresponding interface generation command text by inputting voice signals or non-standard natural language text through mobile devices. This text is then sent to the cloud, where it is further processed by a large language model deployed in the cloud to generate structured commands and visual requirement description text. This allows for querying of the corresponding standardized datasets, ultimately generating the HMI interface. The above HMI interface generation process does not require the participation of professional configuration engineers. Ordinary maintenance personnel can quickly complete the development and iteration of the HMI interface through voice interaction, shortening the business development cycle and significantly reducing labor and equipment maintenance costs. Moreover, the technical solution of this application can rely on natural language voice interaction to replace the traditional cumbersome manual configuration operation, improving the intelligence and convenience of industrial HMI operation and maintenance. Attached Figure Description

[0018] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0019] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0020] Figure 1 This is a flowchart illustrating the HMI interface generation method applied to the cloud in this application embodiment; Figure 2 This is a flowchart illustrating the HMI interface generation method applied to a mobile terminal in an embodiment of this application. Figure 3 This is a schematic diagram of the composition of a feasible HMI interface generation system in an embodiment of this application; Figure 4 This is a schematic diagram of a feasible HMI interface generation process in an embodiment of this application; Figure 5 This is a schematic diagram of the device structure of the hardware operating environment involved in the HMI interface generation method applied to the cloud in this application embodiment; Figure 6 This is a schematic diagram of the device structure of the hardware operating environment involved in the HMI interface generation method applied to a mobile terminal in the embodiments of this application; Figure 7 This is a schematic diagram of the device structure of the hardware operating environment involved in the HMI interface generation method applied to the HMI terminal in the embodiments of this application.

[0021] The purpose, features, and advantages of this application will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0022] To make the above-mentioned objectives, features, and advantages of this application more apparent and understandable, the technical solutions in the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are merely some embodiments of this application, and not all embodiments. All other embodiments obtained by those skilled in the art based on the embodiments of this application without creative effort are within the scope of protection of this application.

[0023] It should be understood that the specific embodiments described herein are merely illustrative of the technical solutions of this application and are not intended to limit this application.

[0024] To better understand the technical solution of this application, a detailed description will be provided below in conjunction with the accompanying drawings and specific implementation methods.

[0025] This application provides an HMI interface generation method applied in the cloud, as described in the following embodiments. Figure 1 , Figure 1 This is a flowchart illustrating the HMI interface generation method applied to the cloud in this application. In this embodiment, the HMI interface generation method includes: Step S10: Receive the interface generation instruction text sent by the mobile terminal, wherein the interface generation instruction text is generated by the mobile terminal based on the collected first voice signal or the first original text. The mobile device can be a mobile phone, which can interact with the user through an application or mini-program, and collect the first voice signal input by the user. The first voice signal refers to the content about HMI interface generation spoken by the user. The mobile device can perform preliminary processing on the first voice signal in a lightweight way to generate a clearer and more specific interface generation instruction text and then send it to the cloud. After receiving the interface generation instruction text, the cloud starts to execute the HMI interface generation method process.

[0026] Alternatively, users can directly input the initial raw text via mobile devices, which will then be processed by the mobile application or mini-program to obtain standard interface generation command text.

[0027] Step S20: Input the interface-generated instruction text into the pre-trained large language model, and the large language model outputs the corresponding structured instruction set and visual requirement description text. A pre-trained large language model is deployed in the cloud. This model not only has natural language understanding capabilities but also learns knowledge about industrial applications and HMI devices. It can further perform deep semantic understanding, intent classification, and key field decomposition on the input interface-generated command text to accurately identify the core elements of the user's question, such as business intent, query dimensions, statistical indicators, time range, and filtering conditions. It completes the grammatical conversion step by step, transforming the disordered interface-generated command text into standard SQL statements through DSL2 SQL (Domain-Specific Language to Structured Query Language). This forms a structured command set that can be directly used to query relevant screen data, reducing the bias of manual configuration.

[0028] The visual requirements description text is used to describe the user's requirements for the HMI interface to be generated in various dimensions, such as layout, framework, components, and data sorting in each dimension.

[0029] The pre-trained large language model, serving as the core intelligent processing unit of the HMI interface generation system, is deployed on cloud computing nodes. Based on a general large language capability model, this large language model combines an industrial configuration-specific knowledge base, equipment data descriptions, configuration rules, and preset prompt templates. Through knowledge retrieval and context enhancement, it forms a professional recognition capability for industrial configuration scenarios. It performs semantic understanding, intent recognition, and key element extraction on the interface generation command text in natural language form uploaded from mobile devices, converting user needs into data query commands that can be executed in the database and structured visualization of requirements. This provides data query basis and screen generation basis for automatic generation of configuration-free screens, realizing the intelligent conversion from natural language requirements to structured industrial configuration commands.

[0030] Step S30: Query the corresponding standardized dataset in the HMI terminal according to each structured instruction in the structured instruction set; Understandably, the structured instruction set includes multiple structured instructions (i.e., standard SQL statements). A communication connection is pre-established between the cloud and the HMI terminal. The cloud sends each structured instruction to the HMI terminal, while the HMI terminal stores various standardized data related to that HMI terminal. The cloud issues a query request, and the HMI terminal performs a data query. After retrieving the standardized data corresponding to all structured instructions, a standardized dataset is formed.

[0031] It should be noted that standardized data is stored in the HMI terminal. Querying is performed on the HMI device, and the query results can be viewed on the HMI terminal, mobile phone, and PC.

[0032] Step S40: Generate the first HMI interface based on the standardized dataset, the visualized requirement description text, and the preset interface generation rules; Step S50: Push the first HMI interface to the HMI terminal for loading and display, and simultaneously push the first HMI interface to the mobile terminal for remote visualization display.

[0033] The interface generation rules are used to characterize the mapping relationship between various standardized data and graphical elements, which can include various types of statistical charts and tables. The final generated first HMI interface uses only the user's input voice signal as its data basis, and combines automated engines or tools such as large language models deployed in the cloud to achieve rapid HMI interface generation without configuration or code.

[0034] After the first HMI interface is generated, it is pushed to the HMI terminal for loading and display. It can also be viewed remotely on mobile devices and displayed synchronously on mobile devices.

[0035] Using the aforementioned HMI interface generation method, without manual programming, the industrial configuration structured instructions output by the large model can autonomously plan the HMI dashboard layout, intelligently match data visualization components, automatically render chart styles, and automatically generate a visualization interface adapted to the on-site working conditions, realizing zero-code, configuration-free, fully automatic construction of HMI dashboards.

[0036] This application's embodiment overcomes the industry pain points of traditional industrial HMIs—high configuration thresholds, long development cycles, and downtime required for iteration—by automatically generating screens via voice-driven, configuration-free methods. It boasts significant technological advancements, engineering practicality, and scenario versatility, resulting in outstanding cost reduction and efficiency improvements, strong production and maintenance stability, and a high degree of intelligence. No professional configuration engineers are needed; ordinary maintenance personnel can quickly complete HMI development and iteration via voice, shortening the business development cycle by over 90% and significantly reducing labor and equipment maintenance costs. Furthermore, it revolutionizes the operation mode and upgrades the interactive experience. This solution relies on natural language voice interaction to replace traditional cumbersome manual configuration operations, greatly improving the intelligence and convenience of industrial HMI operation and maintenance.

[0037] Furthermore, in one feasible embodiment, the interface generation rules at least include the mapping relationship between data types and chart types; Step S40, which generates the first HMI interface based on a standardized dataset, a visual description of requirements, and preset interface generation rules, may include: Step S41: Identify the data types corresponding to each frame in the standardized dataset; Step S41 can be executed by a large language model with standardized data understanding capabilities. The standardized dataset includes multiple screen data, which represent the screen elements in the HMI interface to be generated. Each screen data has at least one data type.

[0038] In another feasible embodiment, the large language model can also identify the dimensional attributes, data types, and business characteristics corresponding to the image data without the need for manual configuration.

[0039] Step S42: Determine the chart type corresponding to each screen data according to the data type of each screen. Step S43: Generate the first HMI interface based on each chart type, each screen data, and the visualization requirement description text.

[0040] Understandably, different data types can be displayed using different chart types to improve visualization and data presentation. Statistical charts can include one or more types such as line charts, pie charts, and bar charts.

[0041] Finally, based on the various requirements contained in the visualization requirements description text, the statistical charts of various chart types are arranged and rendered to generate the first HMI interface.

[0042] Furthermore, step S42, which determines the chart type corresponding to each frame of data based on the data type of each frame, may include: Step S421: In response to the data type being a time-series continuous type, determine the chart type as a line chart; Step S422: In response to the data type being a category comparison type, determine the chart type as a bar chart; Step S423: In response to the data type being multi-dimensional percentage type, determine the chart type as a pie chart.

[0043] Among them, the time-series continuous type refers to the data type that changes over time. Line charts can better reflect the changes in data and are used to show data trends. The classification comparison type refers to the situation where there are multiple different categories of data, and each category has different data. It is used to compare the size relationship between the data. Bar charts are used to compare the differences between multiple categories of data. The multi-dimensional proportion type refers to the situation where multiple dimensions are composed of a whole, and the focus is on reflecting the proportion of each dimension in the whole. Pie charts are used to visually present the proportion distribution of data in each dimension.

[0044] In another feasible embodiment, after step S40, which generates the first HMI interface based on the standardized dataset, the visualized requirement description text, and the preset interface generation rules, the method may further include: Step S41: Based on the runtime configuration reload mechanism, dynamically replace the changed production data, data mapping relationships and / or page display information to obtain updated configuration data; Step S42: In response to meeting the preset update conditions or receiving a manual update command, update the first HMI interface according to the update configuration data to obtain the second HMI interface. Step S43: Push the second HMI interface to the HMI terminal for loading and display, and push it to the mobile terminal for remote visualization display.

[0045] In this embodiment of the application, a dynamic data refresh method is also provided, which is used to update the newly downloaded data (including changed production data, data mapping relationships and / or page display information) to the generated first HMI interface during the operation of the HMI terminal, so as to realize the dynamic update of the HMI interface. Here, production data refers to the data generated by the industrial equipment corresponding to the HMI terminal during the operation.

[0046] The changed production data, data mapping relationships, and / or page display information originate from the HMI terminal. After receiving the changed production data, data mapping relationships, and / or page display information, the HMI terminal sends them to the cloud. The cloud then processes the data by identification, parsing, and loading to obtain updated configuration data. This updated configuration data is used to represent the display format of various data occurrences in the HMI interface. For example, it can be used to add, modify, and / or delete data sources, data chart interactive components, etc., to adjust and update the first HMI page.

[0047] Furthermore, the HMI interface can be automatically updated based on preset update conditions (such as a preset update cycle) or in response to a user-inputted manual update command. Therefore, when the preset update conditions are met or a manual update command is received, the first HMI interface is updated based on the latest update configuration content to obtain the second HMI interface. The updated second HMI interface is then pushed to the mobile device and displayed visually via a mobile application or mini-program.

[0048] This application embodiment utilizes a runtime configuration reload mechanism to dynamically replace newly downloaded and changed production data, data mapping relationships, and page display information on the HMI terminal. This allows the updated configuration to directly participate in subsequent data acquisition, processing, and display without requiring a complete application rebuild. Based on the currently valid configuration, the corresponding configuration data is reacquired through automatic or manual refresh, ensuring that the HMI interface display content remains synchronized with the latest configuration of the HMI terminal.

[0049] In another feasible embodiment, after the HMI interface is dynamically refreshed, the cloud can also transmit the latest dashboard status, device operation data, communication connection status, alarm logs and other information of the HMI terminal back to the HMI terminal display screen and the mobile app in real time. Users can remotely or on-site view and monitor the new HMI operation screen in real time to verify the configuration effect and device control status.

[0050] This application embodiment is based on a runtime configuration reloading mechanism, which dynamically replaces changed production data, data mapping relationships, and page display information, enabling the updated configuration to directly participate in subsequent data acquisition, processing, and display. After the configuration update takes effect, the updated configuration data corresponding to the currently valid configuration can be obtained through automatic or manual refresh, achieving synchronous updates of configuration configuration and display content. Through the above dynamic data mechanism, the offline update method in traditional industrial configuration is transformed into a runtime dynamic loading method, allowing configuration content to be continuously updated and quickly take effect as content changes occur. This reduces the need for system redeployment during configuration changes and improves the online adjustment capability, update efficiency, and operational adaptability of the industrial configuration system.

[0051] In another feasible embodiment, the HMI interface generation method may further include: Step S44: In response to receiving the interface adjustment instruction text sent by the mobile terminal, input the interface adjustment instruction text into the large language model, and output the corresponding structured adjustment instruction from the large language model. Step S45: Adjust the first HMI interface based on the structured adjustment instructions to obtain the third HMI interface; Step S46: Push the third HMI interface to the HMI terminal for loading and display, and push it to the mobile terminal for remote visualization display.

[0052] It should be noted that, in addition to providing a method for generating an HMI interface, this application also provides a method for adjusting an existing HMI interface (i.e., the first HMI interface) based on voice. In order to lower the threshold for adjusting and updating the HMI interface, it also supports users inputting commands on the mobile terminal via voice, generating corresponding interface adjustment command text, sending it to the cloud, and then the cloud processes the interface adjustment command text through a deployed large language model to output the corresponding structured adjustment command. Then, adjustments and updates are made based on the existing first HMI interface to generate the adjusted third HMI interface. Finally, the third HMI interface is pushed to the HMI terminal and simultaneously pushed to the mobile terminal for visualization on both the HMI terminal and the mobile terminal.

[0053] In one feasible embodiment, after browsing the first HMI interface on a mobile device, the user can determine whether the first HMI interface meets their current needs based on actual requirements, and choose whether to further update the first HMI interface by inputting interface adjustment commands via voice to the cloud, depending on the actual situation. Furthermore, after browsing the third HMI interface, the user can also choose whether to further adjust and update the third HMI interface, until the user is satisfied.

[0054] This application also provides an HMI interface generation method, applied to mobile devices, as described in the embodiments below. Figure 4 , Figure 4 This is a flowchart illustrating the HMI interface generation method applied to mobile devices according to this application. In this embodiment, the HMI interface generation method includes: Step A10: In response to the acquisition of the first speech signal or the first original text, preprocess the first speech signal or the first original text to obtain the second speech signal or the second original text. The preprocessing includes at least one of noise reduction processing, filtering processing and sentence segmentation processing. Step A20: Convert the second voice signal or the second original text into interface generation instruction text; Step A30: Verify the interface-generated instruction text based on preset verification rules; Step A40: After the interface generation command text passes the verification, the interface generation command text is sent to the cloud so that the cloud can generate the first HMI interface based on the interface generation command text.

[0055] Understandably, users can input the first voice signal or the first original text via voice or text input. Then, the mobile device can perform preliminary lightweight preprocessing through the built-in application or mini-program. For example, it can perform noise reduction, filtering, and sentence segmentation on the original first voice signal to eliminate noise interference in the industrial environment and generate a second voice signal. Subsequently, the second voice signal is accurately transcribed using a speech-to-text mini-model to generate the interface command text.

[0056] In addition, the interface-generated command text can be validated using preset validation rules to ensure the validity of the text. For example, content unrelated to HMI interface generation is considered to fail validation, and illogical or chaotic text is considered to fail validation.

[0057] Finally, after the verification is passed, the interface generation instruction text is sent to the cloud so that the cloud can generate the first HMI interface based on the interface generation instruction text. The process of the cloud generating the first HMI interface based on the interface generation instruction text can refer to the steps S10 to S40 mentioned above, and will not be repeated here.

[0058] In one feasible embodiment, the mobile terminal uses a mini-program as the lightweight and sole interactive entry point, supporting voice acquisition and text-assisted input in industrial settings and remotely without the need for installing a dedicated client or application. The mini-program can receive natural language commands from users, including creating online dashboards for alarm recording, data recording, and OEE (Overall Equipment Effectiveness), while also enabling HMI screen preview, favorite browsing, and equipment status viewing, adapting to lightweight operations across all scenarios.

[0059] In another feasible embodiment, the HMI interface generation method may further include: Step A50: In response to the acquisition of the third speech signal or the third original text, preprocess the third speech signal or the third original text to obtain the fourth speech signal or the fourth original text. Step A60: Convert the fourth voice signal or the fourth original text into interface adjustment instruction text; Step A70: Verify the interface adjustment command text based on preset verification rules; Step A80: After the interface adjustment instruction text passes the verification, the interface adjustment instruction text is sent to the cloud so that the cloud can adjust the first HMI interface according to the interface adjustment instruction text.

[0060] Steps A50 to A80 are executed after the first HMI interface is generated. In scenarios where the user needs to adjust the first HMI interface, a third voice signal or a third original text is input. The third voice signal or the third original text is used to characterize the description of the adjustment to the first HMI interface, such as adding, modifying and / or deleting data sources, data charts, interactive components, etc.

[0061] The mobile terminal performs preprocessing and verification of the third voice signal or the third original text based on the same approach as steps A10 to A30. After the interface adjustment command text passes verification, it is sent to the cloud so that the cloud can adjust the first HMI interface according to the interface adjustment command text. The steps for the cloud to adjust the first HMI interface based on the interface adjustment command text can refer to steps S44 to S46 of the aforementioned embodiment, and will not be repeated here.

[0062] For example, for ease of understanding, refer to Figure 3 This paper further explains the HMI interface generation system, which consists of cloud, mobile, and HMI terminals. Users can first input voice / text commands via mobile devices. The mobile devices can perform functions such as voice capture and construction, text-assisted input, command validity verification, HMI screen preview, favorites browsing, and device status viewing. Then, the system connects via HTTPS (HyperText Transfer Protocol). Secure (Hypertext Transfer Protocol) encryption is used to upload data to the cloud. The cloud-deployed large language model possesses capabilities such as a general large language model foundation, industry-specific configuration knowledge, knowledge retrieval and context enhancement, semantic understanding and intent recognition, NL (Natural Language)-DSL-SQL conversion, and structured visualization requirement description. Then, the large language model generates query commands and visualization requirements to produce visuals. The visual generation process includes: data feature recognition, intelligent chart matching (time series corresponds to line charts, categories correspond to bar charts, and percentages correspond to pie charts), and automatic layout planning, achieving zero-code visual rendering. When the latest configuration data is received, a dynamic data refresh process is performed based on the dynamic data refresh module. The dynamic data refresh module has configuration data recognition and parsing, runtime configuration reloading, dynamic replacement mechanism, automatic / manual refresh, and other processes, and dynamically loads and takes effect on the HMI terminal without requiring application rebuilding. Additionally, it should be noted that the communication methods between or within devices are as follows: HTTPS protocol is used from mobile devices to the cloud, HTTP (HyperText Transfer Protocol) is used from the front end to the HMI web service, and Socket (Socket Communication) is used within the HMI terminal. The industrial database can be stored in the HMI terminal or the cloud, without any restrictions.

[0063] For illustrative purposes, and for ease of understanding, reference is made to the foregoing embodiments. Figure 4 This paper describes a feasible HMI interface generation process. First, step 1: The mobile terminal completes command acquisition and front-end preprocessing. Specifically, this includes the user initiating a voice command via a mini-program, the voice acquisition module performing noise reduction filtering, the speech-to-text model transcribing the command, verifying its validity, and if valid, encrypting and uploading the standardized text command (i.e., the interface-generated command text). If invalid, the invalid command is discarded, and the user is prompted to re-enter. Step 2: The industrial big data model completes voice parsing and command generation. Specifically, this includes deep semantic understanding and intent classification, key field decomposition, conversion from NL to DSL to SQL, and generation of data query command sets and structured visualization requirements. Please describe the process; Step 3: Perform zero-configuration adaptive screen generation, complete automatic data feature recognition, with time-series types corresponding to line charts, category types corresponding to bar charts, and percentage types corresponding to pie charts, and finally complete zero-configuration automatic rendering; Step 4: Perform dynamic data updates, first identify and parse newly downloaded configuration data, complete dynamic loading in runtime without rebuilding, dynamically replace configuration data and mapping relationships, and display content based on automatic or manual attribute pages; Step 5: Display the HMI interface, which can further verify the configuration effect. If fine-tuning of the HMI page is required, return to Step 1, supporting secondary voice and text fine-tuning, realizing a complete closed loop of voice distribution - intelligent generation - dynamic data refresh - visual intelligent dashboard.

[0064] This application also provides a cloud platform, which includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, which are executed by the at least one processor to enable the at least one processor to execute the HMI interface generation method in the first embodiment above.

[0065] The following is for reference. Figure 5 It illustrates a schematic diagram of a cloud structure suitable for implementing embodiments of the present disclosure. It should be noted that... Figure 5 The cloud shown is merely an example and should not be construed as limiting the functionality and scope of use of the embodiments disclosed herein.

[0066] like Figure 5As shown, the cloud may include a processing device 1001 (e.g., a central processing unit, a graphics processing unit, etc.), which can perform various appropriate actions and processes according to a program stored in read-only memory (ROM) 1002 or a program loaded from storage device 1003 into random access memory (RAM) 1004. The RAM 1004 also stores various programs and data required for cloud operation. The processing device 1001, ROM 1002, and RAM 1004 are interconnected via a bus 1005. An input / output (I / O) interface 1006 is also connected to the bus. Typically, the following systems can be connected to the I / O interface 1006: input devices 1007 including, for example, touchscreens, touchpads, keyboards, mice, image sensors, microphones, accelerometers, gyroscopes, etc.; output devices 1008 including, for example, liquid crystal displays (LCDs), speakers, vibrators, etc.; storage devices 1003 including, for example, magnetic tapes, hard disks, etc.; and communication devices 1009. Communication device 1009 allows the cloud to communicate wirelessly or wiredly with other devices to exchange data. While the figure shows a cloud with various systems, it should be understood that implementation or possession of all the systems shown is not required. More or fewer systems may be implemented alternatively.

[0067] In particular, according to embodiments of this disclosure, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of this disclosure include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via a communication device, or installed from storage device 1003, or installed from ROM 1002. When the computer program is executed by processing device 1001, it performs the functions defined in the methods of embodiments of this disclosure.

[0068] The cloud-based solution provided in this application, employing the HMI interface generation method described in the above embodiments, can solve the technical problems of high configuration threshold, high manpower cost, and high time cost of traditional HMI configuration schemes. Compared with the prior art, the beneficial effects of the cloud-based solution provided in this application are the same as those of the HMI interface generation method provided in the above embodiments, and other technical features of this cloud-based solution are the same as those disclosed in the previous embodiment method, and will not be repeated here.

[0069] It should be understood that various parts of this disclosure can be implemented using hardware, software, firmware, or a combination thereof. In the description of the above embodiments, specific features, structures, materials, or characteristics may be combined in any suitable manner in one or more embodiments or examples.

[0070] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

[0071] This application also provides a mobile terminal, the mobile terminal including: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to execute the HMI interface generation method in the first embodiment above.

[0072] The following is for reference. Figure 6 It illustrates a structural diagram suitable for implementing embodiments of the present disclosure of a mobile terminal. It should be noted that... Figure 6 The mobile device shown is merely an example and should not be construed as limiting the functionality and scope of use of the embodiments disclosed herein.

[0073] like Figure 6As shown, the mobile terminal may include a processing device 2001 (e.g., a central processing unit, a graphics processing unit, etc.), which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 2002 or a program loaded from a storage device 2003 into a random access memory (RAM) 2004. The RAM 2004 also stores various programs and data required for the operation of the mobile terminal. The processing device 2001, ROM 2002, and RAM 2004 are interconnected via a bus 2005. An input / output (I / O) interface 2006 is also connected to the bus. Typically, the following systems can be connected to the I / O interface 2006: input devices 2007 including, for example, a touchscreen, touchpad, keyboard, mouse, image sensor, microphone, accelerometer, gyroscope, etc.; output devices 2008 including, for example, a liquid crystal display (LCD), speaker, vibrator, etc.; storage devices 2003 including, for example, magnetic tape, hard disk, etc.; and communication devices 2009. Communication device 2009 allows a mobile terminal to communicate wirelessly or wiredly with other devices to exchange data. Although the figures show mobile terminals with various systems, it should be understood that implementation or possession of all the systems shown is not required. More or fewer systems may be implemented alternatively.

[0074] In particular, according to embodiments of this disclosure, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of this disclosure include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via a communication device, or installed from storage device 2003, or installed from ROM 2002. When the computer program is executed by processing device 2001, it performs the functions defined in the methods of embodiments of this disclosure.

[0075] The mobile terminal provided in this application, employing the HMI interface generation method described in the above embodiments, can solve the technical problems of high configuration threshold, high manpower cost, and high time cost of traditional HMI configuration schemes. Compared with the prior art, the beneficial effects of the mobile terminal provided in this application are the same as those of the HMI interface generation method provided in the above embodiments, and other technical features of this mobile terminal are the same as those disclosed in the previous embodiment method, and will not be repeated here.

[0076] It should be understood that various parts of this disclosure can be implemented using hardware, software, firmware, or a combination thereof. In the description of the above embodiments, specific features, structures, materials, or characteristics may be combined in any suitable manner in one or more embodiments or examples.

[0077] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

[0078] This application embodiment also provides an HMI terminal, the HMI terminal comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform the following steps: in response to a query request from the cloud, providing a standardized dataset to the cloud for the cloud to generate an HMI interface; receiving and loading the HMI interface pushed by the cloud for visualization on the HMI terminal; and in response to receiving updated configuration data, performing dynamic loading in runtime to obtain an updated HMI interface.

[0079] The following is for reference. Figure 7 It shows a schematic diagram of the structure of an HMI terminal suitable for implementing embodiments of the present disclosure. It should be noted that... Figure 7 The HMI terminal shown is merely an example and should not be construed as limiting the functionality and scope of use of the embodiments disclosed herein.

[0080] like Figure 7As shown, the HMI terminal may include a processing device 3001 (e.g., a central processing unit, a graphics processing unit, etc.), which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 3002 or a program loaded from a storage device 3003 into a random access memory (RAM) 3004. The RAM 3004 also stores various programs and data required for the operation of the HMI terminal. The processing device 3001, ROM 3002, and RAM 3004 are interconnected via a bus 3005. An input / output (I / O) interface 3006 is also connected to the bus. Typically, the following systems can be connected to the I / O interface 3006: input devices 3007 including, for example, a touchscreen, touchpad, keyboard, mouse, image sensor, microphone, accelerometer, gyroscope, etc.; output devices 3008 including, for example, a liquid crystal display (LCD), speaker, vibrator, etc.; storage devices 3003 including, for example, magnetic tape, hard disk, etc.; and communication devices 3009. Communication device 3009 allows the HMI terminal to communicate wirelessly or wiredly with other devices to exchange data. Although the figures show HMI terminals with various systems, it should be understood that implementation or possession of all the systems shown is not required. More or fewer systems may be implemented alternatively.

[0081] In particular, according to embodiments of this disclosure, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of this disclosure include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via a communication device, or installed from storage device 3003, or installed from ROM 3002. When the computer program is executed by processing device 3001, it performs the functions defined in the methods of embodiments of this disclosure.

[0082] The HMI terminal provided in this application, employing the HMI interface generation method described in the above embodiments, can solve the technical problems of high configuration threshold, high manpower cost, and high time cost of traditional HMI configuration schemes. Compared with the prior art, the beneficial effects of the HMI terminal provided in this application are the same as those of the HMI interface generation method provided in the above embodiments, and other technical features of this HMI terminal are the same as those disclosed in the previous embodiment method, and will not be repeated here.

[0083] It should be understood that various parts of this disclosure can be implemented using hardware, software, firmware, or a combination thereof. In the description of the above embodiments, specific features, structures, materials, or characteristics may be combined in any suitable manner in one or more embodiments or examples.

[0084] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

[0085] This application also provides a computer-readable storage medium having computer-readable program instructions stored thereon, which are used to execute the HMI interface generation method in the first embodiment described above.

[0086] The computer-readable storage medium provided in this application embodiment may be, for example, a USB flash drive, but is not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices, or any combination thereof. More specific examples of computer-readable storage media may include, but are not limited to: electrical connections with one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In this embodiment, the computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, system, or device. The program code contained on the computer-readable storage medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (Radio Frequency), etc., or any suitable combination thereof.

[0087] The aforementioned computer-readable storage medium may be contained in the cloud or a mobile device; or it may exist independently without being assembled into the cloud or a mobile device.

[0088] The aforementioned computer-readable storage medium carries one or more programs. When these programs are executed by the cloud or a mobile terminal, the cloud executes the following: receiving interface generation instruction text sent by the mobile terminal, wherein the interface generation instruction text is generated by the mobile terminal based on a first collected voice signal or a first original text; inputting the interface generation instruction text into a pre-trained large language model, and having the large language model output a corresponding structured instruction set and a visual requirement description text; querying the corresponding standardized dataset in the HMI terminal according to each structured instruction in the structured instruction set; generating a first HMI interface based on the standardized dataset, the visual requirement description text, and preset interface generation rules; pushing the first HMI interface to the HMI terminal for loading and display, and simultaneously pushing the first HMI interface to the mobile terminal for remote visualization display.

[0089] The mobile device performs the following actions in response to the acquisition of a first speech signal or a first original text: preprocessing the first speech signal or the first original text to obtain a second speech signal or a second original text, wherein the preprocessing includes at least one of noise reduction processing, filtering processing, and sentence segmentation processing; converting the second speech signal or the second original text into interface generation instruction text; validating the interface generation instruction text based on preset verification rules; and after the interface generation instruction text passes the verification, sending the interface generation instruction text to the cloud so that the cloud can generate a first HMI interface based on the interface generation instruction text.

[0090] Computer program code for performing the operations of this disclosure can be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, and C++, and conventional procedural programming languages ​​such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0091] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0092] The modules described in the embodiments of this disclosure can be implemented in software or hardware. The names of the modules do not necessarily limit the functionality of the unit itself.

[0093] The readable storage medium provided in this application is a computer-readable storage medium that stores computer-readable program instructions for executing the above-described HMI interface generation method. This solves the technical problems of high configuration thresholds, high labor costs, and high time costs associated with traditional HMI configuration schemes. Compared with the prior art, the beneficial effects of the computer-readable storage medium provided in this application are the same as those of the HMI interface generation method provided in Embodiment 1 above, and will not be repeated here.

[0094] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the steps of the HMI interface generation method described above.

[0095] The computer program product provided in this application can solve the technical problems of high configuration threshold, high labor cost, and high time cost of traditional HMI configuration solutions. Compared with the prior art, the beneficial effects of the computer program product provided in this application are the same as the beneficial effects of the HMI interface generation method provided in Embodiment 1 above, and will not be repeated here.

[0096] The above are merely preferred embodiments of this application and do not limit the patent scope of this application. Any equivalent structural or procedural transformations made using the content of this application's specification and drawings, or direct or indirect applications in other related technical fields, are similarly included within the patent scope of this application.

Claims

1. A method for generating an HMI interface, characterized in that, When applied to the cloud, the HMI interface generation method includes: Receive interface generation instruction text sent by the mobile terminal, wherein the interface generation instruction text is generated by the mobile terminal based on the collected first voice signal or first original text; The interface generates instruction text which is then input into a pre-trained large language model. The large language model then outputs the corresponding structured instruction set and a visual description of the requirements. Based on each structured instruction in the structured instruction set, query the corresponding standardized dataset in the HMI terminal; Based on the standardized dataset, the visual requirement description text, and the preset interface generation rules, a first HMI interface is generated. The first HMI interface is pushed to the HMI terminal for loading and display, and the first HMI interface is simultaneously pushed to the mobile terminal for remote visualization display.

2. The HMI interface generation method as described in claim 1, characterized in that, The interface generation rules include at least the mapping relationship between data types and chart types; The step of generating the first HMI interface based on the standardized dataset, the visual requirement description text, and the preset interface generation rules includes: Identify the data types corresponding to each frame in the standardized dataset; Based on the data types of each of the aforementioned screen data, determine the corresponding chart type for each of the aforementioned screen data; Based on the chart types, the screen data, and the visualization requirement description text, a first HMI interface is generated.

3. The HMI interface generation method as described in claim 2, characterized in that, The step of determining the chart type corresponding to each of the screen data according to the respective data types includes: In response to the data type being a time-series continuous type, the chart type is determined to be a line chart; Since the data type is classified comparison, the chart type is determined to be a bar chart. In response to the data type being multi-dimensional percentage type, the chart type is determined to be a pie chart.

4. The HMI interface generation method as described in claim 1, characterized in that, After the step of generating the first HMI interface based on the standardized dataset, the visual requirement description text, and the preset interface generation rules, the method further includes: Based on the runtime configuration reload mechanism, the changed production data, data mapping relationships and / or page display information are dynamically replaced to obtain updated configuration data; In response to meeting preset update conditions or receiving a manual update command, the first HMI interface is updated according to the update configuration data to obtain the second HMI interface; The second HMI interface is pushed to the HMI terminal for loading and display, and then pushed to the mobile terminal for remote visualization.

5. The HMI interface generation method as described in claim 1, characterized in that, The HMI interface generation method further includes: In response to receiving the interface adjustment instruction text sent by the mobile terminal, the interface adjustment instruction text is input into the large language model, and the large language model outputs the corresponding structured adjustment instruction; The first HMI interface is adjusted based on the structured adjustment instructions to obtain the third HMI interface; The third HMI interface is pushed to the HMI terminal for loading and display, and then pushed to the mobile terminal for remote visualization.

6. A method for generating an HMI interface, characterized in that, When applied to mobile devices, the HMI interface generation method includes: In response to the acquisition of a first speech signal or a first original text, the first speech signal or the first original text is preprocessed to obtain a second speech signal or a second original text, wherein the preprocessing includes at least one of noise reduction processing, filtering processing and sentence segmentation processing; Convert the second voice signal or the second original text into interface generation instruction text; The interface-generated instruction text is validated based on preset validation rules; After the interface generation instruction text passes verification, the interface generation instruction text is sent to the cloud so that the cloud can generate the first HMI interface based on the interface generation instruction text.

7. The HMI interface generation method as described in claim 6, characterized in that, The HMI interface generation method further includes: In response to the acquisition of a third speech signal or a third original text, the third speech signal or the third original text is preprocessed to obtain a fourth speech signal or a fourth original text. Convert the fourth voice signal or the fourth original text into interface adjustment instruction text; The interface adjustment instruction text is validated based on preset validation rules; After the interface adjustment instruction text passes verification, the interface adjustment instruction text is sent to the cloud so that the cloud can adjust the first HMI interface according to the interface adjustment instruction text.

8. A cloud platform, characterized in that, The cloud includes: At least one processor; and, A memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the steps of the HMI interface generation method as described in any one of claims 1 to 5.

9. A mobile terminal, characterized in that, The mobile device includes: At least one processor; and, A memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the steps of the HMI interface generation method as described in any one of claims 6 to 7.

10. An HMI terminal, characterized in that, The HMI terminal includes: At least one processor; and, A memory that is communicatively connected to the at least one processor; The memory stores instructions that can be executed by the at least one processor. These instructions are executed by the at least one processor to enable the at least one processor to perform the following actions: providing a standardized dataset to the cloud in response to a query request from the cloud, so that the cloud can generate an HMI interface; receiving and loading the HMI interface pushed by the cloud for visualization on the HMI terminal; and dynamically loading the runtime configuration data in response to receiving updated configuration data to obtain an updated HMI interface.

11. A readable storage medium, characterized in that, The readable storage medium is a computer-readable storage medium, on which a program implementing the HMI interface generation method is stored. The program implementing the HMI interface generation method is executed by a processor to implement the steps of the HMI interface generation method as described in any one of claims 1 to 5 or 6 to 7.