User interface generation method and device, electronic equipment and vehicle

Abstract

The invention discloses a user interface generation method and device, electronic equipment and a vehicle, and relates to the technical field of data processing. The method comprises the following steps: acquiring control input information of a user to a cabin system; generating language description information for describing a user interface according to the control input information, the driving environment information and the historical interaction behavior information; and based on the interface description information, processing to obtain the user interface. Compared with the prior art, the method has the advantages that the interface description information is analyzed through the control input information of the user in combination with the driving environment information and the historical interaction behavior information; and the user interface which is more adaptive to the interaction required by the user can be generated by combining the interface description information. Therefore, the situation that in an existing set user interface, a user can determine a wanted instruction only through multiple times of touch control is avoided, and the problems that in an existing cabin interaction mode, interaction is inconvenient, and the interaction efficiency is low are solved.

Description

Technical Field

[0001] The present application relates to the field of data processing technology, and in particular to a method, device, electronic device and vehicle for generating a user interface. Background Art

[0002] The interactive process of the in-car cockpit configuration system is largely similar to the interactive method of mobile phones and tablets. Users can interact with vehicle functions through the user interface on the car screen.

[0003] Currently, the layout of the user interface, functional content, and switching logic of different pages are all pre-defined fixed patterns. In actual use, there are cases where the functions are too hidden and require users to operate multiple times, which affects the efficiency and experience of users using the cockpit system. Summary of the invention

[0004] In view of this, the present application provides a method, device, electronic device and vehicle for generating a user interface, which can solve the problems of inconvenient interaction and low interaction efficiency in the existing method of interacting through a user interface.

[0005] In a first aspect, the present application provides a method for generating a user interface, comprising:

[0006] Obtain user input information on cockpit system control;

[0007] Generate interface description information for describing a user interface according to the control input information, driving environment information, and historical interaction behavior information;

[0008] Based on the interface description information, a user interface is obtained by processing.

[0009] In a second aspect, the present application provides a device for generating a user interface, comprising:

[0010] An acquisition unit is configured to acquire user control input information of the cockpit system;

[0011] an analysis unit, configured to generate interface description information for describing a user interface according to the control input information, the driving environment information and the historical interaction behavior information;

[0012] The generating unit is configured to process and obtain a user interface based on the interface description information.

[0013] In a third aspect, the present application provides another method for generating a user interface, comprising:

[0014] Get the user's control input information;

[0015] Analyze the control input information to generate interface description information of the user interface;

[0016] Based on the interface description information, a user interface is obtained by processing.

[0017] In a fourth aspect, the present application provides another user interface generation device, comprising:

[0018] An acquisition unit, configured to acquire user's manipulation input information;

[0019] An analysis unit, configured to analyze the manipulation input information and generate interface description information of the user interface;

[0020] The generating unit is configured to process and obtain a user interface based on the interface description information.

[0021] In a fifth aspect, the present application provides a computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the method for generating a user interface described in the first aspect is implemented.

[0022] In a sixth aspect, the present application provides a computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the method for generating a user interface described in the third aspect is implemented.

[0023] In a seventh aspect, the present application provides an electronic device, comprising a storage medium, a processor, and a computer program stored on the storage medium and executable on the processor, wherein the processor implements the method for generating a user interface described in the first aspect when executing the computer program.

[0024] In an eighth aspect, the present application provides an electronic device, comprising a storage medium, a processor, and a computer program stored on the storage medium and executable on the processor, wherein the processor implements the method for generating a user interface described in the third aspect when executing the computer program.

[0025] In a ninth aspect, the present application provides a vehicle, comprising a device for generating a user interface as mentioned in the second aspect or an electronic device as mentioned in the seventh aspect.

[0026] In a tenth aspect, the present application provides a vehicle, comprising a user interface generating device as mentioned in the fourth aspect or an electronic device as mentioned in the eighth aspect.

[0027] By means of the above technical scheme, the present application provides a method, device, electronic device and vehicle for generating a user interface, which first obtains the user's control input information of the cockpit system, analyzes the control input information in combination with the driving environment information and the historical interactive behavior information, and generates interface description information for describing the user interface. The interface description information is processed to obtain the user interface. Compared with the related art, the present application analyzes the interface description information through the user's control input information in combination with the driving environment information and the historical interactive behavior information. Combined with the interface description information, a user interface that is more adapted to the interaction required by the user can be generated. This avoids the situation in the existing user interface that the user needs to touch multiple times to determine the desired command, and solves the problems of inconvenient interaction and low interaction efficiency in the existing cockpit interaction method.

[0028] The above description is only an overview of the technical solution of the present application. In order to more clearly understand the technical means of the present application, it can be implemented in accordance with the contents of the specification. In order to make the above and other purposes, features and advantages of the present application more obvious and easy to understand, the specific implementation methods of the present application are listed below. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0030] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0031] Figure 1 A schematic diagram showing a flow chart of a method for generating a user interface provided in an embodiment of the present application;

[0032] Figure 2 A schematic diagram of a module framework of a user interface provided in an embodiment of the present application is shown;

[0033] Figure 3 A schematic diagram of a module framework of another user interface provided in an embodiment of the present application is shown;

[0034] Figure 4 A schematic diagram showing a flow chart of another method for generating a user interface provided in an embodiment of the present application;

[0035] Figure 5 A schematic diagram of a user interface provided in an embodiment of the present application is shown;

[0036] Figure 6A schematic diagram showing a flow chart of another method for generating a user interface provided in an embodiment of the present application;

[0037] Figure 7 A schematic diagram of the structure of a user interface generation device provided in an embodiment of the present application is shown;

[0038] Figure 8 A schematic diagram of the structure of another user interface generation device provided in an embodiment of the present application is shown. DETAILED DESCRIPTION

[0039] In order to be able to more clearly understand the above-mentioned purposes, features and advantages of the present application, the scheme of the present application will be further described below. It should be noted that, in the absence of conflict, the embodiments of the present application and the features in the embodiments can be combined with each other. In addition, in order to be able to understand the characteristics and technical content of the embodiments of the present disclosure in more detail, the implementation of the embodiments of the present disclosure is described in detail below in conjunction with the accompanying drawings, and the attached drawings are only for reference purposes and are not used to limit the embodiments of the present disclosure. In the following technical description, for the convenience of explanation, a full understanding of the disclosed embodiments is provided by multiple details. However, in the absence of these details, one or more embodiments can still be implemented. In other cases, to simplify the drawings, well-known structures and devices can be simplified for display.

[0040] In order to solve the problems of inconvenient interaction and low interaction efficiency in the existing cockpit interaction mode, this embodiment proposes a method for generating a user interface. Figure 1 As shown, the method includes:

[0041] S101, obtaining user control input information of the cockpit system.

[0042] Among them, the user's control input information for the cockpit system can include multiple methods. Specifically, the user can use one or a combination of voice, panel touch, line of sight movement, body movement, and gestures, such as voice combined with panel touch, voice combined with gesture recognition, etc.

[0043] The control input information may specifically be multimodal feature data, which may be fusion data such as language, image, audio, video, text, gesture, vision, touch, biometrics, voiceprint, face identification (FaceID), user line of sight, environmental information, etc.

[0044] S102: Generate interface description information for describing a user interface according to the control input information, the driving environment information, and the historical interaction behavior information.

[0045] Combined with the current driving environment information and the historical interaction behavior information, the interface description information of the user interface is analyzed from the control input information. The current driving environment information here includes weather, geographic location, temperature inside and outside the vehicle, current driving speed, etc. The historical interaction behavior information can be, for example, the control input information issued by the user to the vehicle within a preset period of time. For example, the user issued a command to turn on the air conditioner to the vehicle 10 minutes ago, which can be used as the historical interaction behavior information of this time.

[0046] The interface description information may be language description information used to describe the user interface, that is, the interface description information refers to the description language used to generate the corresponding interactive user interface, which is analyzed based on the user's control input information such as voice and touch input. For example, DSL language (Domain Specific Language) can directly generate an interactive interface based on input intent without acquisition, and is a way to mechanically describe the generated content; it can also be HTML language (Hyper Text Markup Language) used to describe web page content. In this embodiment, the language type of the interface description information can have multiple forms, and the language type is not specifically specified here, but it needs to ensure the ability to express the displayed content and process user operations.

[0047] S103, obtaining a user interface based on the interface description information.

[0048] This embodiment can process the interface description information through rendering and other methods to obtain a user interface, which can pay more attention to the interaction with the user and the accuracy of the intention recognition. For example, the user interface may include an interactive interface, in which the user can choose to experience corresponding functional services (such as listening to music, watching videos, navigating, playing games, ordering takeout, etc.), or set some functional options in the interactive interface, or perform instant communication with other clients in the interactive interface, etc.

[0049] For another example, the user interface may also include an AI dialogue interface and an interactive interface. Specifically, the right half of the generated user interface is an interactive interface generated based on the user's control input information combined with the driving environment information and historical interactive behavior information, and the left half also has an AI dialogue part for the user to confirm with the user whether the interactive interface currently generated on the right is consistent with the content that the user needs to interact with.

[0050] In addition, the user interface can be a channel for information exchange between people and computer devices. Users can input information to the computer through the user interface to operate, and the computer device provides information to the user through the interactive interface with people for reading, analysis and judgment. For example, it can be: user interface, direct operation user interface, multimedia user interface, voice user interface, gesture user interface, mixed reality (MR) interface, multi-channel user interface (comprehensively adopts new interactive channels, devices and interactive technologies such as sight, voice, gesture, etc., so that users can use multiple channels to conduct human-computer dialogue in a natural, parallel and collaborative way, and capture the user's interactive intention by integrating accurate and imprecise inputs from multiple channels, so as to improve the naturalness and efficiency of human-computer interaction), etc.

[0051] In this embodiment, the user's control input information for the cockpit system is first obtained, and the control input information is analyzed in combination with the driving environment information and the historical interaction behavior information to generate interface description information for describing the user interface. The interface description information is processed to obtain the user interface. Compared with the related art, the present application analyzes the interface description information through the user's control input information in combination with the driving environment information and the historical interaction behavior information. Combined with the interface description information, a user interface that is more adapted to the interaction required by the user can be generated. This avoids the situation in the existing user interface that the user needs to touch multiple times to determine the desired command, and solves the problems of inconvenient interaction and low interaction efficiency in the existing cockpit interaction method.

[0052] It should be noted that the current graphical user interface is the most information-rich interaction method, and its layout, functions, and switching logic of different pages are all pre-defined. Although manufacturers will optimize it based on the usage habits of most users, there will still be problems such as the user's voice input is not standard or the function is hidden too deeply, requiring multiple touch operations. The generative user interface provided in this embodiment is no longer pre-defined by the manufacturer, but the user's intention is obtained based on the user's input information through manipulation, combined with the current vehicle's environmental information and historical interaction information. Therefore, the interface description language is more closely related to the user's intention, thereby avoiding functions that are hidden too deeply and require users to touch multiple times to find them, and solving the problems of inconvenient interaction and low interaction efficiency in existing cockpit interaction methods.

[0053] Optionally, based on the interface description information, a user interface is processed, including: determining basic elements of the user interface, attributes of the basic elements and association rules between the basic elements by identifying the interface description information; and processing the user interface according to the basic elements, attributes of the basic elements and association rules between the basic elements.

[0054] In this embodiment, the interface description information is identified and parsed to determine the basic elements, attributes, and association rules between the basic elements in the final user interface. Then, based on the basic elements, the attributes of the basic elements, and the association rules between the basic elements, the user interface is generated. The basic elements here can be regarded as components, which are the basic units of display and processing. For example, the basic element e1 is a button, and the basic element e2 is a picture. The attributes can be exemplified by the width or length of the basic element e1, or the content of the picture, etc. The association rule can be an association relationship or a progressive relationship between multiple basic elements, or it can be a rule. The reliability of the user interface generation process can be guaranteed in the above manner.

[0055] Optionally, interface description information for describing the user interface is generated based on the control input information, driving environment information and historical interaction behavior information, including: identifying natural language text information corresponding to the control input information; combining the driving environment information and the historical interaction behavior information, analyzing the natural language text information through a large language model to generate interface description information.

[0056] In this embodiment, the control input information is first converted into corresponding natural language text information, and then the natural language text information, combined with the vehicle driving environment information and historical interactive behavior information, is input into the large language model to obtain the interface description information. The large language model is a deep learning model that has achieved many results in the field of natural language processing, such as text generation, machine translation, question-answering systems, etc. Among them, the most well-known are models like GPT and BERT, which have achieved high performance and effects in various natural language processing tasks. The control input information is recognized by the large language model, and the vehicle driving environment information and historical interactive behavior information are also introduced, which can make the recognition more accurate and more precise to understand the user's intention.

[0057] The large language model can be deployed on the vehicle side or in the cloud. If deployed on the cloud side, the natural language text information, driving environment information and historical interactive behavior information corresponding to the control input information can be sent to the cloud server, so that the cloud server can use the large language model to combine the driving environment information and historical interactive behavior information to identify and analyze the natural language text information, obtain the interface description information of the user interface, and then send it to the vehicle side.

[0058] For the training process of the large language model, a training set can be created in advance, and the training set may include different sample vehicle driving environment information and historical interaction behavior information, as well as the interface description information of the user interface selected by the user's control input at that time, etc., and the large language model is trained using the data in these training sets. The large language model can be used to identify the user's current operation input information in combination with the vehicle driving environment information and the historical interaction behavior information, and obtain the interface description information of the corresponding user interface. Subsequently, according to the user's selection operation on the user interface, it can be determined whether the generated user interface is the user interface required by the user, and then according to the user's corresponding user interface adjustment information (such as switching to other user interfaces, or interactive operations on the user interface, etc.), the large language model is updated and trained, and the updated large language model is used to serve the next application scenario. In addition, in the training process of the large language model, the corresponding recognition language can also be added for model training, such as DSL language or HTML language, so that the obtained large language model can output the information described by the language of the corresponding language type when used, so as to meet the needs of different business scenarios.

[0059] Optionally, natural language text information is analyzed through a large language model in combination with driving environment information and historical interaction behavior information to generate interface description information, including: combining driving environment information and historical interaction behavior information, natural language text information is analyzed through a large language model to generate interface description information and user intention information; accordingly, based on the interface description information, processing to obtain a user interface may include: based on the interface description information and the intention information, processing to obtain a user interface.

[0060] For this embodiment, during the training of the large language model, user intention recognition training can also be added, such as training the large language model based on different sample vehicle driving environment information and historical interaction behavior information, as well as the interface description information of the user interface selected by the user control input at that time, user intention information, etc. The large language model can not only recognize the interface description information of the user interface, but also be used to recognize the user's intention information. Subsequently, it can be determined whether the user intention recognition result of the large language model is correct by asking questions, and then the corresponding large language model update training can be performed, so that the large language model can accurately recognize the user's intention information.

[0061] In this embodiment, the natural language text information is analyzed by a large language model in combination with the driving environment information and the historical interaction behavior information to obtain the interface description information and the user's intention information. The left half mentioned above is the AI ​​area for dialogue with the user, and the right half is based on the interface description information to generate an interface for interaction with the user. The intention information is the content used to generate the AI ​​area in the left half for dialogue with the user. Therefore, based on the interface description information, the user interface is processed, and it also includes: based on the interface description information and the intention information, the user interface is processed. That is, the AI ​​area in the left half can be generated according to the intention information, and the interface description information is used to generate the interactive interface in the right half. Thereby making the interface more in line with the user's interaction intention.

[0062] Optionally, natural language text information is analyzed through a large language model in combination with driving environment information and historical interactive behavior information to generate interface description information, including: natural language text information is analyzed through a large language model in combination with driving environment information and historical interactive behavior information to generate interface description information and operation command information, and the operation command information is used to control vehicle functions.

[0063] For this embodiment, during the training of the large language model, user operation command recognition training can also be added, such as training the large language model based on different sample vehicle driving environment information and historical interaction behavior information, as well as the interface description information of the user interface selected by the user control input at that time, user operation command information, etc. The large language model can not only recognize the interface description information of the user interface, but also be used to recognize the user's operation command information. Subsequently, based on the user feedback of the operation action corresponding to the operation command, it can be determined whether the user operation command recognition result of the large language model is correct, and then the corresponding large language model update training can be performed, so that the large language model can accurately recognize the user's operation command information.

[0064] In this embodiment, the large language model can also directly identify and determine the operation command information, so as to directly control the vehicle functions according to the operation command information, which can more directly meet the needs of users.

[0065] Optionally, in the above embodiment, the method of this embodiment may also include: performing a command check on the legality and / or security of the operation command information in combination with the driving environment information and the user's function usage permission information; if the operation command information passes the command check, triggering the execution of the operation command information.

[0066] In this embodiment, it is also mentioned that the legality and / or security of the operation command information is checked, for example, whether the generated command meets the current safety requirements, whether the vehicle can perform the operation during driving, etc., so as to improve safety and reliability.

[0067] Optionally, if the execution of the operation command information requires the user's confirmation, then an inquiry message of whether to execute the operation command information is output; and upon receiving the user's confirmation execution message, the execution of the operation command information is triggered.

[0068] In this embodiment, some operation commands also require user confirmation. When user confirmation is required, a query message is output to the user asking whether to execute the operation command information, and the operation command is executed after the user confirms, thereby improving security and interactivity with the user.

[0069] In some embodiments, combined with the above-mentioned description process, a large language model for identifying user intentions, user operation commands, and interface description information of the user interface can be comprehensively trained. Subsequently, the large language model can be updated and trained based on the corresponding feedback of each model recognition result to continuously improve the information recognition accuracy of the large language model.

[0070] As an optional method, processing the user interface based on the interface description information in S103 may specifically include: performing content verification on the legality and / or security of the interface content described by the interface description information; if the interface content passes the content verification, processing the user interface based on the interface description information.

[0071] In this embodiment, it is also mentioned that the legality and / or security of the generated interface content needs to be verified, for example, whether the code in the generated interface description language contains dangerous information, etc., so as to improve the security of the entire user interface generation process.

[0072] Optionally, the legality and / or security of the interface content described in the interface description information is verified, including: combining driving environment information and user function usage permission information to verify the legality and / or security of the interface content described in the interface description information.

[0073] In this embodiment, the verification of legality and / or safety also includes verifying whether the instruction can be currently executed in combination with the driving environment information. For example, during driving, the user's instruction is recognized as opening the door, which is obviously unreasonable and should not be executed. Therefore, such content needs to be verified to improve safety and avoid misjudgment.

[0074] Optionally, based on the interface description information, processing to obtain a user interface includes: generating different user interfaces based on at least one interface content described corresponding to the interface description information, wherein the different user interfaces are used to be displayed on different display devices respectively.

[0075] In this embodiment, multiple interface information may be generated because there may be multiple display devices in the vehicle cabin, and different display devices can be used to display different content, thereby improving the applicability and convenience of the solution. The display device here can be a display screen or other device as long as it can generate an interactive interface.

[0076] The following will further illustrate the solution of this embodiment in conjunction with an actual application schematic diagram of the module framework. Figure 2 As shown, a schematic diagram of a module framework of a user interface provided in an embodiment of the present application is shown.

[0077] First, there can be one or more input modules, which are used to process multi-modal control input information, including touch, gesture, voice, sight, text, etc., which can be processed by touch processing module, gesture processing module, voice processing module, and text processing module. Among them, some inputs (such as touch mode) can be directly converted into operation commands and input to the command processing module for processing; some inputs (such as voice mode) can be converted into natural language text information and input to the large language model, so as to recognize the interface description language and intention information.

[0078] The environment perception module is used to obtain driving environment information such as vehicle speed, weather, geographic location, temperature inside the vehicle, and temperature inside and outside the vehicle. This helps the large language model to understand and respond to intent information more accurately.

[0079] The large language model mainly processes and recognizes the user's control input information, driving environment information and historical interactive behavior information. Specifically, it can directly generate corresponding operation command information based on the above information and hand it over to the command processing module for processing; it can also generate interface description information described in a specific language (such as DSL) and hand it over to the interface generation module for interface generation; it can also recognize the user's intention information and generate an AI dialogue area accordingly. In addition, it can also be any two or a combination of the above three.

[0080] The command processing module is used to respond to and process the operation command information, including checking whether the operation command is legal and safe, whether user confirmation is required, and the execution of the command.

[0081] The interface generation module responds to and processes the interface description information (which may also include intent information) described in a specific language and generates a corresponding user interface.

[0082] The display content processing module processes the generated user interface content. The processing here may include checking the legality and / or security of the interface content and rendering it on the display device.

[0083] At least one display device displays the display content, and a common display device is a touch screen, a liquid crystal display, etc.

[0084] The memory management module is used to record and provide historical interaction behavior information. It also has an update function, such as recording the previous interaction process and storing it as historical interaction behavior information for use by the large language model for next recognition.

[0085] The following is a typical user interface generation example. Figure 3 , showing a schematic diagram of a module framework of another user interface provided in an embodiment of the present application. Figure 3 correspond, Figure 4 A flow chart of another method for generating a user interface provided in an embodiment of the present application is shown. The method includes:

[0086] S401, obtaining user control input information of the cockpit system.

[0087] For example, the user speaks voice content A, and then the voice processing module processes the voice content A to obtain natural language content T.

[0088] S402, retrieve historical interaction behavior information and driving environment information, identify control input information through a large language model, and generate interface description information.

[0089] The large language model generates an interface description language Ls for the natural language content T, combined with the driving environment information E and the historical interaction behavior information H2, and then can be sent to the interface generation module to generate a user interface. In addition, in this step, the intent information and one or more operation commands can also be identified. The intent information can be sent to the interface generation module to generate a regional interface for AI dialogue. The operation command will be passed to the command processing module for processing (for example, to check the legality and / or security of the instruction, confirm with the user, etc.).

[0090] S403: Generate a corresponding user interface according to the interface description information.

[0091] The interface generation module generates an interface description language Ls based on the large language model and performs operations such as rendering on it to generate a corresponding user interface.

[0092] S404: Perform a legality and / or security check on the interface content of the user interface.

[0093] In addition to the above-mentioned verification of the legality and security of the content, the interface generation module can also verify whether the code in the generated interface description language contains dangerous information, and can also verify whether there are illegal elements or attributes, whether the attribute value is reasonable, and whether the interaction logic of the element can be processed normally. At the same time, a new historical interaction behavior information will be added to the memory management module.

[0094] Furthermore, the interface generation module is mainly responsible for processing the interface description language generated by the large language model, including:

[0095] 1. Verify the legality of the content.

[0096] 2. Parse and layout the page elements, and generate display content for the display content processing module;

[0097] 3. Extraction of page element interaction logic: When users interact with page elements, process the interaction logic described by the Ls language.

[0098] S405: Display the verified user interface on a display device.

[0099] An interactive interface D is generated according to the interface description language Ls and displayed on a display device. Optionally, multiple different interactive interfaces may be generated and displayed on different display devices.

[0100] Regarding generating an interactive interface D according to the interface description language Ls, an example is given, such as Figure 5 As shown. The generated interactive interface D consists of the following contents: elements e1, e2, e3 and rules r1, r2. Among them, e1 has an attribute a1, whose value is v1, expressed as e1(a1:v1); similarly, element e2 is expressed as e2(a2:v2); element e3 has two attributes, expressed as e3(a3:v3, a4:v4). Rule r1 has an attribute m1, whose value is n1; r2 has an attribute m2, whose value is n2.

[0101] When generating the corresponding user interface according to the interface description language Ls, taking the above content as an example, for example, according to the definition of Ls: e1 can be displayed as a text title, and the text content is specified by the value v1 of the attribute a1. E2 can be displayed as a text label, and the text content is defined by the value v2 of the attribute a2. E3 can be displayed as a button, and the text of the button is defined by the value v3 of the attribute a3, and the value v4 of the attribute a4 defines the operation instruction to be executed when the user clicks e3.

[0102] Furthermore, rule r1 indicates that e2 and e3 are grouped as g1 and laid out horizontally, and the value n1 of attribute m1 indicates the total width of e2 and e3. Rule r2 indicates that group g1 and element e1 defined by r1 are treated as group g2 and laid out vertically, and the value n2 of attribute m2 indicates the total height of g1 and e1.

[0103] Thus, a mapping relationship (e3, a4)-v4 is maintained, indicating that if the e3 element is clicked by the user, the interactive content described by v4 is executed. The generation of the interactive interface D is only an exemplary description of the elements in the interface and the relationship between the elements. The elements can be set according to the actual situation.

[0104] In addition, a legal operation set C can be set, for example, c1 represents turning on the air conditioner, and c2 represents opening the window. At the same time, attributes can be associated with c1, such as turning on the front air conditioner or the rear air conditioner, etc. In the actual implementation process, r1 / r2 / r3 can be implemented through different rules, or the same rule r can be implemented by adding specific attributes. The interface description language Ls is used to describe how users interact with the generated content, such as the operations that may be triggered when clicking an element.

[0105] The generative user interface in this embodiment is no longer predefined by the manufacturer, but is generated based on the user's input intention, combined with environmental information and historical information, using a large language model to generate the user interface. The interface content emphasizes content related to the user's intention, and the user does not need to click layer by layer to find the relevant function, but can directly access it. In addition, the generated user interface can continue to respond to the user's further operations and determine the content of the next operation accordingly.

[0106] By applying the solution of this embodiment, the interface description information is analyzed through the user's control input information combined with the driving environment information and the historical interaction behavior information. Then, a user interface that is more adapted to the interaction required by the user can be generated in combination with the interface description information. This avoids the situation in which the user needs to touch multiple times to determine the desired command in the existing set user interface, and solves the problems of inconvenient interaction and low interaction efficiency in the existing cockpit interaction mode.

[0107] In addition to the above-mentioned method for generating a user interface in a cockpit system, the present invention also provides another method for generating a user interface, which is not limited to cockpits, but can be applied to other systems, such as smart devices, robots, etc., and can also achieve the above-mentioned technical effects. Figure 6 As shown, including:

[0108] S601, obtaining user's control input information.

[0109] The control input information obtained here also includes multiple methods, not limited to one or several combinations of the above-mentioned voice, panel touch, and gesture recognition, but can also be combined with specific application environments, such as inputting control input information through a remote control.

[0110] S602: Analyze the control input information to generate interface description information of the user interface.

[0111] By analyzing the above control input information, interface description information for generating an intelligent device or robot interface is obtained.

[0112] S603, based on the interface description information, obtain a user interface through processing.

[0113] In this embodiment, the interface description information used to generate the user interface is first determined based on the user's control input information. The interface description information is then rendered and processed to obtain the corresponding user interface. In the specific application process, the technical details applied to the cockpit environment solution can be referred to to generate a user interface that is more adapted to the user's operation or interaction intention. This avoids the situation in which the user needs to touch multiple times in the existing set user interface to determine the desired command, thereby improving the interaction efficiency.

[0114] Optionally, based on the interface description information, a user interface is processed, including: determining basic elements of the user interface, attributes of the basic elements and association rules between the basic elements by identifying the interface description information; and processing the user interface according to the basic elements, attributes of the basic elements and association rules between the basic elements.

[0115] In this embodiment, the interface description information is identified and parsed to determine the basic elements, attributes, and association rules between the basic elements in the final user interface. Then, based on the basic elements, the attributes of the basic elements, and the association rules between the basic elements, the user interface is generated. The basic elements here can be regarded as components, which are the basic units of display and processing. For example, the basic element e1 is a button, and the basic element e2 is a picture. The attributes can be exemplified by the width or length of the basic element e1, or the content of the picture, etc. The association rule can be an association relationship or a progressive relationship between multiple basic elements, or it can be a rule. The reliability of the user interface generation process can be guaranteed in the above manner.

[0116] Optionally, the interface description information of the user interface is generated based on the analysis of the manipulation input information, including: identifying natural language text information corresponding to the manipulation input information; and analyzing the natural language text information through a large language model to generate the interface description information of the user interface.

[0117] In this embodiment, the control input information is first converted into corresponding natural language text information, and then the natural language text information is input into the large language model to obtain interface description information. Of course, it can also be combined with environment, historical interaction information, etc. to input into the large language model to improve recognition accuracy. The large language model (LLM, Large Language Model) is a natural language processing model based on deep learning, and has achieved many results in the field of natural language processing, such as text generation, machine translation, question-answering system, etc. For example, it can be an autoregressive language model such as GPT (Generative Pre-trained Transforme, generative pre-trained transformation model), a pre-trained model for natural language processing such as BERT (Bidirectional Encoder Representations from Transformers, bidirectional encoder representation technology based on transformers), etc., which have achieved high performance and effects in various natural language processing tasks. It can make recognition more accurate and more accurate in understanding user intentions.

[0118] Optionally, natural language text information is analyzed through a large language model to generate interface description information, including: inputting the natural language text information into the large language model for calculation; and using the calculated interface content style description information and interface element interaction description information corresponding to the semantic recognition results of the natural language text information as the interface description information.

[0119] In this embodiment, the interface description information is the interface content style description information and interface element interaction description information corresponding to the semantic recognition result of the natural language text information obtained by calculation after the natural language text information is input into the large language model. The interface content style description information and interface element interaction description information can be used to generate the corresponding user interface.

[0120] Optionally, natural language text information is analyzed through a large language model to obtain interface description information, including: analyzing natural language text information through a large language model to obtain interface description information and user intention information; based on the interface description information, processing to obtain a user interface, including: generating a user interface based on the interface description information and the intention information.

[0121] In this embodiment, the left half is the AI ​​zone for dialogue with the user, and the right half is a case of generating an interface for interaction with the user based on the interface description information. The intention information can be used as the basis for the AI ​​zone to confirm the interaction with the user, so that the interface is more in line with the user's interaction intention.

[0122] Optionally, the natural language text information is analyzed by a large language model to obtain interface description information, including: analyzing the natural language text information by a large language model to obtain interface description information and operation command information, wherein the operation command information is used to control the target function.

[0123] In this embodiment, the large language model can also directly identify and determine the operation command information, so as to directly control the target function according to the operation command information, which can more directly meet the needs of users.

[0124] Optionally, the above method further includes: performing a command check on the legality and / or security of the operation command information; if the operation command information passes the command check, triggering the execution of the operation command information.

[0125] In this embodiment, it is also mentioned that the legality and / or security of the operation command information is checked, for example, whether the generated command meets the current security requirements, so as to improve security and reliability.

[0126] Optionally, the method further includes: if the execution of the operation command information requires user confirmation, outputting inquiry information on whether to execute the operation command information; and triggering the execution of the operation command information when receiving confirmation execution information from the user.

[0127] In this embodiment, some operation commands also require user confirmation. When user confirmation is required, a query message is output to the user asking whether to execute the operation command information, and the operation command is executed after the user confirms, thereby improving security and interactivity with the user.

[0128] Optionally, based on the interface description information, a user interface is processed, including: performing content verification on the legality and / or security of the interface content described by the interface description information; if the interface content passes the content verification, the user interface is processed based on the interface description information.

[0129] In this embodiment, it is also mentioned that the legality and / or security of the generated interface content needs to be verified, for example, whether the code in the generated interface description language contains dangerous information, etc., so as to improve the security of the entire user interface generation process.

[0130] Optionally, based on the interface description information, processing to obtain a user interface includes: generating different user interfaces based on at least one interface content described corresponding to the interface description information, wherein the different user interfaces are used to be displayed on different display devices respectively.

[0131] In this embodiment, there may be different display devices in certain environments. For example, there may be multiple different display screens on a large robot. Therefore, multiple different user interfaces can be generated and displayed on different display screens, thereby improving the applicability and convenience of the solution.

[0132] Further, as Figures 1 to 5 The specific implementation of the method shown in the embodiment provides a device for generating a user interface, such as Figure 7 As shown, the device includes: an acquisition unit 701, an analysis unit 702, and a generation unit 703.

[0133] An acquisition unit 701 is configured to acquire user control input information of the cockpit system;

[0134] The analysis unit 702 is configured to generate interface description information for describing the user interface according to the control input information, the driving environment information and the historical interaction behavior information;

[0135] The generating unit 703 is configured to process and obtain a user interface based on the interface description information.

[0136] In a specific application scenario, the generation unit 703 is specifically configured to determine the basic elements of the user interface, the attributes of the basic elements, and the association rules between the basic elements by identifying the interface description information; and to process the user interface according to the basic elements, the attributes of the basic elements, and the association rules between the basic elements.

[0137] In a specific application scenario, the analysis unit 702 is further configured to identify the natural language text information corresponding to the control input information; combine the driving environment information and the historical interaction behavior information, analyze the natural language text information through a large language model, and generate the interface description information.

[0138] In a specific application scenario, the analysis unit 702 is further configured to combine the driving environment information and the historical interaction behavior information, analyze the natural language text information through a large language model, and generate the interface description information and the user's intention information; the processing based on the interface description information to obtain the user interface includes: processing based on the interface description information and the intention information to obtain the user interface.

[0139] In a specific application scenario, the analysis unit 702 is further configured to combine the driving environment information and the historical interaction behavior information, analyze the natural language text information through a large language model, and generate the interface description information and operation command information, wherein the operation command information is used to control vehicle functions.

[0140] In a specific application scenario, the analysis unit 702 is further configured to perform a command check on the legality and / or security of the operation command information in combination with the driving environment information and the user's function usage permission information; if the operation command information passes the command check, the execution of the operation command information is triggered.

[0141] In a specific application scenario, the analysis unit 702 is further configured to output inquiry information on whether to execute the operation command information if the execution of the operation command information requires user confirmation; and trigger the execution of the operation command information when receiving the user's confirmation execution information.

[0142] In a specific application scenario, the generation unit 703 is further configured to perform content verification on the legality and / or security of the interface content described by the interface description information; if the interface content passes the content verification, the user interface is processed based on the interface description information.

[0143] In a specific application scenario, the generation unit 703 is further configured to perform a content verification of the legality and / or security of the interface content described by the interface description information in combination with the driving environment information and the function usage permission information of the user.

[0144] In a specific application scenario, the generating unit 703 is further configured to generate different user interfaces based on at least one interface content described corresponding to the interface description information, and the different user interfaces are used to be displayed on different display devices respectively.

[0145] It should be noted that for other corresponding descriptions of the functional units involved in the user interface generation device provided in this embodiment, please refer to Figures 1 to 5 The corresponding description in will not be repeated here.

[0146] Further, as Figure 6 The specific implementation of the method shown in the figure, this embodiment provides another user interface generation device, such as Figure 8 As shown, the device includes: an acquisition unit 801, an analysis unit 802, and a generation unit 803.

[0147] The acquisition unit 801 is configured to acquire the user's control input information;

[0148] The analysis unit 802 is configured to analyze the manipulation input information and generate interface description information of the user interface;

[0149] The generating unit 803 is configured to process and obtain a user interface based on the interface description information.

[0150] In a specific application scenario, the generation unit 803 is specifically configured to determine the basic elements of the user interface, the attributes of the basic elements, and the association rules between the basic elements by identifying the interface description information; and to process the user interface according to the basic elements, the attributes of the basic elements, and the association rules between the basic elements.

[0151] In a specific application scenario, the analysis unit 802 is specifically configured to identify the natural language text information corresponding to the manipulation input information; analyze the natural language text information through a large language model to generate the interface description information.

[0152] In a specific application scenario, the analysis unit 802 is further configured to input the natural language text information into the large language model for calculation; and use the calculated interface content style description information and interface element interaction description information corresponding to the semantic recognition result of the natural language text information as the interface description information.

[0153] In a specific application scenario, the analysis unit 802 is further configured to analyze the natural language text information through a large language model to generate the interface description information and the user's intention information; the processing based on the interface description information to obtain the user interface includes: processing based on the interface description information and the intention information to obtain the user interface.

[0154] In a specific application scenario, the analysis unit 802 is further configured to analyze the natural language text information through a large language model to generate the interface description information and operation command information, where the operation command information is used to control the target function.

[0155] In a specific application scenario, the generation unit 803 is further configured to perform a command check on the legality and / or security of the operation command information; if the operation command information passes the command check, the execution of the operation command information is triggered.

[0156] In a specific application scenario, the generation unit 803 is further configured to output inquiry information on whether to execute the operation command information if the execution of the operation command information requires user confirmation; and trigger the execution of the operation command information when receiving the user's confirmation execution information.

[0157] In a specific application scenario, the generation unit 803 is further configured to perform content verification on the legality and / or security of the interface content described by the interface description information; if the interface content passes the content verification, the user interface is processed based on the interface description information.

[0158] In a specific application scenario, the generating unit 803 is further configured to generate different user interfaces based on at least one interface content described corresponding to the interface description information, and the different user interfaces are used to be displayed on different display devices respectively.

[0159] It should be noted that for other corresponding descriptions of the functional units involved in the user interface generation device provided in this embodiment, please refer to Figures 1 to 5 The corresponding description in will not be repeated here.

[0160] Based on the above Figures 1 to 6 The method shown in the embodiment also provides a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the above-mentioned Figures 1 to 6 The method shown.

[0161] Based on this understanding, the technical solution of the present application can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-RO5, a USB flash drive, a mobile hard disk, etc.), and includes a number of instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods of various implementation scenarios of the present application.

[0162] Based on the above Figures 1 to 6 The method shown, and Figure 7 and Figure 8 In order to achieve the above-mentioned purpose, the embodiment of the present application further provides an electronic device, which can be configured on a computer side or a vehicle side, etc. The device includes a storage medium and a processor; the storage medium is used to store a computer program; the processor is used to execute the computer program to achieve the above-mentioned Figures 1 to 6 The method shown.

[0163] Optionally, the above-mentioned physical device may also include a user interface, a network interface, a camera, a radio frequency (RF) circuit, a sensor, an audio circuit, a WI-FI module, etc. The user interface may include a display, an input unit such as a keyboard, etc., and the optional user interface may also include a USB interface, a card reader interface, etc. The network interface may optionally include a standard wired interface, a wireless interface (such as a WI-FI interface), etc.

[0164] Those skilled in the art will appreciate that the above-mentioned physical device structure provided in this embodiment does not constitute a limitation on the physical device, and may include more or fewer components, or a combination of certain components, or different arrangements of components.

[0165] The storage medium may also include an operating system and a network communication module. The operating system is a program that manages the hardware and software resources of the above-mentioned physical device, and supports the operation of the information processing program and other software and / or programs. The network communication module is used to realize the communication between the components inside the storage medium, and the communication with other hardware and software in the information processing physical device.

[0166] Through the description of the above implementation methods, those skilled in the art can clearly understand that the present application can be implemented by means of software plus a necessary general hardware platform, or by hardware. By applying the solution of this embodiment, user intent information is identified through the user's control input information combined with driving environment information and historical interaction behavior information. The user intent information is then combined with the interface description information to determine the interface description information that is more relevant to it, and then the corresponding graphical user interface is generated through the interface description information. This avoids the situation in which the user needs to touch multiple times in the existing graphical user interface to determine the desired command, and solves the problems of inconvenient interaction and low interaction efficiency in the existing cockpit interaction method.

[0167] It should be noted that, in this article, relational terms such as "first" and "second" are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the term "comprises" or any other variant thereof is intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0168] The above description is only a specific implementation of the present application, so that those skilled in the art can understand or implement the present application. Various modifications to these embodiments will be obvious to those skilled in the art, and the general principles defined herein can be implemented in other embodiments without departing from the spirit or scope of the present application. Therefore, the present application will not be limited to the embodiments described herein, but will conform to the widest scope consistent with the principles and novel features of the present application.

[0169] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software may depend on the specific application and design constraints of the technical solution. The technicians may use different methods for each specific application to implement the described functions, but such implementations should not be considered to exceed the scope of the embodiments of the present disclosure. The technicians may clearly understand that, for the convenience and simplicity of description, the specific working processes of the systems, devices and units described above may refer to the corresponding processes in the aforementioned method embodiments, and will not be repeated here.

[0170] In the embodiments disclosed herein, the disclosed methods and products (including but not limited to devices, equipment, etc.) can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units can be only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the coupling or direct coupling or communication connection between each other shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms. The units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the units may be selected according to actual needs to implement this embodiment. In addition, each functional unit in the embodiment of the present disclosure may be integrated in a processing unit, or each unit may exist physically alone, or two or more units may be integrated in one unit.

[0171] The flowchart and block diagram in the accompanying drawings show the possible architecture, function and operation of the system, method and computer program product according to the embodiment of the present disclosure. In this regard, each box in the flowchart or block diagram can represent a module, a program segment or a part of the code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. In some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two consecutive boxes can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, which can depend on the functions involved. In the description corresponding to the flowchart and the block diagram in the accompanying drawings, the operations or steps corresponding to different boxes can also occur in a different order from the order disclosed in the description, and sometimes there is no specific order between different operations or steps. For example, two consecutive operations or steps can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, which can depend on the functions involved. Each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be implemented by a dedicated hardware-based system that performs the specified functions or actions, or may be implemented by a combination of dedicated hardware and computer instructions.

Claims

1. A method for generating a user interface, It is characterized in that include: Obtain user input information on cockpit system control; Generate interface description information for describing a user interface according to the control input information, driving environment information, and historical interaction behavior information; Based on the interface description information, a user interface is obtained by processing.

2. The method according to claim 1, It is characterized in that The step of obtaining a user interface based on the interface description information includes: Determine basic elements of the user interface, attributes of the basic elements, and association rules between the basic elements by identifying the interface description information; The user interface is obtained by processing according to the basic elements, the attributes of the basic elements and the association rules between the basic elements.

3. The method according to claim 1, It is characterized in that The generating of interface description information for describing the user interface according to the control input information, the driving environment information and the historical interaction behavior information includes: Identifying natural language text information corresponding to the manipulation input information; The driving environment information and the historical interaction behavior information are combined, and the natural language text information is analyzed through a large language model to generate the interface description information.

4. The method according to claim 3, It is characterized in that The combining the driving environment information and the historical interaction behavior information, analyzing the natural language text information through a large language model, and generating the interface description information includes: In combination with the driving environment information and the historical interaction behavior information, the natural language text information is analyzed by a large language model to generate the interface description information and the user's intention information; The step of obtaining a user interface based on the interface description information includes: Based on the interface description information and the intention information, the user interface is obtained by processing.

5. The method according to claim 3, It is characterized in that The combining the driving environment information and the historical interaction behavior information, analyzing the natural language text information through a large language model, and generating the interface description information includes: In combination with the driving environment information and the historical interaction behavior information, the natural language text information is analyzed through a large language model to generate the interface description information and operation command information, and the operation command information is used to control vehicle functions.

6. The method according to claim 5, It is characterized in that The method further comprises: Combining the driving environment information and the function usage permission information of the user, performing a command check on the legality and / or security of the operation command information; If the operation command information passes the command verification, the operation command information is triggered to be executed.

7. The method according to claim 5, It is characterized in that The method further comprises: If the execution of the operation command information requires the user's confirmation, then outputting a query message of whether to execute the operation command information; When the user's confirmation execution information is received, the execution of the operation command information is triggered.

8. The method according to claim 1, It is characterized in that The step of obtaining a user interface based on the interface description information includes: Performing content verification on the legality and / or security of the interface content described by the interface description information; If the interface content passes the content verification, the user interface is obtained by processing based on the interface description information.

9. The method according to claim 8, It is characterized in that The content verification of the legality and / or security of the interface content described by the interface description information includes: In combination with the driving environment information and the function usage permission information of the user, the legality and / or security of the interface content described by the interface description information is verified.

10. The method according to claim 1, It is characterized in that The step of obtaining a user interface based on the interface description information includes: Based on at least one interface content described corresponding to the interface description information, different user interfaces are generated, and the different user interfaces are used to be displayed on different display devices respectively.

11. A method for generating a user interface, It is characterized in that include: Get the user's control input information; Analyze the control input information to generate interface description information of the user interface; Based on the interface description information, a user interface is obtained by processing.

12. The method according to claim 11, It is characterized in that The step of obtaining a user interface based on the interface description information includes: Determine basic elements of the user interface, attributes of the basic elements, and association rules between the basic elements by identifying the interface description information; The user interface is obtained by processing according to the basic elements, the attributes of the basic elements and the association rules between the basic elements.

13. The method according to claim 11, It is characterized in that The step of analyzing the manipulation input information to generate interface description information of the user interface includes: Identifying natural language text information corresponding to the manipulation input information; The natural language text information is analyzed by a large language model to generate the interface description information.

14. The method according to claim 13, It is characterized in that The step of analyzing the natural language text information by using a large language model to generate the interface description information includes: Inputting the natural language text information into the large language model for calculation; The interface content style description information and the interface element interaction description information corresponding to the semantic recognition result of the natural language text information obtained by calculation are used as the interface description information.

15. The method according to claim 13, It is characterized in that The step of analyzing the natural language text information by using a large language model to generate the interface description information includes: Analyzing the natural language text information through a large language model to generate the interface description information and the user's intention information; The step of obtaining a user interface based on the interface description information includes: Based on the interface description information and the intention information, the user interface is obtained by processing.

16. The method according to claim 13, It is characterized in that The step of analyzing the natural language text information by using a large language model to generate the interface description information includes: The natural language text information is analyzed by a large language model to generate the interface description information and operation command information, and the operation command information is used to control the target function.

17. The method according to claim 16, It is characterized in that The method further comprises: Performing a command check on the legality and / or security of the operation command information; If the operation command information passes the command verification, the operation command information is triggered to be executed.

18. The method according to claim 16, It is characterized in that The method further comprises: If the execution of the operation command information requires the user's confirmation, then outputting a query message of whether to execute the operation command information; When the user's confirmation execution information is received, the execution of the operation command information is triggered.

19. The method according to claim 11, It is characterized in that The step of obtaining a user interface based on the interface description information includes: Performing content verification on the legality and / or security of the interface content described by the interface description information; If the interface content passes the content verification, the user interface is obtained by processing based on the interface description information.

20. The method according to claim 11, It is characterized in that The step of obtaining a user interface based on the interface description information includes: Based on at least one interface content described corresponding to the interface description information, different user interfaces are generated, and the different user interfaces are used to be displayed on different display devices respectively.

21. A device for generating a user interface, It is characterized in that include: An acquisition unit is configured to acquire user control input information of the cockpit system; an analysis unit, configured to generate interface description information for describing a user interface according to the control input information, the driving environment information and the historical interaction behavior information; The generating unit is configured to process and obtain a user interface based on the interface description information.

22. A device for generating a user interface, It is characterized in that include: An acquisition unit, configured to acquire user's manipulation input information; An analysis unit, configured to analyze the manipulation input information and generate interface description information of the user interface; The generating unit is configured to process and obtain a user interface based on the interface description information.

23. A computer-readable storage medium having a computer program stored thereon, It is characterized in that When the computer program is executed by a processor, the method according to any one of claims 1 to 20 is implemented.

24. An electronic device comprising a storage medium, a processor, and a computer program stored on the storage medium and executable on the processor, It is characterized in that When the processor executes the computer program, the method according to any one of claims 1 to 20 is implemented.

25. A vehicle, It is characterized in that include: The apparatus of claim 21 or claim 22, or the electronic device of claim 23.

Images