User interface optimization method and device based on large model, equipment and medium

Abstract

The invention relates to a user interface optimization method and device based on a large model, equipment and a medium. The optimization method comprises the following steps: obtaining user preferences; a knowledge base construction step; a retrieval step; a prompt word construction step; and an interface generation step. By dynamically adjusting the layout, style, dynamic effect, output characters, sound, vibration signals and other feedback display elements of the human-computer interface or the user interface element, the interface information display self-adaptive optimization aiming at the user cognitive state is realized. Optimization design is carried out for the automobile driver and a vehicle-mounted AR-HUD interface, the method can be used for maintaining the cognitive state of the driver in a stable interval, the cognitive load of the driver on surrounding reality and machine decision results is reduced, and meanwhile the perception ability of the driver on the environment situation is kept.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of artificial intelligence, in particular to a user interface optimization method and device based on a large model, equipment and medium. BACKGROUND

[0002] The display method of the existing man-machine interface and user interface is mostly based on static templates or rule matching, lacking adaptive dynamic display methods that can meet the changes of complex and diverse application scenarios and the personalized information needs of users in visual, auditory and tactile multi-modal modes. Taking the vehicle-mounted man-machine interface and user interface as an example, the adaptive display of the existing vehicle-mounted interface (such as various vehicle-machine interfaces and various head-up display interfaces) is mostly based on scene events to select a certain preset mode, and cannot dynamically display based on the driving experience and the state of the driver. With the development of intelligent auxiliary driving technology, the driving process gradually shifts to the center of the driving experience, and the lack of personalized display solutions cannot fully utilize visual, audio and tactile feedback to display information, which will not be conducive to reducing the cognitive load of the driver on the surrounding reality and machine decision results.

[0003] The existing interface adaptive display optimization methods mainly include optimization adjustment schemes based on rule libraries, Bandit optimization systems, machine learning and reinforcement learning, each having its own advantages and disadvantages. The adjustment scheme based on the rule library relies on the rule library developed by experts, and it is difficult to cover all interactive situations as the information system develops, and it is even more difficult to cope with new situations; the Bandit optimization system is a global optimization method relying on probability, but it is not suitable for long-term planning applications; machine learning and reinforcement learning rely on a large amount of training data, which is expensive.

[0004] The current large model theory and technology are increasingly mature and perfect, and have broad application prospects. After pre-training on a large number of general domain data sets, the large model has strong context understanding and text reasoning capabilities, and can be used to generate interface layout descriptions. However, the current large model lacks deep semantic capture ability and dynamic planning decision-making ability for interface description language, making it difficult to be directly applied to the field of interface layout optimization, and the existing technology needs to be improved. SUMMARY

[0005] Therefore, it is necessary to provide a user interface optimization method, device, equipment and medium based on a large model, which can dynamically adjust the layout, style, dynamic effect, output text and sound, vibration signal and other feedback display elements of the man-machine interface or user interface elements, and realize adaptive optimization of interface information display for the cognitive state of the user. The present method is optimized for car drivers and vehicle-mounted AR-HUD interfaces, and can be used to maintain the cognitive state of the driver in a stable interval and maintain the driver's perception ability of the environment situation.

[0006] According to a first aspect of the present application, a large model-based user interface optimization method is provided, characterized by comprising the following steps:

[0007] A user preference acquisition step, which predicts the preference degree of the user for different types of information on the interface according to the user historical interaction data, to obtain an information sequence with priority;

[0008] A knowledge base construction step, which constructs a knowledge base including the mapping relationship between the user state, the information sequence with priority, and the interface adjustment example;

[0009] A retrieval step, which takes the user state and the information sequence with priority as input, and retrieves related interface adjustment examples with high similarity in the knowledge base;

[0010] A prompt word construction step, which constructs and encapsulates the input of the large model, and adds the interface adjustment examples with high similarity retrieved from the knowledge base to the prompt words;

[0011] An interface generation step, which inputs the constructed prompt words into the large model, and outputs the adjusted interface code after inference of the large model.

[0012] In some embodiments, it further comprises a fine-tuning dataset construction step, which takes the user state and the information sequence with priority as input data, and takes the interface description adjusted by humans as output data, and takes the input data and the output data as question and answer pairs to construct a fine-tuning dataset;

[0013] A fine-tuning step, which fine-tunes the parameters of the pre-trained large model using the question and answer pairs in the fine-tuning dataset to obtain a large model more suitable for the interface adjustment field.

[0014] In some embodiments, the fine-tuning of the parameters of the pre-trained large model comprises partial parameter fine-tuning of the large model by a low-rank adaptation (LoRA) fine-tuning method.

[0015] In some embodiments, the knowledge base is constructed using a vector database, and the information in the knowledge base is stored in the vector database in the form of vectors;

[0016] The information in the knowledge base is divided into blocks by a segmentation algorithm, encoded, and converted into an array saved in the knowledge base.

[0017] In some embodiments, the retrieval of related interface adjustment examples with high similarity in the knowledge base comprises:

[0018] Obtaining the information sequence with priority and the user state, and converting the two inputs into word vector representations through word vector matching;

[0019] The user state and the priority information sequence stored in the knowledge base are calculated for similarity by using a word vector model and ranked, and the top K interface adjustment examples with the highest similarity are retrieved.

[0020] In some embodiments, the prompt word comprises: an instruction, an adjustment principle, an interface adjustment example, and an output guide.

[0021] The instruction is used to role-set the large model and give a task description and reasoning limit.

[0022] The adjustment principle comprises: a layout principle, an alignment principle, and a color principle.

[0023] The output guide is used to limit the output content and output format of the large model.

[0024] The prompt word further comprises API interface information.

[0025] In some embodiments, the method further comprises an evaluation step of evaluating the effectiveness of the layout by a reward model and user evaluation, and using the evaluation result for training of the large model.

[0026] According to a second aspect of the present application, a user interface optimization device based on a large model is provided, comprising:

[0027] A user preference acquisition module is configured to predict the preference degree of a user for different types of information on an interface according to user historical interaction data, and obtain a priority information sequence.

[0028] A knowledge base construction module is configured to construct a knowledge base, the knowledge base comprising a mapping relationship among a user state, a priority information sequence, and an interface adjustment example.

[0029] A retrieval module is configured to retrieve relevant interface adjustment examples with high similarity to input in the knowledge base, by taking the user state and the priority information sequence as the input.

[0030] A prompt word construction module is configured to construct and encapsulate the input of the large model with a prompt word, and add the interface adjustment examples with high similarity retrieved from the knowledge base to the prompt word.

[0031] An interface generation module is configured to input the constructed prompt word into the large model, and output an adjusted interface code after reasoning of the large model.

[0032] According to a third aspect of the present application, a computer device is provided, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the method of any one of the above embodiments when executing the computer program.

[0033] According to a fourth aspect of the present application, a computer readable storage medium is provided, having stored thereon a computer program which, when executed by a processor, implements the steps of any of the above described methods.

[0034] By implementing the above solutions of the present application, the following beneficial effects can be obtained:

[0035] 1. The usefulness and personalization of interface information feedback can be greatly improved.

[0036] 2. The user preference obtaining step can sort the information priority according to the user preference from the current information sequence to be displayed, and the large model takes the user state and the information sequence with display priority as input, considers the user state and the preference degree of different interface information in the layout generation process to generate a personalized display scheme, and adjusts the interface layout multiple times based on the user state.

[0037] 3. Due to the context understanding ability and text reasoning ability of the large model, the large model is used to adaptively optimize and adjust the interface layout according to the context or task information and the user state, including but not limited to visual interface, auditory interface and tactile interface, without writing complete matching rules at the beginning of interface design to control the man-machine interface or user interface display. BRIEF DESCRIPTION OF DRAWINGS

[0038] Figure 1 is a flowchart of some embodiments of the user interface optimization method based on a large model of the present application;

[0039] Figure 2 is a flowchart of some other embodiments of the user interface optimization method based on a large model of the present application;

[0040] Figure 3 is a flowchart of the interface adaptive display process based on retrieval enhancement generation in the present application;

[0041] Figure 4 is a structural schematic diagram of some embodiments of the user interface optimization device based on a large model of the present application;

[0042] Figure 5 is an internal structure diagram of a computer device for implementing some embodiments of the present application. DETAILED DESCRIPTION

[0043] The embodiments of the present application will be described in more detail below with reference to the drawings, in which embodiments of the present application are shown. However, the present application can be implemented in many different forms and should not be construed as being limited to the embodiments set forth herein.

[0044] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the disclosure. As used herein, the singular forms "a", "an" and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise. It will be further understood that the terms "comprises" and / or "comprising," when used in this specification, specify the presence of stated features, integers, steps, operations, elements, and / or components, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof.

[0045] Unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this disclosure belongs. The terminology used herein should be interpreted as having a meaning that is consistent with its meaning in the context of the specification and the relevant art and will not be interpreted in an idealized or overly formal sense unless expressly so defined herein.

[0046] User state: refers to the current cognitive, behavioral or physiological state of the driver, such as cognitive load, current driving task, emotional state, etc.

[0047] The head-up display technology was first used in the airborne scene, which can avoid the pilot looking down to check the instrument, thereby improving attention and task efficiency. In the late 1980s, automobile manufacturers introduced it to improve driving safety. HUD projects vehicle speed, navigation prompts and other information to the front of the line of sight through a projection device, so that the driver can more intuitively obtain driving information feedback. In recent years, with the development of augmented reality (AR) technology, AR and HUD technology are combined to form AR-HUD. AR-HUD uses augmented projection surface technology to fuse generated images with real-world lane lines, vehicles in front and other objects, providing more accurate navigation and visual interaction effects. With the development of natural human-computer interaction technology, the in-vehicle interface form is not limited to the graphical interface, but can be expanded through the sound interface and the tactile interface to form a multi-modal human-computer interface. With the development of intelligent assisted driving technology, the amount of information received and processed by the advanced driver assistance system (ADAS) has increased significantly. In the case of limited HUD interface capacity but more displayable information, it is crucial to provide the driver with the most critical information during driving and to adaptively optimize the display content of the HUD.

[0048] The goal of display optimization is to adjust the display effect and layout of the display components O = {o i |i = 0, 1, …, n} and o i The category t i , position (x i , y i ), size (w i , h i ), color c i, visibility v i and output sound, text, tactile vibration signal, etc. fields, the display form and content of the human-machine interface or user interface are changed by adjusting the component attributes, thereby improving the cognitive load level and situation awareness ability of the user.

[0049] Figure 1 A flowchart showing some embodiments of the large model-based user interface optimization method of the present application.

[0050] As Figure 1 shown, the method comprises:

[0051] A user preference acquisition step S102, according to the user historical interaction data, the preference degree of the user to different types of information on the interface is predicted, and a priority information sequence is obtained;

[0052] In some embodiments, the preference score of the user (such as the driver) to different types of display information on the HUD is predicted based on the historical interaction behavior of the user. The information sequence S is the information processed by the ADAS and related systems under the current driving situation, which needs to be presented to the driver on the AR-HUD. The preference score of different types of information needs to be obtained through user interaction preference extraction, and then sorted according to the preference score to obtain the information sequence S' containing priority to be displayed, so as to facilitate the large model to adaptively adjust the HUD layout and display in combination with the user interaction preference.

[0053] For example, the current driving situation is "highway driving", and the ADAS and related systems identify the following information to be displayed: navigation, vehicle distance warning, weather, vehicle speed, music. Historical data shows that the user often pays attention to vehicle distance warning and navigation in high-speed state, and the system scores as follows: vehicle distance warning: 0.9, navigation: 0.85, vehicle speed: 0.6, weather: 0.3, music: 0.1. The sorted priority information sequence S' = [vehicle distance warning, navigation, vehicle speed, weather, music].

[0054] A knowledge base construction step S104, a knowledge base is constructed, the knowledge base includes the mapping relationship between the user state, the priority information sequence and the interface adjustment example;

[0055] In some embodiments, the knowledge base is constructed, and the creation of the knowledge base can be through the following key information, such as the grid layout structure rules of the interface description file, the component type, the component name and the field describing the component, etc., the user state and the priority information sequence S', the mapping relationship between the adjusted interface layout, and the adjustment and optimization principles of the interface layout under different user states. The knowledge base is constructed by using a vector database, and the related information is stored in the vector database in the form of a vector.

[0056] In some embodiments, the knowledge base can further include API interfaces, adjustment principles, and in this case, there are mainly three types of information in the knowledge base:

[0057] (1) API interface, mainly including the function and calling method of the function for changing the interface;

[0058] (2) Adjustment principle, mainly including adjustment principles such as alignment, overlap, color and relevance of interface elements;

[0059] (3) Interface adjustment example, mainly including adjustment strategies for the interface under different conditions.

[0060] According to the segmentation algorithm, the text information is divided into uniform blocks according to a certain length, encoded using the embedding model OpenAIEmbeddings, and the corresponding text representation embedding is obtained, which is converted into an array and saved in the knowledge vector library Milvus.

[0061] In the retrieval step S106, the user state and the priority information sequence are input, and the relevant interface adjustment examples with high similarity to the input are retrieved in the knowledge base;

[0062] In some embodiments, the vector database-based large model retrieval enhancement system is connected to the local vector database, the user input is transmitted into the embedding model, the knowledge package with the highest relevance to the user input is encapsulated in the prompt word in the knowledge base, and the context knowledge is provided to the large model; Specifically, the display information sequence S' with priority and the encoded user (such as the driver) state are first converted into word vector representations through word vector matching, and the state description and priority information sequence stored in the knowledge base are calculated and sorted by cosine similarity using the word vector model. Retrieve the top-k smallest metric distance examples from the knowledge base, then encapsulate the retrieved results as adjustment examples in the prompt word of the large model as examples for the large model inference. In addition, the API interface and adjustment principle content in the knowledge base are added as context to the prompt word to avoid forgetting of the large model, so as to facilitate the large model inference and content generation.

[0063] In the prompt word construction step S108, the input of the large model is constructed and encapsulated in the prompt word, and the interface adjustment examples with high similarity retrieved from the knowledge base are added to the prompt word;

[0064] In some embodiments, the input of the large model is constructed and encapsulated in the prompt word to guide the large model to understand the task demand of optimizing and adjusting the interface layout display.

[0065] The structure of the prompt word in the present application includes four elements: instruction, rule, in-context exemplar, and output indicator.

[0066] (1) Instruction, which includes the following content: role setting: give a role to the large model, in the present application, let the large model play the role of the control system of the interactive interface, and adjust and optimize the interface layout according to the user state; task description: describe the task to be completed by the large model, in the present application, the task to be completed by the large model is to adjust part of the field of the interface layout description file according to the current state score of the user to increase / decrease the display content of the interface; reasoning limit: guide the retrieval ability and reasoning process of the large model by adding rule limits to the reasoning process. For example, the rule limit in the present application is: please answer the question according to the step by step thinking chain mode, and strictly follow the retrieved context, if you do not know the answer, do not output other irrelevant messages, the context information is: {In-Context Exemplars}.

[0067] (2) Rule, including the following principles of interface modification, such as the limitation of component position and boundary value, the uniformity of color style, etc. Specifically, component layout: avoid overlapping, ensure that all components on the interface do not overlap, and maintain a clear hierarchical structure; uniform distribution, components in the interface should be evenly distributed, avoiding visual congestion or excessive white space. Component alignment: horizontal alignment, horizontally adjacent components should be aligned along the same horizontal line; vertical alignment, vertically adjacent components should be aligned along the same vertical line; color uniformity: color coordination, the overall color of the interface should be uniform and coordinated, avoiding the use of too many different color tones; color contrast, the text and background color should have sufficient contrast to ensure content readability.

[0068] (3) In-context exemplar

[0069] Give the large model a set of examples, which is a list of input-output pairs that describe a task. In the present application, first, a small number of thinking chain reasoning sample examples are given, which use the step by step thinking chain mode to solve complex interface layout optimization problems, and each example is labeled with detailed thinking chain, so that the large model can simulate the human thinking and reasoning process to generate intermediate reasoning steps before obtaining the result. Through the RAG retrieval enhancement technology, retrieve the top-k examples most similar to the current driver state and information sequence in the knowledge base and add them to the context. The retrieved results are encapsulated as examples in the prompt word given to the large model.

[0070] (4) Output indicator

[0071] Limit the output content and format of the large model. In the present application, there are modification functions for component color, size, position, motion effect, output text, sound, seat vibration, etc. The large model outputs the function call result in JSON format. The result is parsed to obtain the component name to be modified, the modification parameter and the modification sequence. The execution module modifies the layout description file of the interface according to the output of the large model, and obtains the adjusted interface by calling the related functions and function parameters.

[0072] In the interface generation step S110, the constructed prompt word is input into the large model, and the large model outputs the adjusted interface after reasoning.

[0073] In some embodiments, the input information is filled by prompt engineering, and the complete prompt word instruction containing context information, interface adjustment example and user demand is input into the large model after filling. The large model outputs the adjusted interface after reasoning. The adjustment result is presented on the human-machine interface or user interface (such as vehicle-mounted AR-HUD). The user state may change under the influence of the adjusted interface. The new user state is combined with the historical information to further reason to obtain a new round of interface layout description, so as to realize multi-round interactive interface layout content adjustment and optimization.

[0074] In some embodiments, it also includes a fine-tuning data set construction step, constructs a related data set, gives the user state and information sequence S' with priority in the current interaction situation, and takes the user interface description adjusted by the expert under the guidance as the optimization result. In the present application, the main task of this step is to create a database containing the correlation between user state and interface description, and to provide accurate and efficient reference and training data for the prompt engineering of the large model.

[0075] In some embodiments, it also includes a fine-tuning step, which fine-tunes the parameters of the pre-trained large model using the question and answer pairs in the fine-tuning data set, to obtain a large model more suitable for the interface adjustment field. Specifically, a large model such as ChatGLM4-6B model is selected, and LoRA technology based on efficient parameter fine-tuning is used to fine-tune the base model. The core idea is to introduce low-rank decomposition in the original pre-trained model to simulate the modification amount of the parameters, and to realize the fine-tuning of the large model with a small amount of additional parameters. The parameters of the large model are frozen, and two low-rank matrices A and B are introduced in each transformer layer of the pre-trained model. These matrices are multiplied with the original attention matrix or feedforward network weight matrix to introduce new trainable parameters.

[0076] In fine-tuning the interface, the input x is processed by the large model to obtain the representation h1, and the low-rank matrices A and B are processed to obtain the representation h2. The h1 and h2 are fused to obtain h. The difference between the output h and the given standard output is compared to determine the model loss, and the parameters of the matrices A and B are updated. In inference, the parameters of A and B are used together with the parameters of the large model. This fine-tuning method introduces a small number of additional parameters in the original pre-trained model to fine-tune the model, rather than changing all the parameters of the model, which can significantly reduce the training time and cost.

[0077] The data set is divided into training set, validation set and test set. In order to evaluate the standardization of the generated layout, the alignment degree and overlap between components are calculated according to the number and position of the generated bounding box. In order to evaluate the difference between the generated layout and the standard layout, the similarity between the generated layout and the standard layout is calculated, and three evaluation indexes are adopted, including FID (the difference between the generated model and the real data distribution), MaxIoU (the overlap degree of components between the two layouts) and artificial evaluation.

[0078] In some embodiments, the evaluation step is also included, which evaluates the effectiveness of the layout through the reward model and user evaluation, and uses the evaluation results for the training of the large model. The evaluation indexes include: (1) layout standardization: whether there is alignment and overlap between components, and MaxIoU and FID between the generated layout and the standard layout can be calculated in the test set. (2) layout effectiveness: reward model scoring and user evaluation. Among them, the reward model is a reward model trained by data with human preference, which can score each generated layout, and the higher the score, the more the layout conforms to the user's preference. (3) User evaluation: through the induction material to make the driver in different levels of cognitive state, let the driver experience the adaptive interface and the conventional interface in the same driving scene, and obtain the evaluation of the driver for the two interface presentation methods through questionnaire and interview.

[0079] The user interface optimization method based on the large model of the application refines the related fine-tuning data to fine-tune part of the parameters of the large model; inputs the current cognitive state of the driver and the existing HUD layout, retrieves the knowledge base to add relevant information to the context, encapsulates the prompt word based on few-shot learning and thought chain, inputs the encapsulated prompt word into the large model, and the large model outputs the JSON format function call result. The result is parsed to obtain the component name to be modified, the modification parameter and the modification sequence; the system executes the display parameter change to realize the adjustment of the interface display, so as to maintain the cognitive state of the driver in a reasonable range and ensure that the driver has sufficient situation awareness ability.

[0080] Figure 2 Flowcharts showing other embodiments of the user interface optimization method based on the large model of the application are shown. As shown inFigure 2 As shown, the original information sequence is processed according to the driver interaction preference to obtain a priority information sequence, which is saved to the domain knowledge base together with the encoded driver state, the prompt word is encapsulated according to the information obtained by searching in the knowledge base, the adjusted interface layout is obtained according to the prompt word through the large language model after fine-tuning, the new user state is further inferred after the user state is affected by the adjusted interface combined with the historical information to obtain a new round of interface layout description, so as to realize the multi-round interactive interface layout content adjustment and optimization.

[0081] Figure 3 The interface self-adaptive display flowchart based on retrieval enhancement generated in the application is shown. As shown in the figure Figure 3 The interface adjustment example with high similarity to the input is searched in the knowledge base, the prompt word is formed using the interface adjustment example, and the interface is adjusted through the large model.

[0082] The application also provides a user interface optimization device based on a large model. As shown Figure 4 The application also provides a user interface optimization device based on a large model. As shown

[0083] The user preference acquisition module 100 is used to predict the preference degree of the user to different types of information on the interface according to the user historical interaction data, and obtain a priority information sequence;

[0084] The knowledge base construction module 200 is used to construct a knowledge base, and the knowledge base includes the mapping relationship between the user state, the priority information sequence and the interface adjustment example;

[0085] The retrieval module 300 is used to search the relevant interface adjustment example with high similarity to the input in the knowledge base by taking the user state and the priority information sequence as input;

[0086] The prompt word construction module 400 is used to construct and encapsulate the input of the large model, and add the interface adjustment example with high similarity searched from the knowledge base to the prompt word;

[0087] The interface generation module 500 is used to input the constructed prompt word into the large model, and the large model outputs the adjusted interface after reasoning.

[0088] The specific limitation of the large model-based user interface optimization apparatus can refer to the limitation of the large model-based user interface optimization method described above, which will not be repeated here. Each module in the large model-based user interface optimization apparatus described above can be realized by software, hardware and their combination in whole or in part. The above-mentioned modules can be embedded in or independent of the processor in the computer device in hardware form, or stored in the memory in the computer device in software form, so as to call and execute the operation corresponding to each module by the processor.

[0089] The computer device can be a terminal, and its internal structure diagram can be as shown in Figure 5 The computer device includes a processor, a memory, a network interface, a display screen and an input device connected through a system bus. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operating system and the computer program in the non-volatile storage medium. The network interface of the computer device is used to communicate with external terminals through network connection. The computer program is executed by the processor to implement the large model-based user interface optimization method described above. The display screen of the computer device can be a liquid crystal display screen or an electronic ink display screen. The input device of the computer device can be a touch layer overlaid on the display screen, or a key, trackball or touchpad arranged on the shell of the computer device, or an external keyboard, touchpad or mouse, etc. Those skilled in the art can understand that Figure 5 The structure shown in the figure is only a block diagram of part of the structure related to the present application, and does not constitute a limitation on the computer device to which the present application is applied. The specific computer device can include more or fewer components than those shown in the figure, or combine certain components, or have a different component arrangement.

[0090] The present application also provides a computer readable storage medium having a computer program stored thereon, and the computer program is executed by the processor to implement the large model-based user interface optimization method described above.

[0091] Those skilled in the art can understand that all or part of the processes in the above-mentioned method embodiments can be completed by instructing relevant hardware through a computer program. The computer program can be stored in a non-volatile computer readable storage medium, and when executed, can include the processes of the above-mentioned embodiments. Any reference to memory, storage, database or other medium used in the embodiments of the present application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration but not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDR SDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus), direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0092] So far, the embodiments of the present application have been described in detail. In order to avoid obscuring the concept of the present application, some details known in the art are not described. Those skilled in the art can fully understand how to implement the technical solutions disclosed herein according to the above description.

[0093] Although some specific embodiments of the present application have been described in detail through examples, those skilled in the art should understand that the above examples are only for illustration, not for limiting the scope of the present application. Those skilled in the art should understand that the above embodiments can be modified or some technical features can be replaced equivalently without departing from the scope and spirit of the present application. The scope of the present application is defined by the appended claims.

Claims

1. A large model-based user interface optimization method, characterized by, The method comprises the following steps: a user preference acquisition step, which predicts the preference degree of a user for different types of information on an interface according to user historical interaction data, to obtain an information sequence with priorities; a knowledge base construction step, which constructs a knowledge base comprising a mapping relationship between a user state, the information sequence with priorities, and interface adjustment examples; a retrieval step, which retrieves, in the knowledge base, related interface adjustment examples with high similarity to the input, taking the user state and the information sequence with priorities as the input; a prompt word construction step, which constructs and encapsulates prompt words for the input of a large model, and adds the interface adjustment examples with high similarity retrieved from the knowledge base to the prompt words; an interface generation step, which inputs the constructed prompt words into the large model, and outputs adjusted interface codes after inference of the large model.

2. The large model based user interface optimization method of claim 1, wherein, Further comprising: a fine-tuning data set construction step, which takes the user state and the information sequence with priorities as input data, takes the interface description adjusted by a human being as output data, takes the input data and the output data as a question and answer pair, and constructs a fine-tuning data set; a fine-tuning step, which fine-tunes the parameters of the pre-trained large model by using the question and answer pair in the fine-tuning data set, to obtain a large model more suitable for the interface adjustment field.

3. The user interface optimization method based on a large model according to claim 2, characterized in that: the fine-tuning of the parameters of the pre-trained large model comprises: performing partial parameter fine-tuning of the large model by using a low-rank adaptation LoRA fine-tuning method as an adapter.

4. The user interface optimization method based on a large model according to claim 1, characterized in that: the knowledge base is constructed by using a vector database, and information in the knowledge base is stored in the vector database in the form of vectors; the information in the knowledge base is divided into blocks by using a segmentation algorithm, the blocks are encoded to obtain respective encodings, and the encodings are converted into an array and saved in the knowledge base.

5. The user interface optimization method based on a large model according to claim 4, characterized in that: the retrieval of related interface adjustment examples with high similarity in the knowledge base comprises: obtaining the information sequence with priorities and the user state, and converting the two inputs into word vector representations by word vector matching; performing similarity calculation and sorting on the user state and the information sequence with priorities stored in the knowledge base by using a word vector model, and retrieving the top K interface adjustment examples with the highest similarity.

6. The user interface optimization method based on a large model according to claim 1, characterized in that: the prompt words comprise: instructions, adjustment principles, interface adjustment examples, and output guides; the instructions are used to set the role of the large model, and give a task description and inference restrictions; the adjustment principles comprise: layout principles, alignment principles, and color principles; the output guides are used to limit the output content and output format of the large model; the prompt words further comprise API interface information.

7. The large model based user interface optimization method of claim 1, wherein, Further comprising: an evaluation step, which evaluates the effectiveness of the layout by using a reward model and user evaluation, and uses the evaluation results for training of the large model.

8. A large model-based user interface optimization apparatus, characterized by, Further comprising: A user preference acquisition module is configured to predict the preference degree of a user for different types of information on an interface according to user historical interaction data, and obtain an information sequence with priorities. A knowledge base construction module is configured to construct a knowledge base, which includes a mapping relationship between a user state, an information sequence with priorities, and an interface adjustment example. A retrieval module is configured to retrieve, as input, a user state and an information sequence with priorities in the knowledge base to obtain a relevant interface adjustment example with high similarity to the input. A prompt word construction module is configured to construct and encapsulate prompt words for the input of a large model, and add the interface adjustment example with high similarity retrieved from the knowledge base to the prompt words. An interface generation module is configured to input the constructed prompt words into the large model, and output an adjusted interface code after the inference of the large model.

9. A computer device comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, characterized in that, The processor executes the computer program to implement the steps of the method of any one of claims 1 to 7.

10. A computer-readable storage medium having stored thereon a computer program, characterized in that, The computer program is executed by the processor to implement the steps of the method of any one of claims 1 to 7.

Images