Interactive front end GUI automatic layout method and device based on large model

Abstract

The invention discloses an interactive front-end GUI automatic layout method and device based on a large model, and relates to the technical field of man-machine interaction. Comprising the steps of 1, collecting interaction data related to user behaviors of a front-end GUI in real time, 2, generating interface layout information for user behavior feedback by using a deep reinforcement learning algorithm according to the interaction data, and 3, generating interface layout information under different user equipment and user use scenes by using a multi-task learning algorithm, 4, distributing interface layouts of different versions for different users, collecting interaction data as user feedback data, recognizing user behaviors by using a self-supervised learning algorithm according to the user feedback data, and obtaining a recognition result, and 5, evaluating the interface layouts of different versions according to the recognition result by using a dobby machine algorithm, and obtaining layout optimization suggestions. Adjusting the layout into a final interface layout according to the layout optimization suggestion; the method improves the user experience, and is suitable for multi-device, multi-user and complex interaction scenes.

Description

Technical Field

[0001] The invention discloses a large-model-based interactive front-end GUI automatic layout method and device, and relates to the technical field of human-computer interaction. Background Art

[0002] With the development of information technology, the interface design of modern applications has become increasingly complex. Existing GUI design methods mainly rely on manual design and lack the ability to dynamically respond to and adapt to users' real-time interactive behaviors. At the same time, users' demands for personalization and multi-device support are growing, and static interface design methods can no longer meet these demands. Summary of the invention

[0003] In view of the problems of the prior art, the present invention provides a method and device for automatic layout of an interactive front-end GUI based on a large model, which improves the user experience by dynamically optimizing the interface layout and is suitable for multiple devices, multiple users and complex interactive scenarios.

[0004] The specific scheme proposed by the present invention is: The present invention provides an interactive front-end GUI automatic layout method based on a large model, comprising: Step 1: Collect interactive data about user behavior in front-end GUI in real time. Step 2: Based on the interaction data, use the deep reinforcement learning algorithm to generate interface layout information for user behavior feedback. Step 3: Based on the interface layout information, user device information and user usage scenarios, a multi-task learning algorithm is used to generate interface layout information for different user devices and user usage scenarios. Step 4: Generate different versions of interface layouts based on different user devices and interface layout information in user usage scenarios, assign different versions of interface layouts to different users, collect interaction data as user feedback data, and use self-supervised learning algorithms to identify user behaviors based on user feedback data to obtain identification results. Step 5: Use the multi-armed bandit algorithm to evaluate different versions of the interface layout based on the recognition results, obtain layout optimization suggestions based on user feedback data, and adjust the final interface layout based on the layout optimization suggestions.

[0005] Furthermore, the interaction data collected in step 1 of the large-model-based interactive front-end GUI automatic layout method includes click frequency, sliding frequency, dwell time and other behavior data.

[0006] Furthermore, step 2 of the automatic layout method of an interactive front-end GUI based on a large model specifically includes: selecting a deep Q network algorithm in a deep reinforcement learning algorithm to generate interface layout information for user behavior feedback, wherein the deep Q network algorithm is used to record the current GUI interface state according to the interaction data, and adjust the elements on the interface according to the current GUI interface state, and at the same time obtain feedback according to real-time interaction data, and use a reward function to adjust the GUI interface to generate interface layout information according to the feedback.

[0007] Furthermore, in step 3 of the interactive front-end GUI automatic layout method based on a large model, the method specifically includes: constructing a multi-task learning neural network, processing interface layout tasks under different user devices and user usage scenarios through a multi-task learning algorithm, The neural network includes a shared layer and a task layer. The shared layer is used to extract the features of the interface layout information obtained in step 2, and the task layer is used to predict different interface layout tasks based on the features, and adjust and generate interface layout information for different user devices and user usage scenarios.

[0008] Furthermore, in step 4 of the method for automatic layout of an interactive front-end GUI based on a large model, a recognition model is trained using a self-supervised learning algorithm based on unlabeled interaction data, and the recognition model is fine-tuned using labeled data. The recognition model is used to identify user behavior based on user feedback data to obtain a recognition result.

[0009] The present invention also provides an interactive front-end GUI automatic layout device based on a large model, comprising a user input processing module, a deep reinforcement learning module, a multi-task learning module, a self-supervised learning module, an optimization module and an interface layout presentation module. The user input processing module collects the interactive data of the front-end GUI related to user behavior in real time. The deep reinforcement learning module uses the deep reinforcement learning algorithm to generate interface layout information for user behavior feedback based on the interaction data. The multi-task learning module generates interface layout information for different user devices and user usage scenarios using a multi-task learning algorithm based on interface layout information, user device information and user usage scenarios. Different versions of interface layouts are generated based on the interface layout information for different user devices and user usage scenarios, and different versions of interface layouts are allocated to different users. The user input processing module collects interaction data as user feedback data. The self-supervised learning module identifies user behavior based on the user feedback data using a self-supervised learning algorithm to obtain identification results. The optimization module uses a multi-armed bandit algorithm to evaluate different versions of interface layouts based on the recognition results, and obtains layout optimization suggestions based on user feedback data. The interface layout presentation module adjusts the final interface layout based on the layout optimization suggestions.

[0010] Furthermore, the interaction data collected by the user input processing module of the interactive front-end GUI automatic layout device based on a large model includes click frequency, sliding frequency, dwell time and other behavior data.

[0011] Furthermore, the deep reinforcement learning module of the interactive front-end GUI automatic layout device based on a large model selects the deep Q network algorithm in the deep reinforcement learning algorithm to generate interface layout information for user behavior feedback, wherein the deep Q network algorithm is used to record the current GUI interface state according to the interaction data, and adjust the elements on the interface according to the current GUI interface state, and at the same time obtain feedback according to the real-time interaction data, and use the reward function to adjust the GUI interface to generate interface layout information according to the feedback.

[0012] Furthermore, the multi-task learning module of the large-model-based interactive front-end GUI automatic layout device constructs a multi-task learning neural network, and processes interface layout tasks under different user devices and user usage scenarios through a multi-task learning algorithm. The neural network includes a shared layer and a task layer. The shared layer is used to extract the features of the interface layout information obtained by the deep reinforcement learning module, and the task layer is used to predict different interface layout tasks based on the features, and adjust and generate interface layout information under different user devices and user usage scenarios.

[0013] Furthermore, the self-supervised learning module of the interactive front-end GUI automatic layout device based on a large model uses a self-supervised learning algorithm to train a recognition model according to unlabeled interaction data, and uses label data to fine-tune the recognition model, and uses the recognition model to identify user behavior according to user feedback data to obtain recognition results.

[0014] The benefits of the method of the present invention are: It combines deep reinforcement learning, multi-task learning, self-supervised learning and A / B testing optimization to achieve dynamic user interface layout, enabling the system to adapt to user interaction behaviors, support complex application scenarios of multiple devices, multiple users and multiple tasks, improve the intelligence level of interface design, and greatly enhance user experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] Figure 1 It is a schematic diagram of the interaction flow of the device modules of the present invention. DETAILED DESCRIPTION

[0016] The present invention is further described below in conjunction with the accompanying drawings and specific embodiments so that those skilled in the art can better understand the present invention and implement it, but the embodiments are not intended to limit the present invention.

[0017] Embodiment 1: The present invention provides an automatic layout method of an interactive front-end GUI based on a large model, comprising: Step 1: Collect interactive data of user behaviors on the front-end GUI in real time. The collected interactive data includes click frequency, sliding frequency, dwell time, and other behavioral data.

[0018] Step 2: Based on the interaction data, use the deep reinforcement learning algorithm to generate interface layout information for user behavior feedback.

[0019] Specifically, it may include: selecting the deep Q network algorithm in the deep reinforcement learning algorithm to generate interface layout information for user behavior feedback, wherein the deep Q network algorithm is used to record the current GUI interface state according to the interaction data, and adjust the elements on the interface according to the current GUI interface state, such as adjusting the button size, color, position and other elements, and obtaining feedback according to the real-time interaction data, and adjusting the GUI interface to generate interface layout information according to the feedback using the reward function. For example, according to the click frequency, sliding frequency, dwell time, etc. in the interaction data, the reward function is used in the training process to continuously try, generate reward signals if it is correct, and penalty signals if it is wrong, so as to obtain the best GUI interface layout information in each GUI interface state.

[0020] Step 3: Based on the interface layout information, user device information and user usage scenarios, a multi-task learning algorithm is used to generate interface layout information for different user devices and user usage scenarios.

[0021] Specifically, it may include: building a multi-task learning neural network, processing interface layout tasks under different user devices and user usage scenarios through multi-task learning algorithms, The neural network includes a shared layer and a task layer. The shared layer is used to extract the features of the interface layout information obtained in step 2 through a convolutional neural network CNN or a Transformer model. The task layer is used to predict different interface layout tasks based on the features, and adjust and generate interface layout information for different user devices and user usage scenarios. A weighted multi-task loss function is also set to simultaneously optimize multiple interface layouts through the multi-task loss function. The loss value of each task can be weighted and summed according to its priority and difficulty to ensure the trade-off between different tasks. Alternatively, adaptive weight adjustment can be performed: the GradNorm algorithm is used to automatically adjust the weight of each task to ensure a balance when optimizing different tasks and prevent a certain task from excessively dominating the layout generation.

[0022] Step 4: Generate different versions of interface layouts based on the interface layout information of different user devices and user usage scenarios, assign different versions of interface layouts to different users, collect interaction data as user feedback data, and use a self-supervised learning algorithm to identify user behavior based on the user feedback data to obtain recognition results.

[0023] The recognition model is trained using a self-supervised learning algorithm based on unlabeled interaction data, and the label data is used to fine-tune the recognition model. The recognition model is used to identify user behavior based on user feedback data to obtain recognition results.

[0024] Step 5: Use the multi-armed bandit algorithm to evaluate different versions of the interface layout based on the recognition results, and obtain layout optimization suggestions based on user feedback data, and adjust the final interface layout based on the layout optimization suggestions. The Thompson Sampling algorithm in the multi-armed bandit algorithm can be used to estimate the effect distribution of different layout schemes through Bayesian inference. After each user interaction, the probability distribution is updated, and the optimized layout and layout optimization suggestions are gradually generated.

[0025] Embodiment 2: The present invention also provides an interactive front-end GUI automatic layout device based on a large model, comprising a user input processing module, a deep reinforcement learning module, a multi-task learning module, a self-supervised learning module, an optimization module and an interface layout presentation module. The user input processing module collects the interactive data of the front-end GUI involving user behaviors in real time, and passes the interactive data as input to subsequent modules, providing a basis for optimization.

[0026] The deep reinforcement learning module generates interface layout information for user behavior feedback using the deep reinforcement learning algorithm based on the interaction data, and passes the interface layout information for user behavior feedback to the multi-task learning module.

[0027] The multi-task learning module generates interface layout information for different user devices and user usage scenarios using a multi-task learning algorithm based on interface layout information, user device information and user usage scenarios. Different versions of interface layouts are generated based on the interface layout information for different user devices and user usage scenarios, and different versions of interface layouts are allocated to different users. The user input processing module collects interaction data as user feedback data. The self-supervised learning module identifies user behavior based on the user feedback data using a self-supervised learning algorithm to obtain identification results. The optimization module uses the multi-armed bandit algorithm to evaluate different versions of interface layouts based on the recognition results, and obtains layout optimization suggestions based on user feedback data. The interface layout presentation module adjusts the final interface layout according to the layout optimization suggestions and presents it to the user to form an interactive interface between the user and the system.

[0028] As the information interaction and execution process between the modules of the above-mentioned device are based on the same concept as the embodiment of the method of the present invention, the specific contents can be found in the description of the embodiment of the method of the present invention and will not be repeated here.

[0029] Likewise, the benefits of the device of the present invention are: It combines deep reinforcement learning, multi-task learning, self-supervised learning and A / B testing optimization to achieve dynamic user interface layout, enabling the system to adapt to user interaction behaviors, support complex application scenarios of multiple devices, multiple users and multiple tasks, improve the intelligence level of interface design, and greatly enhance user experience.

[0030] It should be noted that not all steps and modules in the above-mentioned processes and device structures are necessary, and some steps or modules can be ignored according to actual needs. The execution order of each step is not fixed and can be adjusted as needed. The system structure described in the above-mentioned embodiments can be a physical structure or a logical structure, that is, some modules may be implemented by the same physical entity, or some modules may be implemented by multiple physical entities, or some components in multiple independent devices may be implemented together.

[0031] The above-described embodiments are only preferred embodiments for fully illustrating the present invention, and the protection scope of the present invention is not limited thereto. Equivalent substitutions or changes made by those skilled in the art based on the present invention are within the protection scope of the present invention. The protection scope of the present invention shall be subject to the claims.

Claims

1. A large-model-based interactive front-end GUI automatic layout method, characterized by include: Step 1: Collect interactive data about user behavior in front-end GUI in real time. Step 2: Based on the interaction data, use the deep reinforcement learning algorithm to generate interface layout information for user behavior feedback. Step 3: Based on the interface layout information, user device information and user usage scenarios, a multi-task learning algorithm is used to generate interface layout information for different user devices and user usage scenarios. Step 4: Generate different versions of interface layouts based on different user devices and interface layout information in user usage scenarios, assign different versions of interface layouts to different users, collect interaction data as user feedback data, and use self-supervised learning algorithms to identify user behaviors based on user feedback data to obtain identification results. Step 5: Use the multi-armed bandit algorithm to evaluate different versions of the interface layout based on the recognition results, obtain layout optimization suggestions based on user feedback data, and adjust the final interface layout based on the layout optimization suggestions.

2. According to the large model-based interactive front-end GUI automatic layout method of claim 1, characterized in that The interaction data collected in step 1 includes click frequency, sliding frequency, dwell time, and other behavioral data.

3. The method for automatic layout of interactive front-end GUI based on a large model according to claim 1, characterized in that Step 2 specifically includes: selecting a deep Q network algorithm in a deep reinforcement learning algorithm to generate interface layout information for user behavior feedback, wherein the deep Q network algorithm is used to record the current GUI interface state according to the interaction data, and adjust the elements on the interface according to the current GUI interface state, and at the same time obtain feedback according to the real-time interaction data, and use the reward function to adjust the GUI interface according to the feedback to generate interface layout information.

4. The method for automatic layout of interactive front-end GUI based on a large model according to claim 1, characterized in that Step 3 specifically includes: building a multi-task learning neural network, processing interface layout tasks under different user devices and user usage scenarios through a multi-task learning algorithm, The neural network includes a shared layer and a task layer. The shared layer is used to extract the features of the interface layout information obtained in step 2, and the task layer is used to predict different interface layout tasks based on the features, and adjust and generate interface layout information for different user devices and user usage scenarios.

5. The method for automatic layout of interactive front-end GUI based on a large model according to claim 1, characterized in that In step 4, the recognition model is trained using a self-supervised learning algorithm based on the unlabeled interaction data, and the recognition model is fine-tuned using the labeled data. The recognition model is used to identify user behavior based on the user feedback data to obtain the recognition result.

6. An interactive front-end GUI automatic layout device based on a large model, characterized by It includes user input processing module, deep reinforcement learning module, multi-task learning module, self-supervised learning module, optimization module and interface layout presentation module. The user input processing module collects the interactive data of the front-end GUI related to user behavior in real time. The deep reinforcement learning module uses the deep reinforcement learning algorithm to generate interface layout information for user behavior feedback based on the interaction data. The multi-task learning module generates interface layout information for different user devices and user usage scenarios using a multi-task learning algorithm based on interface layout information, user device information and user usage scenarios. Different versions of interface layouts are generated based on the interface layout information for different user devices and user usage scenarios, and different versions of interface layouts are allocated to different users. The user input processing module collects interaction data as user feedback data. The self-supervised learning module identifies user behavior based on the user feedback data using a self-supervised learning algorithm to obtain identification results. The optimization module uses a multi-armed bandit algorithm to evaluate different versions of interface layouts based on the recognition results, and obtains layout optimization suggestions based on user feedback data. The interface layout presentation module adjusts the final interface layout based on the layout optimization suggestions.

7. The interactive front-end GUI automatic layout device based on a large model according to claim 6 is characterized in that the user The interaction data collected by the input processing module includes click frequency, sliding frequency, dwell time and other behavioral data.

8. The interactive front-end GUI automatic layout device based on a large model according to claim 6 is characterized by: The deep reinforcement learning module selects the deep Q network algorithm in the deep reinforcement learning algorithm to generate interface layout information for user behavior feedback. The deep Q network algorithm is used to record the current GUI interface state according to the interaction data, and adjust the elements on the interface according to the current GUI interface state. At the same time, feedback is obtained based on real-time interaction data, and the reward function is used to adjust the GUI interface to generate interface layout information based on the feedback.

9. The interactive front-end GUI automatic layout device based on a large model according to claim 6, characterized in that The multi-task learning module builds a multi-task learning neural network and processes interface layout tasks under different user devices and user usage scenarios through a multi-task learning algorithm. The neural network includes a shared layer and a task layer. The shared layer is used to extract the features of the interface layout information obtained by the deep reinforcement learning module, and the task layer is used to predict different interface layout tasks based on the features, and adjust and generate interface layout information under different user devices and user usage scenarios.

10. The large-model-based interactive front-end GUI automatic layout device according to claim 6, characterized in that The self-supervised learning module uses a self-supervised learning algorithm to train the recognition model based on the unlabeled interaction data, and uses the labeled data to fine-tune the recognition model. It uses the recognition model to identify user behavior based on the user feedback data to obtain the recognition results.

Images