Interface element position adjustment method and apparatus, device, medium, and program product

Abstract

The application provides a position adjustment method and device of an interface element, equipment, a medium and a program product, and relates to the technical field of Internet. The method comprises the following steps: obtaining an initial position of an interface element in an interface according to a pre-trained deep learning model and an interface element sequence, wherein the initial position comprises a stacking attribute, and the stacking attribute is used for indicating a stacking order of the interface element in the interface; and in the case that the interface is in an editing mode, the initial position is adjusted through a position adjustment event to obtain a target position of a target interface element in the interface. According to the scheme, an accurate initial page layout can be obtained, so that accurate element position adjustment is realized based on the initial page layout, and user experience is improved.

Description

Technical Field

[0001] This application relates to the field of Internet technology, specifically to a method, apparatus, device, medium, and program product for adjusting the position of interface elements. Background Technology

[0002] On touchscreen devices, users need to adjust the position of interface elements. For example, in content management scenarios within a business hall application, users need to drag and drop to adjust the order of content or lists; in mobile office applications, users need to drag and drop to adjust the order of items in workflows or task lists; and in user interface customization scenarios on smart terminal devices, such as smart speakers or smart TVs, users need to drag and drop to adjust the position of control panels or menus. Therefore, in all of the above scenarios, users need to adjust the position of interface elements on the touchscreen.

[0003] Currently, there are two methods to adjust the position of UI elements; however, both methods have some drawbacks. Method one involves using a third-party visual layout plugin. This method suffers from plugin compatibility issues due to differences in devices and browser versions, impacting the user experience. Furthermore, the additional third-party plugin can cause excessive loading of front-end static resources, affecting interface loading speed. Method two involves obtaining the initial position information of all UI elements and the target position information the user wants to move them to, and updating the entire UI layout based on the positional differences. This method cannot automatically provide a relatively accurate initial page layout. If the initial position information is inaccurate or contains errors, subsequent positional differences and target position information may also be affected, leading to an inaccurate final layout and impacting the UI display. Summary of the Invention

[0004] At least one embodiment of this application provides a method, apparatus, device, medium, and program product for adjusting the position of interface elements, which solves the problems of poor user experience, slow interface loading speed, and inaccurate layout faced by the prior art in adjusting the position of interface elements.

[0005] To solve the above-mentioned technical problems, this application is implemented as follows:

[0006] In a first aspect, embodiments of this application provide a method for adjusting the position of interface elements, including:

[0007] Based on the pre-trained deep learning model and the sequence of interface elements, the initial position of the interface elements in the interface is obtained. The initial position includes a stacking attribute, which is used to indicate the stacking order of the interface elements in the interface.

[0008] When the interface is in edit mode, the initial position is adjusted through the position adjustment event to obtain the target position of the target interface element in the interface.

[0009] Optionally, the method for adjusting the position of interface elements, wherein obtaining the initial position of the interface elements in the interface based on the pre-trained deep learning model and the sequence of interface elements includes:

[0010] Based on the pre-trained deep learning model, the sequence of interface elements, and the context information corresponding to each interface element, an element position sequence is output; wherein, the interface element sequence includes at least one of the following attributes of each interface element: the x-coordinate of the top left corner, the y-coordinate of the top left corner, the width, and the height; the context information corresponding to each interface element includes the attributes of adjacent interface elements; and the element position sequence includes the initial position of each interface element.

[0011] Based on the element position sequence, obtain the initial position of the target interface element in the interface.

[0012] Optionally, in the method for adjusting the position of interface elements, the deep learning model includes:

[0013] The sequence-to-sequence model and the recurrent neural network are used to generate the x-coordinate and y-coordinate of each interface element based on the interface element sequence. The recurrent neural network is used to generate the element weight of each interface element and generate the element position sequence based on the x-coordinate, y-coordinate and element weight of each interface element. The stacking attribute in the initial position of each interface element in the element position sequence is related to the element weight.

[0014] Optionally, in the method for adjusting the position of interface elements, the position adjustment events include: a position adjustment start event, a position movement event, and a position adjustment end event.

[0015] The initial position is adjusted using a position adjustment event to obtain the target position of the target interface element within the interface, including:

[0016] The initial position of the control point is obtained by adjusting the position and starting the event.

[0017] Based on the initial position of the control point and the initial position, the control point offset is obtained, and the control point offset is used to indicate the offset of the control point relative to the initial position;

[0018] The real-time position of the control point is obtained through the position movement event;

[0019] Based on the real-time position of the control point and the offset of the control point, the real-time position of the target interface element during the position movement process is obtained;

[0020] The target position of the target interface element in the interface is obtained by using the position adjustment end event.

[0021] Optionally, the method for adjusting the position of interface elements further includes:

[0022] Obtain the boundary detection results of the target location;

[0023] Based on the boundary detection results, determine whether to update the target position and obtain the determination result;

[0024] Based on the judgment result, the target location is determined.

[0025] Optionally, in the method for adjusting the position of the interface element, the target position includes an abscissa relative to the left boundary of the interface and a ordinate relative to the top boundary of the interface.

[0026] Obtain the boundary detection results of the target location, including at least one of the following:

[0027] Determine whether the x-coordinate of the target location is less than 0 to obtain the left boundary detection result of the target location;

[0028] Determine whether the sum of the horizontal coordinate of the target location and the width of the target interface element is greater than the interface width to obtain the right boundary detection result of the target location;

[0029] Determine whether the ordinate of the target location is less than 0 to obtain the upper boundary detection result of the target location;

[0030] Determine whether the sum of the ordinate of the target location and the height of the target interface element is greater than the interface height to obtain the lower boundary detection result of the target location.

[0031] Optionally, the method for adjusting the position of the interface element, wherein determining the target position when the determination result indicates that the target position needs to be updated, includes at least one of the following:

[0032] The x-coordinate of the target location is determined to be 0;

[0033] The x-coordinate of the target location is determined to be the interface width minus the target interface element width;

[0034] The ordinate of the target location is set to 0;

[0035] The vertical coordinate of the target location is determined as the interface height minus the height of the target interface element.

[0036] Secondly, embodiments of this application also provide a device for adjusting the position of an interface element, comprising:

[0037] The first acquisition module is used to acquire the initial position of the interface elements in the interface based on the pre-trained deep learning model and the sequence of interface elements. The initial position includes a stacking attribute, which is used to indicate the stacking order of the interface elements in the interface.

[0038] The first acquisition module is used to adjust the initial position through a position adjustment event when the interface is in edit mode, so as to obtain the target position of the target interface element in the interface.

[0039] Thirdly, embodiments of this application also provide a device for adjusting the position of interface elements, including: a processor, a memory, and a program or instructions stored in the memory and executable on the processor, wherein the processor executes the program or instructions to implement the method for adjusting the position of interface elements as described in the first aspect.

[0040] Fourthly, embodiments of this application also provide a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the interface element position adjustment method as described in the first aspect.

[0041] Fifthly, embodiments of this application also provide a computer program product, including computer instructions, which, when executed by a processor, implement the method for adjusting the position of interface elements as described in the first aspect.

[0042] Compared with existing technologies, the interface element position adjustment method described in this application obtains the initial position of the interface elements in the interface based on a pre-trained deep learning model and an interface element sequence. The initial position includes a stacking attribute, which indicates the stacking order of the interface elements in the interface. When the interface is in edit mode, the initial position is adjusted through a position adjustment event to obtain the target position of the target interface element in the interface. Thus, a precise initial page layout can be obtained based on the deep learning model. Real-time dynamic position adjustment based on this initial page layout can effectively improve user experience and interface loading speed. Attached Figure Description

[0043] Various other advantages and benefits will become apparent to those skilled in the art upon reading the following detailed description of preferred embodiments. The accompanying drawings are for illustrative purposes only and are not intended to limit the scope of this application. Furthermore, the same reference numerals denote the same parts throughout the drawings. In the drawings:

[0044] Figure 1 This is a flowchart illustrating the method for adjusting the position of interface elements according to an embodiment of this application;

[0045] Figure 2 This is a schematic diagram illustrating the element weights described in the embodiments of this application;

[0046] Figure 3 This is a schematic diagram of the interface element sequence and the element position sequence described in the embodiments of this application;

[0047] Figure 4 This is a schematic diagram of the initial page layout of the interface described in the embodiments of this application;

[0048] Figure 5 This is a schematic diagram of the interface described in the embodiments of this application in edit mode;

[0049] Figure 6 This is a schematic diagram showing the initial position of the target interface element in the interface according to an embodiment of this application;

[0050] Figure 7 This is a schematic diagram of the control point offset described in the embodiments of this application;

[0051] Figure 8 This is a schematic diagram illustrating the movement of the target interface element described in the embodiments of this application;

[0052] Figure 9 This is a schematic diagram of the right boundary detection results described in an embodiment of this application;

[0053] Figure 10 This is a schematic diagram showing the target interface element extending beyond the left boundary as described in the embodiments of this application;

[0054] Figure 11 This is a schematic diagram showing the target interface element extending beyond the right boundary as described in an embodiment of this application;

[0055] Figure 12 This is a schematic diagram showing the target interface element extending beyond the upper boundary as described in an embodiment of this application.

[0056] Figure 13 This is a schematic diagram showing the target interface element extending beyond the lower boundary as described in an embodiment of this application.

[0057] Figure 14 This is a schematic diagram of the architecture of the interface element position adjustment system described in the embodiments of this application;

[0058] Figure 15 This is a flowchart illustrating the interface element position adjustment system described in the embodiments of this application;

[0059] Figure 16 This is a schematic diagram of the structure of the interface element position adjustment device described in the embodiments of this application;

[0060] Figure 17 This is a hardware block diagram of the interface element position adjustment device described in the embodiments of this application. Detailed Implementation

[0061] The terms "first," "second," etc., used in this application are used to distinguish similar objects and not to describe a specific order or sequence. It should be understood that such terms can be used interchangeably where appropriate so that embodiments of this application can be implemented in orders other than those illustrated or described herein, and the objects distinguished by "first" and "second" are generally of the same class, without limiting the number of objects; for example, the first object can be one or more. Furthermore, "or" in this application indicates at least one of the connected objects. For example, "A or B" covers three scenarios: Scenario 1: including A but not B; Scenario 2: including B but not A; Scenario 3: including both A and B. The character " / " generally indicates that the preceding and following objects are in an "or" relationship.

[0062] Please refer to Figure 1 This application provides a method for adjusting the position of interface elements, including the following steps:

[0063] 101. Based on the pre-trained deep learning model and the sequence of interface elements, obtain the initial position of the interface elements in the interface. The initial position includes a stacking attribute, which is used to indicate the stacking order of the interface elements in the interface.

[0064] In this embodiment of the application, the initial position of each interface element in the interface is obtained based on the pre-trained deep learning model and the sequence of interface elements, and the initial page layout of the interface elements is generated based on the initial position. That is, the interface elements are set one by one in the interface based on the initial position to realize the automatic initial page layout of the interface elements.

[0065] The following describes the training process of a deep learning model, including:

[0066] Step 1: Collect a large amount of interface layout data as the training dataset. This interface layout data can come from application interfaces, web page layouts, and any other available UI (User Interface) design resources. For example, in a marketing poster creation scenario, interface elements include, but are not limited to: main title, subtitle, background image, text content, and QR code. The training dataset can be obtained through preprocessing, which includes at least: data cleaning, data transformation, feature selection, and data partitioning. The training dataset includes at least one of the following:

[0067] Tag name: HTML tags for interface elements (such as div, button, input, etc.).

[0068] Content: Text or other content within interface elements.

[0069] Attributes: CSS styles, class names, IDs, and other attributes of UI elements.

[0070] Location: The coordinates of an element within the interface (e.g., top-left corner coordinates, width, and height).

[0071] Contextual information: Information about other UI elements surrounding a UI element to understand its role in the overall layout.

[0072] In this embodiment of the application, optionally, the deep learning model includes:

[0073] The Seq2Seq (Sequence to Sequence) model and the RNN (Recursive Neural Network) are used. The Seq2Seq model is used to generate the x-coordinate and y-coordinate of each interface element based on the interface element sequence. The RNN is used to generate the element weight of each interface element in the interface element sequence and generate the element position sequence based on the x-coordinate, y-coordinate and element weight of each interface element. The stacking attribute in the initial position of each interface element in the element position sequence is related to the element weight.

[0074] Step 2: Using the Seq2Seq model, combined with RNN, the input sequence (training dataset) is transformed into the output sequence (predicted initial positions of interface elements);

[0075] It should be noted that the core idea of ​​the Seq2Seq model is to accept an input sequence, map it to a fixed-length representation through an Encoder, and then map this representation back to the output sequence through a Decoder.

[0076] RNN: Adding an RNN to a model can help the model better understand the important parts of the input sequence, thereby predicting the output more accurately. For example, when generating each output element, different weights can be given to multiple elements of the input sequence.

[0077] Optionally, element weights can be generated based on the HTML tags of the interface elements.

[0078] Specifically, based on the HTML tags of the interface elements, we obtain the importance and interactivity information of the interface elements; based on the importance and interactivity information of the interface elements, we generate element weights.

[0079] Figure 2 This is a schematic diagram illustrating an example of element weights as described in an embodiment of this application.

[0080] Next, combined Figure 2 Explanation of element weight settings:

[0081] (1) Interface elements with the HTML tag div (container): Although div is often used to organize content, it does not directly participate in user interaction and is more used to define structure than function. Therefore, the interface element corresponding to div can be given a low weight (e.g., element weight of 0), indicating that it is relatively less critical in the initial layout decision.

[0082] (2) Interface elements with HTML tags for button and input: These interface elements are usually the main way for users to interact with the interface, and their position and visibility are crucial. Therefore, the interface elements corresponding to button and input are given high weight (e.g., element weight of 1) to ensure their prominent position in the initial layout.

[0083] (3) Interface elements with the HTML tag img (image): Images often carry important visual information or brand identity, which is crucial for attracting users' attention. Therefore, the interface elements corresponding to img are also given high weight (e.g., element weight is 2).

[0084] Step 3: Arrange the interface elements in the element position sequence in ascending order of element weight, and set the element weight to the stacking property (z-index) of the interface elements.

[0085] z-index: Sets the stacking order of UI elements. UI elements with a higher stacking order will always appear in front of UI elements with a lower stacking order.

[0086] Step 4: For the interface elements in the arranged element position sequence, set them one by one in the interface according to the predicted initial position to realize the automatic initial layout of the interface elements.

[0087] In one implementation method, optionally, obtaining the initial position of the target interface element in the interface based on the pre-trained deep learning model and the interface element sequence includes:

[0088] Based on the pre-trained deep learning model, the sequence of interface elements, and the context information corresponding to each interface element, an element position sequence is output; wherein, the interface element sequence includes at least one attribute of each interface element, including label name, content, identifier, identifier of the preceding element, identifier of the succeeding element, x-coordinate of the top left corner, y-coordinate of the top left corner, width, and height; the context information corresponding to the interface element includes the attributes of adjacent interface elements; the element position sequence includes the initial position of each interface element.

[0089] Based on the element position sequence, obtain the initial position of the target interface element in the interface.

[0090] Figure 3 This is an example schematic diagram of the interface element sequence and the element position sequence described in the embodiments of this application.

[0091] like Figure 3 As shown, the sequence of interface elements is converted into a sequence of element positions. Figure 3 The input sequence is the sequence of interface elements, and the output sequence is the sequence of element positions.

[0092] right Figure 3 The input element attributes are explained as follows: name represents the tag name; content represents the content; id represents the element ID (identifier); preEle represents the identifier of the preceding element; sucELe represents the identifier of the following element; initX represents the horizontal coordinate of the top left corner; initY represents the vertical coordinate of the top left corner; width represents the width; and height represents the height.

[0093] right Figure 3 The output element attributes are explained as follows: id represents the element identifier; top represents the vertical distance from the top of the screen; bottom represents the vertical distance from the bottom of the screen; left represents the horizontal distance from the left side of the screen; z-index represents the stacking property; position represents the positioning property (the positioning property is absolute positioning, and the positioning is based on the parent of the positioned element until the root page element is found. Absolutely positioned elements will be removed from the document flow, thus defining the element position relative to the root page); text-align represents the horizontal alignment of the element content.

[0094] Figure 4This is a schematic diagram of the initial page layout of the interface described in the embodiments of this application.

[0095] like Figure 4 As shown in the example, this application embodiment takes the marketing poster production scenario as an example. The interface is a marketing poster, and the interface elements include a background image, a main title, a subtitle, text content, and a QR code. The background image has a stacking attribute of 1 and is placed at the bottom layer, while the other interface elements have a stacking attribute of 2 and are placed on the top layer.

[0096] 102. When the interface is in edit mode, the initial position is adjusted through the position adjustment event to obtain the target position of the target interface element in the interface.

[0097] It should be noted that the interface modes include browsing mode and editing mode, with browsing mode being the default. Users can only adjust the position of interface elements when the interface is in editing mode. Editing mode can be activated via specific gestures (such as double-clicking) or voice commands to prevent accidental position adjustments during normal use. The position adjustment event will first determine if the user is in editing mode; position adjustments can only be performed when the interface is in editing mode.

[0098] It should also be noted that position adjustment events include touch events and voice events. Touch events are those that adjust the position of interface elements through touch actions, including touchstart, touchmove, and touchend events. When the interface is in browsing mode, the above three touch events of the interface elements are listened for. When triggered, the selectEl (touch selection) handler function, moveEl (touch movement) handler function, and upEl (touch end) handler function are executed respectively, with two parameters passed in: the first is the currently clicked interface element, and the second is the current interface mode.

[0099] The three touch events mentioned above can be JavaScript touch events, implemented through the browser's Touch API. When a user interacts with a touchscreen device, the browser captures these touch events. Specifically, the `touchstart` event is triggered when a finger touches the screen; the `touchmove` event is triggered when a finger slides across the screen; and the `touchend` event is triggered when a finger leaves the screen.

[0100] Voice events refer to adjusting the position of interface elements via voice commands, including voice start events, voice move events, and voice end events. When the interface is in browsing mode, the above three voice events of the interface elements are listened for. When triggered, the selectEl (voice selection) processing function, moveEl (voice move) processing function, and upEl (voice end) processing function are executed respectively, and two parameters are passed in: the first is the currently selected interface element, and the second is the current interface mode.

[0101] Figure 5 This is a schematic diagram of the interface described in the embodiment of this application in edit mode.

[0102] like Figure 5 As shown, when the interface is in edit mode, you can drag and drop interface elements, and manually modify the stacking properties and other element properties of interface elements.

[0103] In one embodiment, optionally, the position adjustment event includes: a position adjustment start event, a position movement event, and a position adjustment end event;

[0104] The initial position is adjusted using a position adjustment event to obtain the target position of the target interface element within the interface, including:

[0105] The initial position of the control point is obtained by adjusting the position and starting the event.

[0106] Based on the initial position of the control point and the initial position, the control point offset is obtained, and the control point offset is used to indicate the offset of the control point relative to the initial position;

[0107] The real-time position of the control point is obtained through the position movement event;

[0108] Based on the real-time position of the control point and the offset of the control point, the real-time position of the target interface element during the position movement process is obtained;

[0109] The target position of the target interface element in the interface is obtained by using the position adjustment end event.

[0110] It should be noted that position adjustment events include touch events and voice events; position adjustment start events include touch start events and voice start events; position movement events include touch movement events and voice movement events; and position adjustment end events include touch end events and voice end events. It can be understood that a control point is the point selected on the target interface element in a position adjustment event. Moving this control point adjusts the position of the target interface element within the interface. In touch events, the control point is the touch point, or simply the touch point. Therefore, in touch events, the touch point can replace the control point; that is, the initial position of the control point is the initial position of the touch point, the offset of the control point is the offset of the touch point, and the real-time position of the control point is the real-time position of the touch point.

[0111] In this embodiment of the application, taking a position adjustment event as a touch event, a position adjustment start event as a touch start event, a position movement event as a touch movement event, and a position adjustment end event as a touch end event as an example, the method of adjusting the initial position through position adjustment events to obtain the target position of the target interface element in the interface is explained:

[0112] In the position adjustment event, the position of the target interface element is adjusted in real time. Figure 6 This is a schematic diagram showing the initial position of the target interface element in the interface according to an embodiment of this application.

[0113] Based on the `left` and `top` properties included in the initial position (the `left` property returns the distance from the target element to the left side of the screen, and the `top` property returns the distance from the target element to the top of the screen), obtain the distance string, which includes the unit "px" in addition to the number. Parse the string of the initial position of the target element, extract the value before the px unit, and store it in the initialized horizontal axis coordinate `initX` and initialized vertical axis coordinate `initY` (`initX`: the horizontal coordinate of the top left corner, which is the horizontal distance to the right from the origin of the screen; `initY`: the vertical coordinate of the top left corner, which is the vertical distance downward from the origin of the screen; the origin of the screen is located at the top left corner of the screen).

[0114] Here, for Figure 6 The markings in the text are explained:

[0115] 1 - Graphical user interface (i.e., interface); 2 - Interface origin, located at the top left corner of the interface; 3 - Top left corner origin of the target interface element; S1 - Top left horizontal coordinate; S2 - Top left vertical coordinate.

[0116] In the touch start event, the selectEl handler function is executed. First, it checks whether the current interface is in edit mode. If not, the function exits directly without executing subsequent logic. Otherwise, if it is, the initial position of the control point is obtained based on the target interface element. This initial position includes the initial x-coordinate of the control point (i.e., the X-coordinate of the control point relative to the left boundary of the interface) and the initial y-coordinate of the control point (i.e., the Y-coordinate of the control point relative to the top boundary of the interface).

[0117] Specifically, the page coordinates are obtained through the `targetTouches[0].pageX` and `targetTouches[0].pageY` properties. `targetTouches` retrieves a list of data representing the finger currently touching the DOM element. The first item in the list is retrieved, along with its `pageX` and `pageY` properties, which are used to obtain the initial x and y coordinates of the control point.

[0118] Figure 7 This is a schematic diagram of the control point offset described in the embodiments of this application.

[0119] Here, for Figure 7 The markings in the text are explained:

[0120] 2 - Target interface element; 5 - Control point; S3 - Initial horizontal coordinate of control point; S4 - Initial vertical coordinate of control point; S5 - Horizontal offset of control point; S6 - Vertical offset of control point.

[0121] Based on the initial position of the control point and the initial position of the target interface element, calculate the control point offset, which is the offset of the control point relative to the origin of the target interface element (the upper left corner of the target interface element is the origin). This control point offset includes the horizontal offset and the vertical offset of the control point.

[0122] Specifically, the horizontal offset of the control point = the initial x-coordinate of the control point - the initial x-coordinate (initX) of the target interface element;

[0123] Vertical offset of control point = initial y-coordinate of control point - initial y-coordinate of target interface element (initY).

[0124] In the touch move event, the moveE1 handler function is executed. First, it checks whether the current interface is in edit mode. If so, the function exits directly without executing subsequent logic. Otherwise, if it is in edit mode, the real-time position of the control point is obtained based on the target interface element. This real-time position includes the real-time horizontal coordinate of the control point (i.e., the X coordinate of the control point relative to the left boundary of the interface) and the real-time vertical coordinate of the control point (i.e., the Y coordinate of the control point relative to the top boundary of the interface).

[0125] Based on the real-time position and offset of the control point, the real-time position of the target interface element during the position movement process is obtained.

[0126] Specifically, the real-time horizontal coordinate of the target interface element during its position movement is equal to the real-time horizontal coordinate of the control point minus the horizontal offset of the control point (Fx).

[0127] The real-time ordinate of the target interface element during its position movement = the real-time ordinate of the control point - the vertical offset of the control point (Fy).

[0128] When the touch ends, the `upEl` handler function is executed. First, it checks if the current interface is in edit mode. If not, the function exits directly without executing subsequent logic. Otherwise, if it is, it retrieves the target position of the target interface element within the interface—that is, the position of the target interface element at the end of the touch. This target position is obtained in the same way as the real-time position: based on the real-time position and offset of the control point at the end of the touch, the target position is obtained, including both horizontal and vertical coordinates. This target position is saved in the target interface element's attributes for subsequent editing operations.

[0129] Figure 8 This is a schematic diagram illustrating the movement of the target interface element described in the example of this application.

[0130] Here, for Figure 8 The markings in the text are explained:

[0131] 6 - Target interface element after position adjustment; S5 - Horizontal offset of control point; S6 - Vertical offset of control point; S7 - Real-time horizontal coordinate of control point; S8 - Real-time vertical coordinate of control point; S9 - Target horizontal coordinate of target interface element; S10 - Target vertical coordinate of target interface element.

[0132] It should be noted that when the position adjustment event is a voice event, the position adjustment start event is a voice start event, the position movement event is a voice movement event, and the position adjustment end event is a voice end event, the specific method of adjusting the initial position through the position adjustment event to obtain the target position of the target interface element in the interface is similar to the above process, and will not be repeated here.

[0133] It should also be noted that the target interface elements directly respond to the user's touch actions or voice commands to perform dynamic position adjustments.

[0134] In one embodiment, optionally, the above method further includes:

[0135] Obtain the boundary detection results of the target location;

[0136] Based on the boundary detection results, determine whether to update the target position and obtain the determination result;

[0137] Based on the judgment result, the target location is determined.

[0138] In this embodiment of the application, during the process of performing the above-mentioned dynamic position adjustment on the target interface element, it is necessary to obtain the boundary detection results of the real-time position of the target interface element and the boundary detection results of the target position; based on the boundary detection results, it is determined whether to update the real-time position and the target position, and a determination result is obtained; if the determination result is to update the real-time position and / or the target position, then the real-time position is determined to be the updated real-time position, and / or the target position is determined to be the updated target position, thereby ensuring that the target interface element does not exceed the interface range.

[0139] Optionally, the boundary detection results include at least one of the following: left boundary detection result; right boundary detection result; upper boundary detection result; lower boundary detection result.

[0140] Figure 9 This is a schematic diagram of the right boundary detection results described in an embodiment of this application. Figure 9 As shown, the target interface element is a QR code, which extends beyond the right boundary.

[0141] In one embodiment, the target position may include an abscissa relative to the left boundary of the interface and a ordinate relative to the top boundary of the interface.

[0142] Obtain the boundary detection results of the target location, including at least one of the following:

[0143] Determine whether the x-coordinate of the target location is less than 0 to obtain the left boundary detection result of the target location;

[0144] Determine whether the sum of the horizontal coordinate of the target location and the width of the target interface element is greater than the interface width to obtain the right boundary detection result of the target location;

[0145] Determine whether the ordinate of the target location is less than 0 to obtain the upper boundary detection result of the target location;

[0146] Determine whether the sum of the ordinate of the target location and the height of the target interface element is greater than the interface height to obtain the lower boundary detection result of the target location.

[0147] In one implementation, optionally, when the determination result is to update the target location, determining the target location includes at least one of the following:

[0148] The x-coordinate of the target location is determined to be 0;

[0149] The x-coordinate of the target location is determined to be the interface width minus the target interface element width;

[0150] The ordinate of the target location is set to 0;

[0151] The vertical coordinate of the target location is determined as the interface height minus the height of the target interface element.

[0152] Here, combining the two implementation methods described above, we will specifically explain how the judgment result leads to updating the target location and determining the target location. It should be noted that the method for determining the real-time location is the same as the method for determining the target location, and will not be repeated here.

[0153] Figure 10 This is a schematic diagram showing the target interface element extending beyond the left boundary, as described in an embodiment of this application. Figure 10 As shown, if the x-coordinate of the target position of the target interface element is less than 0, it means that the detection result of the left boundary of the target position exceeds the left boundary of the interface. The x-coordinate of the target position needs to be set to 0 to prevent it from continuing to move to the left and exceeding the left boundary of the interface.

[0154] Here, for Figure 10 The markings in the diagram are explained as follows: S9 - the horizontal coordinate of the target location.

[0155] Figure 11 This is a schematic diagram showing the target interface element extending beyond the right boundary as described in an embodiment of this application. Figure 11 As shown, if the sum of the x-coordinate of the target position of the target interface element and the width of the target interface element is greater than the width of the interface, it means that the right boundary detection result of the target position exceeds the right boundary of the interface. The x-coordinate of the target position needs to be set to the width of the interface minus the width of the target interface element to prevent it from continuing to move to the right and exceeding the right boundary of the interface.

[0156] Here, for Figure 11 The markings in the diagram are explained as follows: S9 - x-coordinate of the target location; S11 - width of the target interface element; S13 - interface width.

[0157] Figure 12 This is a schematic diagram showing the target interface element extending beyond the upper boundary as described in an embodiment of this application. Figure 12 As shown, if the ordinate of the target position of the target interface element is less than 0, it means that the upper boundary detection result of the target position exceeds the upper boundary of the interface. The ordinate of the target position needs to be set to 0 to prevent it from continuing to move upward and exceeding the lower boundary of the interface.

[0158] Here, for Figure 12The markings in the diagram are explained as follows: S10 - the vertical coordinate of the target location.

[0159] Figure 13 This is a schematic diagram showing the target interface element extending beyond the lower boundary as described in an embodiment of this application. Figure 13 As shown, if the sum of the ordinate of the target position and the height of the target interface element is greater than the interface height, it means that the bottom boundary detection result of the target position exceeds the bottom boundary of the interface. The ordinate of the target position needs to be set to the interface height minus the height of the target interface element to prevent it from moving further down and exceeding the bottom boundary of the interface.

[0160] Here, for Figure 13 The markings in the diagram are explained as follows: S10 - the vertical coordinate of the target location; S12 - the height of the target interface element; S14 - the interface height.

[0161] It should be noted that after obtaining the target position of the target interface element within the interface, the horizontal and vertical coordinates of the target position are concatenated with the unit "px", and the `left` and `top` properties of the target interface element are updated. As long as the position movement event continues to trigger, the target coordinates will be continuously updated to the element's properties in real time, achieving real-time updates and smooth movement of the element's position. It is understood that this application provides a novel boundary detection algorithm that considers the influence of interface element size on the boundary, dynamically adjusting the position of the interface element by calculating its width and height, ensuring that the element remains fully visible even near the interface edge, thus improving the stability and aesthetics of the layout.

[0162] The following describes the application scenarios of the interface element position adjustment method described in the embodiments of this application:

[0163] Let's say we're creating a campus marketing poster that includes a background image, a main title "XX University Poster Main Title", body text, a QR code linking to the event details page, and a "Register Now" button.

[0164] First, the initial page layout, automatically generated by a deep learning model, features a main title at the top center of the poster, followed by the main text, a QR code in the center of the poster (the element with the highest weight), and buttons at the bottom center. The overall layout is clear and attractive.

[0165] Then, if users wish to adjust the position of the QR code, they simply need to activate the page in edit mode and drag it to change its position. The system will update the QR code's position in real time, while ensuring that other interface elements are reasonably positioned to avoid it, maintaining the overall aesthetics of the poster.

[0166] After making all adjustments, users can save the current layout as the final version.

[0167] Figure 14 This is a schematic diagram of the architecture of the interface element position adjustment system described in this application embodiment. This application embodiment provides an application system for the interface element position adjustment method, namely, an interface element position adjustment system. Please refer to... Figure 14 The interface element position adjustment system described in this application is applicable to GUI (Graphical User Interface) in edit mode. It automatically generates an initial page layout using a deep learning model and implements the movement of interface elements using native JavaScript technology. Using initial position reference points, it precisely calculates and adjusts element positions through touch events (selection, movement, release), ensuring smooth movement of elements within the interface while limiting the possibility of them exceeding interface boundaries.

[0168] Figure 15 This is a flowchart illustrating the interface element position adjustment system described in an embodiment of this application. Figure 15 As shown, in the context of a business scenario involving the intelligent generation of marketing posters, the main process of the system is described as follows:

[0169] 1501, Create a new marketing poster name with a business scenario description (e.g., XX University Enrollment Marketing Poster);

[0170] 1502 uses a deep learning model to generate an element position sequence based on the input sequence of interface elements and their context information, and automatically generates an initial page layout.

[0171] 1503, the interface defaults to browsing mode, listens for double-click events of elements (dblclick: triggered when an element is double-clicked), and sets the current interface mode to edit mode when triggered.

[0172] 1504: When the touch moves, adjust the position of the interface elements in real time according to steps 101 and 102 above.

[0173] In summary, this application provides a method for adjusting the position of interface elements, applicable to graphical user interfaces on touchscreen devices. This method can be implemented using native JavaScript, including: the graphical user interface defaults to browsing mode, meaning only interface elements can be viewed and not edited; an editing mode is activated through a specific gesture (double-tap), allowing dynamic adjustment of element positions. This native JavaScript implementation avoids compatibility issues between different devices when loading third-party plugins, while achieving optimal performance, significantly improving the smoothness of dynamic adjustment of interface elements, enabling smooth movement of interface elements, and maintaining intuitive and convenient operation.

[0174] The Seq2Seq model is chosen, and the model is trained using a training dataset containing UI element position information. The deep learning model generates the initial positions of UI elements based on the input UI elements and their context information. These initial positions include stacking attributes to indicate the stacking order. During page initialization, the initial layout of UI elements is automatically generated based on element weight ordering, saving users significant time spent manually initializing multiple elements, improving the efficiency of UI element operations, and greatly enhancing the user experience.

[0175] Functions are defined to handle touch start, touch movement, and touch end events for UI elements. The touch start event handler determines the initial position of the UI element by parsing the initial position string and converting it into numerical coordinates, which serve as the reference for position adjustment. The touch movement event handler directly follows the user's touch action, calculating the element's displacement relative to its initial position in real time and executing a dynamic position adjustment algorithm to ensure the element does not exceed the screen boundaries. The touch end event handler calculates the new position of the UI element in real time based on the initial position and displacement value, updates the element's position attributes, and achieves smooth movement of the UI element. This precise calculation and adjustment of element position ensures smooth movement on the screen while limiting the possibility of exceeding boundaries. This not only improves the naturalness of the interaction but also guarantees the consistency and stability of the interface.

[0176] Please refer to Figure 16 This application embodiment also provides a device for adjusting the position of interface elements, including:

[0177] The first acquisition module 1601 is used to acquire the initial position of the interface element in the interface according to the pre-trained deep learning model and the interface element sequence. The initial position includes a stacking attribute, which is used to indicate the stacking order of the interface element in the interface.

[0178] The first obtaining module 1602 is used to adjust the initial position through a position adjustment event when the interface is in edit mode, so as to obtain the target position of the target interface element in the interface.

[0179] Optionally, in the interface element position adjustment device, the first acquisition module 1601 is specifically used for:

[0180] Based on the pre-trained deep learning model, the sequence of interface elements, and the context information corresponding to each interface element, an element position sequence is output; wherein, the interface element sequence includes at least one of the following attributes of each interface element: the x-coordinate of the top left corner, the y-coordinate of the top left corner, the width, and the height; the context information corresponding to each interface element includes the attributes of adjacent interface elements; and the element position sequence includes the initial position of each interface element.

[0181] Based on the element position sequence, obtain the initial position of the target interface element in the interface.

[0182] Optionally, the interface element position adjustment device, wherein the deep learning model includes:

[0183] The sequence-to-sequence model and the recurrent neural network are used to generate the x-coordinate and y-coordinate of each interface element based on the interface element sequence. The recurrent neural network is used to generate the element weight of each interface element and generate the element position sequence based on the x-coordinate, y-coordinate and element weight of each interface element. The stacking attribute in the initial position of each interface element in the element position sequence is related to the element weight.

[0184] Optionally, in the interface element position adjustment device, the position adjustment event includes: a position adjustment start event, a position movement event, and a position adjustment end event;

[0185] The first obtaining module 1602 is specifically used for:

[0186] The initial position of the control point is obtained by adjusting the position and starting the event.

[0187] Based on the initial position of the control point and the initial position, the control point offset is obtained, and the control point offset is used to indicate the offset of the control point relative to the initial position;

[0188] The real-time position of the control point is obtained through the position movement event;

[0189] Based on the real-time position of the control point and the offset of the control point, the real-time position of the target interface element during the position movement process is obtained;

[0190] The target position of the target interface element in the interface is obtained by using the position adjustment end event.

[0191] Optionally, the interface element position adjustment device further includes:

[0192] The second acquisition module is used to obtain the boundary detection result of the target location;

[0193] The judgment module is used to determine whether to update the target position based on the boundary detection result, and obtain the judgment result;

[0194] The determination module is used to determine the target location based on the judgment result.

[0195] Optionally, in the interface element position adjustment device, the target position includes an abscissa relative to the left boundary of the interface and a ordinate relative to the top boundary of the interface.

[0196] The second obtaining module is specifically used for at least one of the following:

[0197] Determine whether the x-coordinate of the target location is less than 0 to obtain the left boundary detection result of the target location;

[0198] Determine whether the sum of the horizontal coordinate of the target location and the width of the target interface element is greater than the interface width to obtain the right boundary detection result of the target location;

[0199] Determine whether the ordinate of the target location is less than 0 to obtain the upper boundary detection result of the target location;

[0200] Determine whether the sum of the ordinate of the target location and the height of the target interface element is greater than the interface height to obtain the lower boundary detection result of the target location.

[0201] Optionally, in the aforementioned interface element position adjustment device, when the determination result indicates that the target position should be updated, the determining module specifically includes at least one of the following:

[0202] The x-coordinate of the target location is determined to be 0;

[0203] The x-coordinate of the target location is determined to be the interface width minus the target interface element width;

[0204] The ordinate of the target location is set to 0;

[0205] The vertical coordinate of the target location is determined as the interface height minus the height of the target interface element.

[0206] It should be noted that the device in this embodiment is the same as the above-described method for adjusting the position of interface elements. The implementation methods in the above embodiments are all applicable to the embodiments of this device and can achieve the same technical effect.

[0207] This application also provides a device for adjusting the position of interface elements, such as... Figure 17 As shown, it includes:

[0208] The processor 1701, memory 1702, transceiver 1703, and programs or instructions stored in the memory 1702 and executable on the processor 1701; when the processor 1701 executes the programs or instructions, it implements the various processes of the above-described multi-task speech large language model training method embodiment and achieves the same technical effect. To avoid repetition, these will not be described again here.

[0209] The transceiver 1703 is used to receive and send data under the control of the processor 1701.

[0210] Among them, Figure 17 In this context, the bus architecture can include any number of interconnected buses and bridges, specifically linking various circuits of one or more processors represented by processor 1701 and memory represented by memory 1702 together. The bus architecture can also link various other circuits such as peripheral devices, voltage regulators, and power management circuits, which are well known in the art and therefore will not be described further herein. The bus interface provides an interface. The transceiver 1703 can be multiple elements, including transmitters and receivers, providing a unit for communicating with various other devices over a transmission medium. For different user equipment, the user interface 1704 can also be an interface capable of connecting external or internal devices, including but not limited to keypads, displays, speakers, microphones, joysticks, etc.

[0211] Processor 1701 is responsible for managing the bus architecture and general processing, while memory 1702 can store the data used by processor 1701 when performing operations.

[0212] This application also provides a computer-readable storage medium storing a computer program. When executed by a processor, the computer program implements the various processes of the above-described interface element position adjustment method embodiments and achieves the same technical effects. To avoid repetition, it will not be described again here. The computer-readable storage medium may be a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.

[0213] This application also provides a computer program product, including computer instructions. When the computer instructions are executed by a processor, they implement the various processes of the above-described interface element position adjustment method embodiment and achieve the same technical effect. To avoid repetition, they will not be described again here.

[0214] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0215] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. The computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), and includes several instructions to cause a terminal (which may be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in the various embodiments of this application.

[0216] The embodiments of this application have been described above with reference to the accompanying drawings. However, this application is not limited to the specific embodiments described above. The specific embodiments described above are merely illustrative and not restrictive. Those skilled in the art, under the guidance of this application, can make many other modifications without departing from the spirit and scope of the claims, all of which fall within the protection scope of this application.

[0217] The above description represents the preferred embodiments of the present invention. It should be noted that those skilled in the art can make various improvements and modifications without departing from the principles of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.

Claims

1. A method for adjusting the position of interface elements, characterized in that, include: Based on the pre-trained deep learning model and the sequence of interface elements, the initial position of the interface elements in the interface is obtained. The initial position includes a stacking attribute, which is used to indicate the stacking order of the interface elements in the interface. When the interface is in edit mode, the initial position is adjusted through the position adjustment event to obtain the target position of the target interface element in the interface; Based on the pre-trained deep learning model and the sequence of interface elements, obtain the initial positions of the interface elements in the interface, including: Based on the pre-trained deep learning model, the sequence of interface elements, and the context information corresponding to each interface element, an element position sequence is output; wherein, the interface element sequence includes at least one of the following attributes of each interface element: the x-coordinate of the top left corner, the y-coordinate of the top left corner, the width, and the height; the context information corresponding to each interface element includes the attributes of adjacent interface elements; and the element position sequence includes the initial position of each interface element. Based on the element position sequence, obtain the initial position of the target interface element in the interface; The deep learning model includes: The sequence-to-sequence model and the recurrent neural network are used to generate the x-coordinate and y-coordinate of each interface element based on the interface element sequence. The recurrent neural network is used to generate the element weight of each interface element and generate the element position sequence based on the x-coordinate, y-coordinate and element weight of each interface element. The stacking attribute in the initial position of each interface element in the element position sequence is related to the element weight.

2. The method for adjusting the position of interface elements according to claim 1, characterized in that, The position adjustment events include: position adjustment start event, position movement event, and position adjustment end event; The initial position is adjusted using a position adjustment event to obtain the target position of the target interface element within the interface, including: The initial position of the control point is obtained by adjusting the position and starting the event. Based on the initial position of the control point and the initial position, the control point offset is obtained, and the control point offset is used to indicate the offset of the control point relative to the initial position; The real-time position of the control point is obtained through the position movement event; Based on the real-time position of the control point and the offset of the control point, the real-time position of the target interface element during the position movement process is obtained; The target position of the target interface element in the interface is obtained by using the position adjustment end event.

3. The method for adjusting the position of interface elements according to claim 1, characterized in that, The method further includes: Obtain the boundary detection results of the target location; Based on the boundary detection results, determine whether to update the target position and obtain the determination result; Based on the judgment result, the target location is determined.

4. The method for adjusting the position of interface elements according to claim 3, characterized in that, The target location includes an abscissa relative to the left boundary of the interface and a ordinate relative to the top boundary of the interface. Obtain the boundary detection results of the target location, including at least one of the following: Determine whether the x-coordinate of the target location is less than 0 to obtain the left boundary detection result of the target location; Determine whether the sum of the horizontal coordinate of the target location and the width of the target interface element is greater than the interface width to obtain the right boundary detection result of the target location; Determine whether the ordinate of the target location is less than 0 to obtain the upper boundary detection result of the target location; Determine whether the sum of the ordinate of the target location and the height of the target interface element is greater than the interface height to obtain the lower boundary detection result of the target location.

5. The method for adjusting the position of interface elements according to claim 3, characterized in that, If the determination result is to update the target location, the target location is determined by at least one of the following: The x-coordinate of the target location is determined to be 0; The x-coordinate of the target location is determined to be the interface width minus the target interface element width; The ordinate of the target location is set to 0; The vertical coordinate of the target location is determined as the interface height minus the height of the target interface element.

6. A device for adjusting the position of an interface element, characterized in that, include: The first acquisition module is used to acquire the initial position of the interface elements in the interface based on the pre-trained deep learning model and the sequence of interface elements. The initial position includes a stacking attribute, which is used to indicate the stacking order of the interface elements in the interface. The first obtaining module is used to adjust the initial position through a position adjustment event when the interface is in edit mode, so as to obtain the target position of the target interface element in the interface. The first acquisition module is specifically used for: Based on the pre-trained deep learning model, the sequence of interface elements, and the context information corresponding to each interface element, an element position sequence is output; wherein, the interface element sequence includes at least one of the following attributes of each interface element: the x-coordinate of the top left corner, the y-coordinate of the top left corner, the width, and the height; the context information corresponding to each interface element includes the attributes of adjacent interface elements; and the element position sequence includes the initial position of each interface element. Based on the element position sequence, obtain the initial position of the target interface element in the interface; The deep learning model includes: The sequence-to-sequence model and the recurrent neural network are used to generate the x-coordinate and y-coordinate of each interface element based on the interface element sequence. The recurrent neural network is used to generate the element weight of each interface element and generate the element position sequence based on the x-coordinate, y-coordinate and element weight of each interface element. The stacking attribute in the initial position of each interface element in the element position sequence is related to the element weight.

7. A device for adjusting the position of an interface element, characterized in that, include: A processor, a memory, and a program or instructions stored in the memory and executable on the processor, wherein the processor, when executing the program or instructions, implements the method for adjusting the position of interface elements as described in any one of claims 1 to 5.

8. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the method for adjusting the position of interface elements as described in any one of claims 1 to 5.

9. A computer program product, characterized in that, It includes computer instructions, which, when executed by a processor, implement the method for adjusting the position of interface elements as described in any one of claims 1 to 5.

Images