A zero-code multi-terminal application automatic construction method based on AI semantic understanding

Abstract

The present invention discloses a method for automatically constructing zero-code multi-terminal applications based on AI semantic understanding, including the following steps: receiving UI design draft images and layer metadata uploaded by users; extracting visual features and structural features respectively through a dual-branch feature extractor based on AI semantic understanding technology; generating an enhanced feature map by combining a cross-modal attention module and a void space pyramid pooling module; generating a high-precision UI component mask and an interaction dependency graph; generating dynamic weights through a multi-layer perceptron according to target terminal device parameters, adjusting the layout constraints of the interaction dependency graph and optimizing the layout objective function to minimize component overlap and visual imbalance; mapping UI components to atomic controls of the target platform based on a predefined control library and mapping rules, converting the interaction logic into an event-action chain, and generating target platform code. The present invention improves the automation level and cross-platform consistency of zero-code development by dynamically adapting to target terminal devices and automatically generating code.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence-driven zero-code development technology, and in particular to a method for automatically building zero-code multi-terminal applications based on AI semantic understanding. Background Art

[0002] In recent years, with the rapid development of artificial intelligence and software development technologies, zero-code development platforms and cross-platform application generation technologies have become research hotspots in the software engineering field. Traditional software development relies on manual coding, which is time-consuming and requires high developer expertise. To address this challenge, zero-code development platforms have significantly lowered the development barrier through visual design tools and automated code generation. For example, template-based low-code platforms (such as OutSystems and Mendix) allow users to quickly build application interfaces by dragging and dropping components and generate platform-specific code. Simultaneously, advances in computer vision and semantic understanding technologies have enabled automated parsing of UI design drafts. Deep learning-based image segmentation techniques (such as U-Net and DeepLab) can identify component boundaries in UI designs, while natural language processing and graph neural network technologies are used to analyze interaction logic and component relationships. Furthermore, responsive layout algorithms, with predefined breakpoints that adapt to different device resolutions and interaction methods, are facilitating the development of multi-device applications.

[0003] Although existing technologies have made significant progress in zero-code development and cross-platform adaptation, there are still some shortcomings, especially in the semantic understanding, dynamic layout optimization and automatic generation of interaction logic of complex UI design drafts. First, existing UI parsing methods mostly rely on single-modal image processing, which makes it difficult to effectively integrate the visual features of the design draft and the structured information of the layer metadata, resulting in insufficient recognition accuracy for nested components or dynamic controls, and widespread boundary fuzzy problems. Secondly, traditional responsive layouts rely on fixed breakpoints and lack the ability to dynamically adapt to the characteristics of the target device end, which can easily cause component overlap or visual imbalance, especially in scenarios with diverse devices (such as foldable screen phones and tablets). In addition, existing zero-code platforms usually require users to manually configure event triggers for interaction logic generation, with limited automation, making it difficult to extract complex interaction dependencies directly from design drafts. These shortcomings limit the universality and efficiency of zero-code development in complex application scenarios.

[0004] CN118587713A discloses a UI element parsing method, system, terminal, and medium for a human-computer interaction interface. The main solution is to perform semantic region segmentation on the human-computer interaction interface image and perform fine processing on the semantic region segmentation results to obtain the block structure information of the human-computer interaction interface. Functional UI elements are then located and identified in the human-computer interaction interface image, and text information within specific regions of the human-computer interaction interface image is extracted. Finally, the text information is matched with functional UI elements, and the functional UI elements are integrated with the block structure information. The nested relationship between functional UI elements is analyzed to obtain the UI element parsing results for the human-computer interaction interface. Although this invention mentions visual effects, it does not consider the layout problem of automatically generating UIs in diverse device scenarios.

[0005] CN118227124A discloses a Flutter code self-generation method, system, and device based on image analysis. The method mainly pre-processes the UI effect diagram, uses a convolutional neural network through supervised learning, and performs semantic segmentation on the UI effect diagram to obtain the structural information of the UI interface and the information of each UI component. At the same time, the UI components are subjected to unsupervised learning and optimization through a generative adversarial network until the generator and discriminator reach a stable state and the UI components meet the standard of the generated code setting value. Using the Q-learning algorithm in deep reinforcement learning, the generator learns interactively with the environment and converts the UI components into Flutter code when the code setting value is reached. This solution is only used to extract information and features from UI components, and does not take into account the interactive dependencies existing in the extracted UI components. Summary of the Invention

[0006] The purpose of this section is to summarize some aspects of the embodiments of the present invention and briefly introduce some preferred embodiments. Some simplifications or omissions may be made in this section and the abstract and title of this application to avoid obscuring the purpose of this section, the abstract and the title of the invention, and such simplifications or omissions should not be used to limit the scope of the present invention.

[0007] In view of the above existing problems, the present invention is proposed. Therefore, the present invention provides a zero-code multi-terminal application automatic construction method based on AI semantic understanding to solve the problems raised in the background technology.

[0008] To solve the above technical problems, the present invention provides the following technical solution: a method for automatically building zero-code multi-terminal applications based on AI semantic understanding, comprising:

[0009] Receive UI design draft images and layer metadata uploaded by users;

[0010] Based on AI semantic understanding technology, the UI design draft image and layer metadata are parsed to generate semantic representations of UI component masks and interaction dependency graphs;

[0011] According to the parameters of the target device and the semantic representation, the control mapping and interaction code of the target platform are automatically generated, thereby realizing zero-code cross-platform application construction.

[0012] As a preferred solution of the zero-code multi-terminal application automatic construction method based on AI semantic understanding described in the present invention, the layer metadata includes the ID, location, hierarchy and interaction annotation of the UI component.

[0013] As a preferred solution of the zero-code multi-terminal application automatic construction method based on AI semantic understanding described in the present invention, wherein: based on AI semantic understanding technology, the UI design draft image and layer metadata are parsed, including:

[0014] A dual-branch feature extractor is designed to extract visual features from the UI design draft image to obtain a visual feature map, and to extract structural features from the layer metadata to obtain a structural feature map.

[0015] As a preferred solution of the method for automatically building zero-code multi-terminal applications based on AI semantic understanding described in the present invention, it also includes:

[0016] Designing a cross-modal attention module to fuse the visual feature map and the structural feature map to generate a fused feature map;

[0017] A dilated spatial pyramid pooling module is applied to the fused feature map to generate an enhanced feature map.

[0018] As a preferred solution of the zero-code multi-terminal application automatic construction method based on AI semantic understanding described in the present invention, generating a semantic representation of the UI component mask and the interaction dependency graph includes:

[0019] Perform pixel-level classification based on the enhanced feature map to generate a UI component mask;

[0020] Combining the UI component mask with the layer metadata to construct an interaction dependency graph;

[0021] The nodes in the interaction dependency graph are represented as UI component instances, and the edges in the interaction dependency graph are represented as interaction logic and layout constraints between UI components.

[0022] As a preferred solution of the zero-code multi-terminal application automatic construction method based on AI semantic understanding described in the present invention, the parameters of the target terminal device include resolution, pixel density and interaction mode.

[0023] As a preferred solution of the method for automatically building zero-code multi-terminal applications based on AI semantic understanding described in the present invention, it also includes:

[0024] A context vector is generated based on the parameters of the target device, and a dynamic weight is generated through a multi-layer perceptron.

[0025] As a preferred solution of the zero-code multi-terminal application automatic construction method based on AI semantic understanding described in the present invention, the layout constraints in the interaction dependency graph are parameterized, the initial layout constraints are adjusted according to the context vector and the dynamic weight, and the adjusted layout parameters are generated.

[0026] As a preferred solution of the method for automatically building zero-code multi-terminal applications based on AI semantic understanding described in the present invention, it also includes:

[0027] Based on the adjusted layout parameters, a layout objective function is constructed to minimize the overlap rate and visual imbalance of UI components.

[0028] As a preferred solution of the method for automatically building zero-code multi-terminal applications based on AI semantic understanding described in the present invention, wherein: based on the parameters of the target terminal device and the semantic representation, the control mapping and interaction code of the target platform are automatically generated, including:

[0029] Based on the predefined control library and mapping rules, the components in the UI component mask are mapped to the atomic controls of the target platform;

[0030] The interaction logic in the interaction dependency graph is converted into an event-action chain, and the target platform code is obtained according to the grammatical rules of the target platform.

[0031] Compared with the prior art, the invention has the following beneficial effects:

[0032] 1. Compared with the deficiency of traditional single-modality image processing in accurately identifying nested components, this paper designs a dual-branch feature extractor to fuse visual feature maps and structural feature maps, combines the cross-modal attention mechanism and the Atrous Spatial Pyramid Pooling (ASPP) module to generate high-precision UI component masks and interaction dependency graphs (IDGs). This not only extracts the interaction dependencies between UI components, but also effectively solves the boundary ambiguity problem.

[0033] 2. To address the limitations of traditional responsive layouts, which rely on fixed breakpoints and result in poor adaptability, this invention uses a dynamic weight adjustment strategy (DWA) and a layout objective function to adaptively adjust layout parameters based on device context, reducing component overlap and visual imbalance. This approach is suitable for new devices and diverse UI scenarios (such as e-commerce and enterprise applications), ensuring the long-term applicability and universality of automatically generated code.

[0034] 3. Unlike the inefficient manual configuration of interaction triggers in traditional zero-code platforms, the solution of the present invention automatically extracts interaction dependencies from layer metadata, generates event-action chains and converts them into target platform code, thereby improving the automation level and cross-platform consistency of zero-code development. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] To more clearly illustrate the technical solutions of the embodiments of the present invention, the following briefly introduces the drawings required for describing the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. Those skilled in the art can also derive other drawings based on these drawings without inventive effort. Among them:

[0036] Figure 1 This is an overall flowchart of a method for automatically building zero-code multi-terminal applications based on AI semantic understanding, as described in one embodiment of the present invention. DETAILED DESCRIPTION

[0037] To make the above-mentioned objects, features, and advantages of the present invention more clearly understood, the following detailed description of the specific embodiments of the present invention is given in conjunction with the accompanying drawings. It is obvious that the described embodiments are only part of the embodiments of the present invention, not all of them. Based on the embodiments of the present invention, all other embodiments obtained by ordinary persons in this field without creative work should fall within the scope of protection of the present invention.

[0038] In the following description, many specific details are set forth to facilitate a full understanding of the present invention. However, the present invention may also be implemented in other ways different from those described herein. Those skilled in the art may make similar generalizations without violating the connotation of the present invention. Therefore, the present invention is not limited to the specific embodiments disclosed below.

[0039] Secondly, the term "one embodiment" or "embodiment" herein refers to a specific feature, structure, or characteristic that may be included in at least one implementation of the present invention. The phrase "in one embodiment" appearing in various places throughout this specification does not necessarily refer to the same embodiment, nor does it refer to a separate or selective embodiment that is mutually exclusive of other embodiments.

[0040] The present invention is described in detail with reference to schematic diagrams. For ease of illustration, cross-sectional views of device structures may be partially enlarged and not to scale when describing embodiments of the present invention. Furthermore, the schematic diagrams are merely illustrative and should not limit the scope of the present invention. Furthermore, in actual production, the three-dimensional dimensions of length, width, and depth should be included.

[0041] In the description of the present invention, it should be noted that the terms "upper, lower, inner, and outer" and other references to orientations or positional relationships are based on the orientations or positional relationships shown in the accompanying drawings and are intended solely to facilitate and simplify the description of the present invention. They are not intended to indicate or imply that the devices or components referred to must have, be constructed, or operate in a specific orientation, and therefore should not be construed as limitations on the present invention. Furthermore, the terms "first, second, or third" are used for descriptive purposes only and should not be construed as indicating or implying relative importance.

[0042] In this disclosure, unless otherwise specified or limited, the terms "mounted," "connected," and "connected" should be interpreted broadly. For example, they may refer to fixed, removable, or integral connections. They may also refer to mechanical, electrical, or direct connections, indirect connections through an intermediary, or internal communication between two components. Those skilled in the art will understand the specific meanings of these terms in this disclosure.

[0043] Example 1

[0044] Reference Figure 1 , which is the first embodiment of the present invention, provides a zero-code multi-terminal application automatic construction method based on AI semantic understanding, including:

[0045] S1. Receive UI design draft images and layer metadata uploaded by users;

[0046] Specifically, the UI design draft images uploaded by users must be in PNG (pixel-level) or SVG (vector-level) format;

[0047] Specifically, the layer metadata is in JSON format, including the UI component's ID, location, level, and interaction annotations;

[0048] Specifically, the ID of a UI component is a unique identifier for each type of UI component. For example, {"component_id":"btn_001"} indicates that the type of the current UI component is a button, and its identifier is 001.

[0049] Specifically, the position of the UI component is the bounding box coordinate, which consists of the upper left corner coordinates (x_min, y_min) and the lower right corner coordinates (x_max, y_max);

[0050] Specifically, the hierarchy of UI components is a parent-child relationship, that is, other UI components may exist within the current UI component. For example, {"parent":"container_001","children":["btn_001","txt_001"]} indicates that the parent object (parent) of the current UI component is a container component (container_001), which also includes the child objects (children) of the button component (btn_001) and the text box component (txt_001).

[0051] Specifically, the interaction annotation of the UI component is the user-specified interaction behavior, such as {"component_id":"btn_001","target":"list_001","click":"refresh_list"}, which means that the user specifies that by clicking the button component (btn_001), the target component (list_001) will be refreshed (refresh_list).

[0052] S2. Based on AI semantic understanding technology, it parses UI design draft images and layer metadata to generate semantic representations of UI component masks and interaction dependency graphs;

[0053] Furthermore, for UI design draft images, the pre-trained ResNet-50 is used to scale the image resolution of the UI design draft to 512×512;

[0054] Specifically, the pre-trained ResNet-50 in the solution of the present invention consists of several residual blocks, including an initial convolutional layer, batch normalization, ReLU activation, and a pooling layer. Batch normalization and ReLU activation are used to stabilize model training and enhance the model's expressiveness, respectively. They are not the innovative direction of the solution of the present invention and are therefore not described in detail.

[0055] It needs to be explained that ResNet-50 (Residual Network with 50 layers) is a deep convolutional neural network (CNN) model;

[0056] Preferably, ResNet-50 can solve the gradient vanishing problem in deep networks through residual connections (skip connections) and effectively extract multi-level features of images (from low-level edges to high-level semantic information). It is particularly important for the visual feature extraction of UI design drafts in the solution of the present invention, because UI design drafts contain complex visual elements (such as buttons, text boxes, icons, etc.) that require multi-scale feature support; and compared with shallow networks (such as VGG) or deeper networks (such as ResNet-101), ResNet-50 can achieve a balance between performance and computational efficiency, and is suitable for zero-code development scenarios with higher real-time requirements; in addition, although the higher resolution Increasing the resolution (e.g., 1024×1024) can further improve the accuracy of UI component extraction, but it will significantly increase the computational cost (memory and inference time). ResNet-50 generally uses 224×224 as the initial resolution of image input. Since the aforementioned UI design draft contains complex visual elements, the resolution needs to be increased to preserve these complex visual elements. Therefore, using 512×512 as the input image resolution is a compromise. This resolution is within the processing capability of ResNet-50, reducing processing complexity while preserving the details of UI component boundaries and interaction annotations.

[0057] Furthermore, a dual-branch feature extractor is designed to extract visual features from the UI design draft image to obtain a visual feature map, and to extract structural features from the layer metadata to obtain a structural feature map;

[0058] Specifically, the pre-trained ResNet-50 is used as the convolutional layer of the backbone network to extract visual features and output a visual feature map. ; Among them, the process of extracting visual features is as follows:

[0059] Input the scaled UI design image (512×512), and express the resolution of the image according to the RGB value of the image as , 3 represents RGB three channels;

[0060] Construct the initial convolution layer (Conv_f), set the convolution kernel of the initial convolution layer to: 7×7, the number of filters to 64 (the number of output channels), the stride to: 2, and the padding to: 3 (the same as the RGB three channels). The processing process is expressed by the following formula:

[0061]

[0062]

[0063] in, Indicates the height of the input UI design draft image. Indicates the height of the input UI design draft image. Represented as the width of the output feature map, Represented as the width of the output feature map, Indicates rounding down; , padding=3, Conv_kernel_size=7 means the size of the convolution kernel is 7, stride=2, and we get:

[0064]

[0065] Similarly, , and according to the number of filters, the output feature map is expressed as: ;

[0066] The output feature map will be Batch normalization is used to standardize the feature map of each channel, which can stabilize model training and accelerate convergence. At the same time, ReLU activation is used to perform nonlinearity on the output feature map to enhance the expressiveness of the model.

[0067] The feature map output by the maximum pooling layer , set the pooling kernel to: 3×3, stride to: 2, padding to: 1 (to retain edge information), the processing process calls the above and Formula, replace Conv_kernel_size with Pool_kernel_size, and get:

[0068] , padding=1, Pool_kernel_size=3 means the size of the pooling kernel is 3, stride=2, , similarly, , according to the number of filters (pooling does not change the number of channels), the output feature map is expressed as: ;

[0069] The training process is divided into four stages, each consisting of multiple residual blocks. Each residual block contains three convolutional layers (1×1, 3×3, and 1×1). The kernel size is fixed at 3, and the 1×1 convolutional layer is directly added to the output through a skip connection. Each residual block group is processed one by one as follows:

[0070] The first residual block group, Conv_RB1 (containing 3 residual blocks), input ;

[0071] Main path: 1×1 convolution layer: reduce dimension to 64 channels (64 filters), stride 1; 3×3 convolution layer: maintain 64 channels (number of filters), stride 1, padding 1; 1×1 convolution layer: increase dimension to 256 channels (256 filters), stride 1;

[0072] Residual path: Block 1: 1×1 convolutional layer (256 filters), increasing from 64 channels to 256 channels, stride 1, Blocks 2-3: add 256 channels directly to the output;

[0073] Among them, all convolutional layers of the main path and residual path have a stride of 1 and a padding of 1, resulting in:

[0074]

[0075] Similarly, ;

[0076] It should be noted that the main path is: 64→64 (1×1)→64 (3×3)→256 (1×1); the residual path is: 64→256 (1×1); at this time, we can get ;

[0077] The second residual block group, Conv_RB2 (containing 4 residual blocks), input ;

[0078] Main path (downsampling): 1×1 convolution layer: reduce dimension to 128 channels (128 filters), stride 1; 3×3 convolution layer: maintain 128 channels (128 filters), stride 2 (downsampling), padding 1; 1×1 convolution layer: increase dimension to 512 channels (512 filters), stride 1;

[0079] Residual path: Block 1: 1×1 convolution layer (512 filters), 256→512 channels, stride 2; Blocks 2–4: 1×1 convolution layer: 512 dimensionality reduction to 128 channels, stride 1; 3×3 convolution: 128 channels, stride 1, padding 1; 1×1 convolution layer: 128 dimensionality increase to 512 channels, stride 1;

[0080] Among them, the stride of the main path 3×3 convolution layer is 2:

[0081]

[0082] Similarly, ;

[0083] The residual path 1×1 convolution layer has a stride of 2:

[0084]

[0085] Similarly, ;

[0086] Among them, the 2nd to 4th blocks are the same as the 1st block, keeping ;

[0087] It should be noted that the main path is: 256→128 (1×1)→128 (3×3)→512 (1×1), and the residual path is: 256→512 (1×1); at this time, we can get ;

[0088] The third residual block group, Conv_RB3 (containing 6 residual blocks), input ;

[0089] Main path (downsampling): 1×1 convolution layer: 512→256 channels, stride 1; 3×3 convolution layer: 256 channels, stride 2, padding 1; 1×1 convolution layer: 256→1024 channels, stride 1;

[0090] Residual path: Block 1: 1×1 convolutional layer, 512→1024 channels, stride 2; Blocks 2–6: add 1024 channels directly to the output, stride 1;

[0091] Among them, the stride of the main path 3×3 convolution layer is 2:

[0092]

[0093] Similarly, ;

[0094] The residual path 1×1 convolution layer has a stride of 2:

[0095]

[0096] Similarly, ;

[0097] Among them, the 2nd to 6th blocks are the same as the 1st block, keeping ;

[0098] It should be noted that the main path is: 512→256 (1×1)→256 (3×3)→1024 (1×1); the residual path is: 512→1024 (1×1); at this time, we can get ;

[0099] The fourth residual block group, Conv_RB4 (containing 3 residual blocks), input ;

[0100] Main path (downsampling): 1×1 convolution layer: 1024→512 channels, stride 1; 3×3 convolution layer: 512 channels, stride 2, padding 1; 1×1 convolution layer: 512→2048 channels, stride 1;

[0101] Residual path: Block 1: 1×1 convolutional layer, 1024→2048 channels, stride 2; Blocks 2-3: add 2048 channels directly to the output, stride 1;

[0102] Among them, the stride of the main path 3×3 convolution layer is 2:

[0103]

[0104] Similarly, ;

[0105] The residual path 1×1 convolution layer has a stride of 2:

[0106]

[0107] Similarly, ;

[0108] Among them, the 2nd to 3rd blocks are the same as the 1st block, keeping ;

[0109] It should be noted that the main path is: 1024→512 (1×1)→512 (3×3)→2048 (1×1); the residual path is: 1024→2048 (1×1); at this time, we can get ;

[0110] Furthermore, structural features are extracted from the layer metadata to obtain a structural feature map. The processing flow is as follows:

[0111] Based on the graph convolutional network, the layer metadata is used as input to extract the ID, location, level and interaction annotation of the UI component as the initial features;

[0112] By inputting the position of each UI component, a two-dimensional sinusoidal position code is generated to reflect the spatial position of the UI component in the design draft. The two-dimensional sinusoidal position code is expressed as follows:

[0113]

[0114] in, Represents the position encoding vector, x and y represent the horizontal and vertical coordinates of the UI component respectively, d represents the dimension, and u represents the dimension index of the position encoding vector. Indicates the wavelength that determines the sine or cosine function;

[0115] It should be explained that because the convolution operation of ResNet-50 is insensitive to spatial position (convolution kernels share weights), the precise position information of pixels in the UI design may be lost. Position encoding, on the other hand, explicitly encodes the spatial position of each pixel by adding a 128-dimensional vector to it, thereby distinguishing UI components with the same appearance but different positions (such as two identical buttons in different areas).

[0116] The graph convolutional network uses a two-layer convolutional network. Each layer updates the initial features (layer 1: 128→256, layer 2: 256→256) and aggregates them through the second layer of convolutional network to 256 neighbor information, outputting a 256-dimensional structure embedding vector. ; Output Broadcast to the corresponding pixel area (based on the bounding box mapping to the 512×512 original input image) and resize down to 16×16 (same as the visual feature map Align), each pixel is aligned The dimension is filled into the corresponding UI component (if there is no UI component, fill it with a zero vector) to obtain the structural feature map ;

[0117] It should be noted that this 128 only considers the dimensions of the position encoding, and does not consider the UI component type encoding and the bounding box center coordinates;

[0118] Furthermore, by designing a cross-modal attention module, the visual feature map and structural feature maps Perform fusion and generate a fusion feature map;

[0119] Specifically, the cross-modal attention module is expressed as follows:

[0120]

[0121] in, is the output of the cross-modal attention module, and the output result is the fusion feature map , D is the number of input channels; Softmax is the attention weight; The query projection matrix maps the visual feature map into a query, which helps identify the semantic requirements of button pixels in UI design drafts, such as "This shape is a button, what is its function?" is the key projection matrix, which maps the structural feature map to a key for matching with the visual feature map; It is the value projection matrix, which maps the structural feature map to a value, which contains the actual content of the structural feature map; Expressed as a scaling factor, it is used to stabilize the gradient of Softmax and ensure that the distribution of attention weights is reasonable;

[0122] Furthermore, a dilated spatial pyramid pooling module is applied to the fused feature map to generate an enhanced feature map;

[0123] It should be explained that Atrous Spatial Pyramid Pooling (ASPP) is a feature extraction module that can capture multi-scale features through atrous convolution and multi-scale pooling, which is used to solve the problem of scale change of target objects in semantic segmentation. For the fused feature map that combines the visual feature map and the structural feature map, due to the limited receptive field, it is difficult to capture small UI components (such as labels) and large containers (nested multiple UI components) at the same time, and ResNet-50 requires multi-scale features for support. Therefore, calling the Atrous Spatial Pyramid Pooling module can not only enhance the fused feature map, but also capture small UI components and large containers at the same time.

[0124] Specifically, the expansion rate in the dilated spatial pyramid pooling is set to [1, 6, 12, 18], the size of all convolution kernels is fixed to 3, the output channel is 128, and the enhanced feature map is generated by splicing with the fusion feature map;

[0125] Furthermore, pixel-level classification is performed based on the enhanced feature map to generate UI component masks;

[0126] Specifically, transposed convolution is used to enlarge the enhanced feature map from 16×16 to 512×512 as follows:

[0127]

[0128] in, Represented as upsampling, is the output channel; Deconv represents the transposed convolution operation, Represented as a transposed convolution kernel;

[0129] Specifically, the features of each pixel are mapped to the category score through a 1×1 convolution classifier:

[0130]

[0131] in, Represented as a classification convolution kernel, K is the number of categories (container, icon, background);

[0132] It should be noted that using a 1×1 convolution classifier is equivalent to applying a fully connected layer to the feature vector of each pixel, and mapping it to a category score is equivalent to outputting the category probability of each pixel. For example, the pixel in the button area is predicted to be in the "button" category, and the pixel in the text box area is predicted to be in the "text box" category.

[0133] Specifically, output the category probability of each pixel:

[0134]

[0135] in, represents the probability that pixel (i, j) belongs to category k;

[0136] Specifically, the generated UI component mask Expressed as:

[0137]

[0138] in, is a multi-channel mask;

[0139] It should be noted that by generating the UI component mask, the UI component boundary of each pixel can be clearly identified;

[0140] Furthermore, the UI component mask and layer metadata are combined to build an interaction dependency graph;

[0141] It should be explained that the Interaction Dependency Graph (IDG) is a graph structure that is the key output of the zero-code platform and can transform static UI design drafts into dynamic, interactive multi-terminal applications.

[0142] Specifically, the interaction dependency graph is represented as:

[0143]

[0144] Among them, V represents the UI component instance (a button with functionality), and E represents the interaction logic between UI components (such as clicking a button to trigger form submission) and layout constraints (such as the button is to the right of the text box);

[0145] Specifically, for the interaction logic between UI components, such as "clicking a button" triggers "form submission", an edge is added between the button and the form (button to form);

[0146] It should be noted that the interaction logic is designed to add edges to UI components that have functional relationships, and not to those without functional relationships;

[0147] Specifically, for layout constraints, the connected area of each UI component category and the hierarchical information of the layer metadata are identified through the UI component mask, and the bounding box is calculated for each connected area to obtain the center point of the bounding box. , thereby inferring the spatial relationship of UI components, and its spatial relationship is calculated as follows:

[0148]

[0149] in, It is expressed as a distance threshold (in pixels); right means left, left means right, below means below, above means above, and include means include;

[0150] It should be noted that by constructing an interaction dependency graph, not only can the interaction dependency relationship between UI components be clearly displayed, but also the spatial position relationship between UI components can be inferred, thereby improving the accuracy of the interaction relationship between UI components;

[0151] S3. Automatically generate the control mapping and interaction code for the target platform based on the parameters and semantic representation of the target device, thereby achieving zero-code cross-platform application construction;

[0152] It should be noted that traditional methods lack the ability to dynamically adapt to the characteristics of target devices, which can easily lead to overlapping UI components or visual imbalance. This can result in poor UI component layout in diverse device scenarios. The present invention overcomes the limitations of traditional methods by dynamically optimizing layout parameters by considering the characteristics of the target device (such as the aspect ratio of the output device display and the interaction method).

[0153] Furthermore, the device’s resolution, pixel density, and interaction mode are obtained;

[0154] It should be noted that this interaction method refers specifically to the user's interaction with the device, not the interaction between UI components;

[0155] Specifically, the device resolution refers to the width w and height h of the target device;

[0156] Specifically, pixel density refers to the pixel density of the target device, measured in pixels per inch (PPI);

[0157] Specifically, the interaction mode refers to the user's operation of the device, such as touch or keyboard and mouse;

[0158] Furthermore, a context vector is generated based on the parameters of the target device, and dynamic weights are generated through a multi-layer perceptron.

[0159] Specifically, the context vector C is expressed as:

[0160]

[0161] in, is the pixel density, M is the interaction mode;

[0162] It should be explained that a multi-layer perceptron (MLP) is a feed-forward neural network consisting of multiple fully connected layers. It is used to learn nonlinear mappings from input features and output predicted values or weights. In the solution of the present invention, the MLP is used to process input features (i.e., context vectors) and output dynamic weights, which are used to adjust the layout constraints of the interaction dependency graph or the rendering parameters of UI components (such as button size).

[0163] Specifically, the multi-layer perceptron has a three-layer structure including an input layer, a hidden layer, and an output layer;

[0164] Specifically, the input layer of the multilayer perceptron receives the context vector and generates 128 hidden features to capture the "high PPI, small screen" pattern. The hidden layer then further refines the 128 hidden features generated by the input layer into 64 features to identify the need to zoom in or out of UI components. Finally, the output layer generates dynamic weights based on the hidden layer's refinement results, where the dynamic weights include the total number of adjustable parameters in the UI component layout.

[0165] Furthermore, the layout constraints in the interaction dependency graph are parameterized, and the initial layout constraints are adjusted according to the context vector and dynamic weights to generate adjusted layout parameters.

[0166] Specifically, each UI component is parameterized as follows: ,in, is the parameterized qth UI component, align is the alignment of the UI component, and margin is the margin of the UI component;

[0167] Specifically, the adjusted layout parameters are expressed as:

[0168]

[0169] in, Indicates the margin of the qth UI component, To adjust the margin of the qth UI component, Expressed as dynamic weight, Expressed as the target device width, Indicates the width of the UI component design draft;

[0170] Furthermore, based on the adjusted layout parameters, a layout objective function is constructed to minimize the overlap rate and visual imbalance of UI components;

[0171] It should be noted that the process of minimizing visual imbalance is the process of maintaining visual balance;

[0172] Specifically, the constructed layout objective function L is expressed as:

[0173]

[0174] in, Represent the bounding boxes of the qth UI component and the pth UI component respectively, q≠p; Expressed as and The intersection-over-union ratio is used to measure the degree of overlap of UI components. Expressed as The center coordinates of are recorded as ,otherwise ; Mvf represents the visual center of gravity, which is the weighted average of the center coordinates of all UI components; Expressed as a balance factor, it is used to adjust the balance between the degree of overlap and the visual center of gravity;

[0175] Specifically, the simulated annealing algorithm is used with an initial temperature of 100, a cooling rate of 0.95, and 100 iterations to continuously adjust the output of L so that it gradually approaches the global optimal solution, thereby minimizing L.

[0176] Furthermore, based on the predefined control library and mapping rules, the components in the UI component mask are mapped to atomic controls of the target platform;

[0177] It should be explained that atomic controls are standard controls, that is, the smallest UI elements that cannot be divided, such as HTML <button>SwiftUI Button

[0178] Specifically, the predefined control library includes control categories (such as buttons and text boxes), control properties (text content and style), event support (click, input), and platform adaptation (the implementation of each control on different platforms, such as HTML and Swift);

[0179] Specifically, the mapping rules include component category matching (such as the UI component mask "button" is mapped to< / button> <button>), the visual properties of the UI component mask (such as blue color, size 100×100) and semantic properties (such as the "submit" function of the UI component mask "button");

[0180] It should be noted that the actual code mapped to each target platform is different. Take the mapping rules as an example:

[0181] Web:< / button> <button style=""width:" 479px;background-color:blue;"> Submit< / button> ;

[0182] iOS: Button(label:{Text("Submit").frame(width:479px)});

[0183] Android: <button android:width=""479px"android:text="Submit" / ">;

[0184] Furthermore, the interaction logic in the interaction dependency graph is converted into an event-action chain, and the target platform code is obtained according to the grammatical rules of the target platform;

[0185] Specifically, event-action represents user operation events (such as clicks) and component operation response actions (such as submitting a form). Chains represent the events and actions that need to be executed when adding edges (such as a button to a form, attribute "trigger submit" → event "click" + action "submit"). Then, converting the edges of the interaction dependency graph into event-action chains can be obtained as follows: Chain 1: {event = "click button", action = "submit form"};

[0186] Specifically, the three commonly used target platform syntax rules are as follows:

[0187] Web: HTML (structure), CSS (style), JavaScript (interaction);

[0188] iOS: SwiftUI (structure + style + interaction), Swift (logic);

[0189] Android: XML (structure + style), Kotlin (logic + interaction);

[0190] It should be noted that by building an event-action chain, the event-action relationship between UI components can be quickly obtained, thereby improving the automation level of zero-code development and cross-platform consistency.

[0191] Those skilled in the art will appreciate that embodiments of the present invention may be provided as methods, systems, or computer program products. Therefore, the present application may take the form of a fully hardware embodiment, a fully software embodiment, or an embodiment combining software and hardware. Furthermore, the present application may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code. The solutions in the embodiments of the present application may be implemented using various computer languages, such as the object-oriented programming language Java and the interpreted scripting language JavaScript.

[0192] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0193] These computer program instructions may also be stored in a computer readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 a process or multiple processes and / or boxes Figure 1 The function specified in one or more boxes.

[0194] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operational steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing the instructions executed on the computer or other programmable device for implementing the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0195] Although the preferred embodiments of the present application have been described, those skilled in the art may make additional changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the present application.

[0196] Obviously, those skilled in the art may make various changes and modifications to this application without departing from the spirit and scope of this application. Thus, if these modifications and variations of this application fall within the scope of the claims of this application and their equivalents, this application is intended to include these modifications and variations.< / button>

Claims

1. A zero-code multi-terminal application automatic construction method based on AI semantic understanding, characterized by: include: Receive UI design draft images and layer metadata uploaded by users; Based on AI semantic understanding technology, the UI design draft image and layer metadata are parsed to generate semantic representations of UI component masks and interaction dependency graphs; The AI semantic understanding technology is used to analyze the UI design draft image and layer metadata, including: Design a dual-branch feature extractor to extract visual features from the UI design draft image to obtain a visual feature map, and extract structural features from the layer metadata to obtain a structural feature map; Also includes: Designing a cross-modal attention module to fuse the visual feature map and the structural feature map to generate a fused feature map; Applying a dilated spatial pyramid pooling module to the fused feature map to generate an enhanced feature map; Generating the semantic representation of the UI component mask and the interaction dependency graph includes: Perform pixel-level classification based on the enhanced feature map to generate a UI component mask; Combining the UI component mask with the layer metadata to construct an interaction dependency graph; Representing nodes in the interaction dependency graph as UI component instances, and representing edges in the interaction dependency graph as interaction logic and layout constraints between UI components; Automatically generate control mapping and interaction code for the target platform based on the parameters of the target device and the semantic representation, thereby achieving zero-code cross-platform application construction; The automatic generation of the control mapping and interaction code of the target platform according to the parameters of the target terminal device and the semantic representation includes: Based on the predefined control library and mapping rules, the components in the UI component mask are mapped to the atomic controls of the target platform; The interaction logic in the interaction dependency graph is converted into an event-action chain, and the target platform code is obtained according to the grammatical rules of the target platform.

2. The method for automatically building a zero-code multi-terminal application based on AI semantic understanding according to claim 1, characterized in that: The layer metadata includes the ID, location, level, and interaction annotation of the UI component.

3. The method for automatically building zero-code multi-terminal applications based on AI semantic understanding according to claim 1, characterized in that: The parameters of the target device include resolution, pixel density and interaction mode.

4. The method for automatically building a zero-code multi-terminal application based on AI semantic understanding according to claim 3 is characterized in that: Also includes: A context vector is generated based on the parameters of the target device, and a dynamic weight is generated through a multi-layer perceptron.

5. The method for automatically building a zero-code multi-terminal application based on AI semantic understanding according to claim 4, characterized in that: The layout constraints in the interaction dependency graph are parameterized, and the initial layout constraints are adjusted according to the context vector and the dynamic weight to generate adjusted layout parameters.

6. The method for automatically building zero-code multi-terminal applications based on AI semantic understanding according to claim 5, characterized in that: Also includes: Based on the adjusted layout parameters, a layout objective function is constructed to minimize the overlap rate and visual imbalance of UI components.

Images