Zero-code multi-terminal application automatic construction method based on AI semantic understanding

Abstract

The invention discloses an AI semantic understanding-based zero-code multi-terminal application automatic construction method, which comprises the following steps of: receiving a UI design draft image and layer metadata uploaded by a user, respectively extracting visual features and structural features through a double-branch feature extractor based on an AI semantic understanding technology, and establishing an AI semantic understanding model; a cross-modal attention module and a cavity space pyramid pooling module are combined to generate an enhanced feature graph, and a high-precision UI component mask and an interaction dependency graph are generated; according to target end equipment parameters, dynamic weights are generated through a multi-layer perceptron, layout constraints of the interaction dependency graph are adjusted, a layout target function is optimized, and component overlapping and visual unbalance are minimized; based on the predefined control library and the mapping rule, the UI component is mapped into the atomic control of the target platform, the interaction logic is converted into the event-action chain, and the target platform code is generated, and the automation degree and the cross-platform consistency of zero code development are improved through dynamic adaptation of the target end equipment and automatic code generation.

Description

Technical Field

[0001] The present invention relates to the technical field of artificial intelligence-driven zero-code development, and particularly to a zero-code multi-terminal application automatic construction method based on AI semantic understanding. Background Art

[0002] In recent years, with the rapid development of artificial intelligence and software development technologies, zero-code development platforms and cross-platform application generation technologies have become research hotspots in the field of software engineering. Traditional software development relies on manual coding, which is time-consuming and requires high professional skills of developers. To address this challenge, zero-code development platforms have significantly reduced the development threshold through visual design tools and automated code generation technologies. For example, template-based low-code platforms (such as OutSystems, Mendix) allow users to quickly build application interfaces by dragging and dropping components and generate code for specific platforms. At the same time, the progress of computer vision and semantic understanding technologies has made it possible to automatically parse UI design drafts. Deep learning-based image segmentation technologies (such as U-Net, DeepLab) can identify the boundaries of components in UI design drafts, while natural language processing and graph neural network technologies are used to parse interaction logic and component relationships. In addition, responsive layout algorithms adapt the resolution and interaction methods of different devices through predefined breakpoints, promoting the development of multi-terminal applications.

[0003] Although the existing technologies have made significant progress in zero-code development and cross-platform adaptation, there are still some deficiencies, especially in the semantic understanding of complex UI design drafts, dynamic layout optimization, and automated generation of interaction logic. First, existing UI parsing methods mostly rely on single-modal image processing and are difficult to effectively integrate the visual features of design drafts and the structured information of layer metadata, resulting in insufficient recognition accuracy for nested components or dynamic controls and widespread boundary blur problems. Second, traditional responsive layouts rely on fixed breakpoints and lack the ability to dynamically adapt to the characteristics of target device ends, easily causing component overlap or visual imbalance, especially in scenarios of diverse devices (such as foldable mobile phones, tablets). In addition, existing zero-code platforms usually require users to manually configure event triggers in the generation of interaction logic, with limited automation and difficulty in directly extracting complex interaction dependencies from design drafts. These deficiencies limit the universality and efficiency of zero-code development in complex application scenarios.

[0004] CN118587713A discloses a method, system, terminal and medium for parsing UI elements of a human-computer interaction interface. Its main solution is to perform semantic region segmentation on the human-computer interaction interface image, and perform fine processing on the result of the semantic region segmentation to obtain the block structure information of the human-computer interaction interface. Then, perform the positioning and recognition of functional UI elements on the human-computer interaction interface image, extract the text information in a specific area of the human-computer interaction interface image, and finally match the text information with the functional UI elements, and fuse the functional UI elements with the block structure information to analyze the nested relationship between the functional UI elements and obtain the UI element parsing result of the human-computer interaction interface. Although the invention mentions the visual effect, it does not consider the problem of automatically generating the UI layout in diverse device scenarios.

[0005] CN118227124A discloses a method, system and device for automatically generating Flutter code based on image analysis. It mainly preprocesses the UI effect diagram, uses a convolutional neural network to perform semantic segmentation on the UI effect diagram through supervised learning to obtain the structure information of the UI interface and the information of each UI component. At the same time, through a generative adversarial network, the UI components are optimized through unsupervised learning until the generator and discriminator reach a stable state, and the UI components reach the standard of the set value of the generated code. Using the Q-learning algorithm in deep reinforcement learning, the generator learns through interaction with the environment to convert the UI components into Flutter code when the set value of the code is reached. This solution is only used to extract information and features from UI components and does not consider the interaction dependency relationships existing in the extraction of UI components. Summary of the Invention

[0006] The purpose of this part is to outline some aspects of the embodiments of the present invention and briefly introduce some preferred embodiments. Simplifications or omissions may be made in this part, as well as in the abstract and title of the specification of this application, to avoid obscuring the purpose of this part, the abstract and the title of the specification, and such simplifications or omissions shall not be used to limit the scope of the present invention.

[0007] In view of the above existing problems, the present invention is proposed. Therefore, the present invention provides a zero-code multi-terminal application automatic construction method based on AI semantic understanding to solve the problems raised in the background technology.

[0008] To solve the above technical problems, the present invention provides the following technical solution: A zero-code multi-terminal application automatic construction method based on AI semantic understanding, including: Receiving a UI design draft image and layer metadata uploaded by a user; Based on AI semantic understanding technology, parsing the UI design draft image and layer metadata to generate semantic representations of UI component masks and interaction dependency graphs; Automatically generate control mappings and interaction codes for the target platform based on the parameters of the target device and the semantic representation, thereby realizing cross-platform application construction with zero code.

[0009] As a preferred solution of the zero-code multi-terminal application automatic construction method based on AI semantic understanding according to the present invention, wherein: the layer metadata includes the ID, position, level, and interaction annotation of the UI component.

[0010] As a preferred solution of the zero-code multi-terminal application automatic construction method based on AI semantic understanding according to the present invention, wherein: based on the AI semantic understanding technology, analyze the UI design draft image and the layer metadata, including: Design a dual-branch feature extractor to extract visual features from the UI design draft image respectively to obtain a visual feature map, and extract structural features from the layer metadata to obtain a structural feature map.

[0011] As a preferred solution of the zero-code multi-terminal application automatic construction method based on AI semantic understanding according to the present invention, wherein: further include: Design a cross-modal attention module to fuse the visual feature map and the structural feature map to generate a fused feature map; Apply an atrous spatial pyramid pooling module to the fused feature map to generate an enhanced feature map.

[0012] As a preferred solution of the zero-code multi-terminal application automatic construction method based on AI semantic understanding according to the present invention, wherein: generate the semantic representation of the UI component mask and the interaction dependency graph, including: Perform pixel-level classification based on the enhanced feature map to generate a UI component mask; Combine the UI component mask and the layer metadata to construct an interaction dependency graph; Represent the nodes in the interaction dependency graph as UI component instances, and represent the edges in the interaction dependency graph as the interaction logic and layout constraints between UI components.

[0013] As a preferred solution of the zero-code multi-terminal application automatic construction method based on AI semantic understanding according to the present invention, wherein: the parameters of the target device include resolution, pixel density, and interaction method.

[0014] As a preferred solution of the zero-code multi-terminal application automatic construction method based on AI semantic understanding according to the present invention, wherein: further include: Generate a context vector based on the parameters of the target device, and generate a dynamic weight through a multi-layer perceptron.

[0015] As a preferred solution of the zero-code multi-terminal application automatic construction method based on AI semantic understanding according to the present invention, wherein: parameterize the layout constraints in the interaction dependency graph, and adjust the initial layout constraints according to the context vector and the dynamic weight to generate adjusted layout parameters.

[0016] As a preferred solution of the zero-code multi-terminal application automatic construction method based on AI semantic understanding according to the present invention, wherein: it further includes: Based on the adjusted layout parameters, construct a layout objective function to minimize the overlap rate and visual imbalance of UI components.

[0017] As a preferred solution of the zero-code multi-terminal application automatic construction method based on AI semantic understanding according to the present invention, wherein: according to the parameters of the target end device and the semantic representation, automatically generate control mappings and interaction codes for the target platform, including: Based on a predefined control library and mapping rules, map the components in the UI component mask to atomic controls of the target platform; Convert the interaction logic in the interaction dependency graph into an event-action chain, and obtain the code of the target platform according to the syntax rules of the target platform.

[0018] Compared with the prior art, the beneficial effects of the invention are: 1. Compared with the deficiency that traditional single-modal image processing is difficult to accurately identify nested components, the present invention designs a dual-branch feature extractor to fuse the visual feature map and the structural feature map, combines the cross-modal attention mechanism and the atrous spatial pyramid pooling (ASPP) module to generate a high-precision UI component mask and an interaction dependency graph (IDG), which can not only extract the interaction dependency relationship between UI components, but also effectively solve the problem of blurred boundaries; 2. Aiming at the limitation that traditional responsive layout depends on fixed breakpoints and has poor adaptability, the present invention adopts a dynamic weight adjustment strategy (DWA) and a layout objective function to adaptively adjust layout parameters according to the device context, reduce component overlap and visual imbalance, and is applicable to new devices and diverse UI scenarios (such as e-commerce and enterprise applications), ensuring the long-term applicability and universality of the automatically generated code; 3. Different from the inefficient way of manually configuring interaction triggers in traditional zero-code platforms, the solution of the present invention automatically extracts the interaction dependency relationship from the layer metadata, generates an event-action chain and converts it into the code of the target platform, improving the automation degree and cross-platform consistency of zero-code development. Description of the Drawings

[0019] To more clearly illustrate the technical solutions of the embodiments of the present invention, the following will briefly introduce the drawings required for the description of the embodiments. Obviously, the drawings in the following description are only some embodiments of the present invention. For those of ordinary skill in the art, without creative efforts, other drawings can be obtained based on these drawings. Among them: Figure 1 It is the overall flowchart of the zero-code multi-terminal application automatic construction method based on AI semantic understanding according to an embodiment of the present invention. Detailed implementation manners

[0020] To make the above objects, features, and advantages of the present invention more obvious and understandable, the following will describe the detailed implementation manners of the present invention in conjunction with the drawings of the specification. Obviously, the described embodiments are some embodiments of the present invention, rather than all embodiments. Based on the embodiments in the present invention, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of the present invention.

[0021] In the following description, many specific details are set forth to fully understand the present invention. However, the present invention can also be implemented in other ways different from those described herein. Those skilled in the art can make similar generalizations without departing from the connotation of the present invention. Therefore, the present invention is not limited by the specific embodiments disclosed below.

[0022] Secondly, the so-called "one embodiment" or "embodiment" herein refers to a specific feature, structure, or characteristic that can be included in at least one implementation manner of the present invention. The "in one embodiment" that appears in different places in this specification does not necessarily refer to the same embodiment, nor is it a separate or alternative embodiment that mutually excludes other embodiments.

[0023] The present invention is described in detail in conjunction with schematic diagrams. When detailing the embodiments of the present invention, for the convenience of explanation, the cross-sectional views showing the device structure will be enlarged locally out of the general proportion, and the schematic diagrams are only examples and should not limit the protection scope of the present invention herein. In addition, in actual production, three-dimensional spatial dimensions including length, width, and depth should be included.

[0024] At the same time, in the description of the present invention, it should be noted that the orientation or positional relationships indicated by terms such as "upper, lower, inner, and outer" are based on the orientation or positional relationships shown in the drawings. It is only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as a limitation to the present invention. In addition, the terms "first, second, or third" are only used for descriptive purposes and cannot be understood as indicating or implying relative importance.

[0025] Unless otherwise clearly defined and limited in the present invention, the terms "installation, connection, and coupling" shall be understood in a broad sense. For example, it may be a fixed connection, a detachable connection, or an integral connection; it may also be a mechanical connection, an electrical connection, or a direct connection, or may be indirectly connected through an intermediate medium, or may be the communication inside two components. For those of ordinary skill in the art, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.

[0026] Embodiment 1 Referring to Figure 1 , this is the first embodiment of the present invention. This embodiment provides a zero-code multi-terminal application automatic construction method based on AI semantic understanding, including: S1. Receive the UI design draft image and layer metadata uploaded by the user; Specifically, the UI design draft image uploaded by the user is in PNG (pixel level) or SVG (vector level) format; Specifically, the layer metadata is in JSON format and includes the ID, position, hierarchy, and interaction annotation of the UI component; Specifically, the ID of the UI component is the unique identifier of each type of UI component. For example, {"component_id": "btn_001"} indicates that the type of the current UI component is a button and the identifier is 001; Specifically, the position of the UI component is the Bounding Box coordinate, which is composed of the upper left corner coordinate (x_min, y_min) and the lower right corner coordinate (x_max, y_max); Specifically, the hierarchy of the UI component is a parent-child relationship, that is, there may be other UI components in the current UI component. For example, {"parent": "container_001", "children": ["btn_001", "txt_001"]} indicates that the current UI component is a parent object (parent) which is a container component (container_001), and the container component also includes child objects (children) button component (btn_001) and text box component (txt_001); Specifically, the interaction annotation of the UI component is the interaction behavior specified by the user. For example, {"component_id": "btn_001", "target": "list_001", "click": "refresh_list"} indicates that the user specifies that by clicking on this button component (btn_001), a refresh operation (refresh_list) is performed on the target component (list_001); S2. Based on AI semantic understanding technology, parse the UI design draft image and layer metadata to generate the semantic representations of UI component masks and interaction dependency graphs; Further, for the UI design draft image, use the pre-trained ResNet-50 to scale the image resolution of the UI design draft to 512×512; Specifically, in the solution of the present invention, the pre-trained ResNet-50 is composed of several residual blocks, including an initial convolutional layer, batch normalization, ReLU activation, and pooling layers; among them, batch normalization and ReLU activation are respectively used to stabilize model training and enhance the expression ability of the model, which are not the innovation directions of the solution of the present invention, so they will not be elaborated herein; It should be explained that ResNet-50 (Residual Network with 50 layers) is a deep convolutional neural network (CNN) model; Preferably, since ResNet-50 can solve the gradient vanishing problem in deep networks through skip connections and effectively extract multi-level features of images (from low-level edges to high-level semantic information), it is particularly important for the visual feature extraction of the UI design draft in the solution of the present invention, because the UI design draft contains complex visual elements (such as buttons, text boxes, icons, etc.) that require multi-scale feature support; and compared with shallow networks (such as VGG) or deeper networks (such as ResNet-101), ResNet-50 can achieve a balance between performance and computational efficiency and is suitable for zero-code development scenarios with high real-time requirements; in addition, although a higher resolution (such as 1024×1024) can further improve the accuracy of UI component extraction, it will significantly increase the computational cost (memory and inference time); while ResNet-50 basically uses 224×224 as the initial resolution of image input, and based on the fact that the aforementioned UI design draft contains complex visual elements, it is necessary to increase the resolution to retain these complex visual elements; therefore, adopting 512×512 as the resolution of the input image is a compromise solution, and this resolution is within the processing capacity of ResNet-50, reducing the processing complexity while retaining the details of UI component boundaries and interaction annotations; Further, design a dual-branch feature extractor to extract visual features from the UI design draft image respectively to obtain a visual feature map, and extract structural features from the layer metadata to obtain a structural feature map; Specifically, use the convolutional layer of the pre-trained ResNet-50 as the backbone network to extract visual features and output a visual feature map ; among them, the process of extracting visual features is as follows: Input the scaled UI design draft image (512×512). According to the RGB values of this image, represent the resolution of this image as , where 3 represents the three RGB channels; Construct the initial convolutional layer (Conv_f). Set the convolutional kernel of the initial convolutional layer to: 7×7, the number of filters to 64 (number of output channels), the stride to: 2, and the padding to: 3 (the same as the three RGB channels). The processing process is expressed by the following formula: Among them, represents the height of the input UI design draft image, represents the height of the input UI design draft image, represents the width of the output feature map, represents the width of the output feature map, represents rounding down; , padding = 3, Conv_kernel_size = 7 represents that the size of the convolutional kernel is 7, stride = 2, and we get: Similarly, , and according to the number of filters, the output feature map is represented as: ; Send the obtained output feature map into batch normalization to standardize the feature map of each channel, which can stabilize model training and accelerate convergence. At the same time, use ReLU activation to non - linearize the output feature map to enhance the expression ability of the model; Receive the output feature map through the max - pooling layer , set the pooling kernel to: 3×3, the stride to: 2, and the padding to: 1 (to retain edge information). The processing process calls the above and formulas, and replace Conv_kernel_size with Pool_kernel_size in them to get: , padding = 1, Pool_kernel_size = 3 represents that the size of the pooling kernel is 3, stride = 2, , similarly, , according to the number of filters (pooling does not change the number of channels), the output feature map is represented as: ; It is divided into 4 residual block groups (stages), each group consists of multiple residual blocks (Bottleneck Blocks), each residual block contains 3 convolutional layers (1×1, 3×3, 1×1), the convolutional kernel size is fixed at 3, and the 1×1 convolutional layer is directly added to the output through a skip connection; among them, each residual block group is processed one by one as follows: The first residual block group, Conv_RB1 (containing 3 residual blocks), input ; Main path: 1×1 convolutional layer: reduce the dimension to 64 channels (64 filters), stride 1; 3×3 convolutional layer: keep 64 channels (number of filters), stride 1, padding 1; 1×1 convolutional layer: increase the dimension to 256 channels (256 filters), stride 1; Residual path: The first block: 1×1 convolutional layer (256 filters), from 64 channels to 256 channels, stride 1, the second to third blocks: directly add 256 channels to the output; Among them, the stride of all convolutional layers in the main path and the residual path is 1, and the padding is 1, obtaining: Similarly, ; It should be noted that at this time, the main path is: 64→64 (1×1)→64 (3×3)→256 (1×1); the residual path is: 64→256 (1×1); at this time, we can obtain ; The second residual block group, Conv_RB2 (containing 4 residual blocks), input ; Main path (downsampling): 1×1 convolutional layer: reduce the dimension to 128 channels (128 filters), stride 1; 3×3 convolutional layer: keep 128 channels (128 filters), stride 2 (downsampling), padding 1; 1×1 convolutional layer: increase the dimension to 512 channels (512 filters), stride 1; Residual path: The first block: 1×1 convolutional layer (512 filters), 256→512 channels, stride 2; the second to fourth blocks: 1×1 convolutional layer: reduce 512 to 128 channels, stride 1; 3×3 convolution: 128 channels, stride 1, padding 1; 1×1 convolutional layer: increase 128 to 512 channels, stride 1; Among them, the stride of the 3×3 convolutional layer in the main path is 2: Similarly, ; The stride of the 1×1 convolutional layer in the residual path is 2: Similarly, ; Among them, the second to fourth blocks are the same as the first block, maintaining ; It should be noted that at this time, the main path is: 256 → 128 (1×1) → 128 (3×3) → 512 (1×1), and the residual path is: 256 → 512 (1×1); at this time, ; The third residual block group, Conv_RB3 (including 6 residual blocks), input ; Main path (downsampling): 1×1 convolutional layer: 512 → 256 channels, stride 1; 3×3 convolutional layer: 256 channels, stride 2, padding 1; 1×1 convolutional layer: 256 → 1024 channels, stride 1; Residual path: The first block: 1×1 convolutional layer, 512 → 1024 channels, stride 2; The second to sixth blocks: directly add 1024 channels to the output, stride 1; Among them, the stride of the 3×3 convolutional layer in the main path is 2: Similarly, ; The stride of the 1×1 convolutional layer in the residual path is 2: Similarly, ; Among them, the second to sixth blocks are the same as the first block, maintaining ; It should be noted that at this time, the main path is: 512 → 256 (1×1) → 256 (3×3) → 1024 (1×1); the residual path is: 512 → 1024 (1×1); at this time, ; The fourth residual block group, Conv_RB4 (including 3 residual blocks), input ; Main path (downsampling): 1×1 convolutional layer: 1024 → 512 channels, stride 1; 3×3 convolutional layer: 512 channels, stride 2, padding 1; 1×1 convolutional layer: 512 → 2048 channels, stride 1; Residual path: The first block: 1×1 convolutional layer, 1024 → 2048 channels, stride 2; The second to third blocks: directly add 2048 channels to the output, stride 1; Among them, the stride of the 3×3 convolutional layer in the main path is 2: Similarly, ; The stride of the 1×1 convolutional layer in the residual path is 2: Similarly, ; Among them, the second to third blocks are the same as the first block, maintaining ; It should be noted that at this time, the main path is: 1024 → 512 (1×1) → 512 (3×3) → 2048 (1×1); the residual path is: 1024 → 2048 (1×1); at this time, we can obtain ; Furthermore, extract the structural features from the layer metadata to obtain the structural feature map. The processing flow is as follows: Based on the graph convolutional network, use the layer metadata as the input to extract the ID, position, layer level, and interaction annotations of the UI components as the initial features; By inputting the position of each UI component, generate a two-dimensional sine position encoding to reflect the spatial position of the UI component in the design draft; this two-dimensional sine position encoding is expressed by the formula: Among them, represents the position encoding vector, x and y respectively represent the abscissa and ordinate of the UI component, d represents the dimension, and u represents the dimension index of the position encoding vector, represents the wavelength that determines the sine or cosine function; It should be explained that due to the convolutional operation of ResNet-50 being insensitive to spatial position (the convolutional kernel shares weights), the precise position information of the pixels in the UI design draft may be lost; while the position encoding can explicitly encode its spatial position by adding a 128-dimensional vector to each pixel, thereby distinguishing UI components with the same appearance but different positions (such as two identical buttons in different areas); The graph convolutional network uses a two-layer convolutional network. Each layer updates the initial features (the first layer: 128 → 256, the second layer: 256 → 256), and aggregates through the second-layer convolutional network to aggregate into neighbor information of 256, and outputs a 256-dimensional structural embedding vector ; Broadcast the output to the corresponding pixel area (mapped to the original input image of 512×512 based on the bounding box), and downsample and adjust to 16×16 (aligned with the visual feature map ), and fill each pixel with dimensions to the corresponding UI component (if there is no UI component, fill with a zero vector) to obtain the structural feature map ; It should be noted that the 128 only considers the dimension of the position encoding and does not consider the UI component type encoding and the center coordinates of the bounding box; Furthermore, by designing a cross-modal attention module, the visual feature map and the structural feature map are fused to generate a fused feature map; Specifically, the cross-modal attention module is represented by the following formula: where, is the output of the cross-modal attention module, and the output result is the fused feature map , D is the number of input channels; Softmax is the attention weight; is the query projection matrix, whose role is to map the visual feature map into a query, which helps to identify the semantic requirements of button pixels in the UI design draft, such as "this shape is a button, what is its function?"; is the key projection matrix, whose role is to map the structural feature map into a key for the visual feature map to match; is the value projection matrix, whose role is to map the structural feature map into a value, which contains the actual content of the structural feature map; represents the scaling factor, which is used to stabilize the gradient of Softmax and ensure that the distribution of the attention weight is reasonable; Even further, the atrous spatial pyramid pooling module is applied to the fused feature map to generate an enhanced feature map; It should be explained that atrous spatial pyramid pooling (ASPP) is a feature extraction module, which can capture multi-scale features through atrous convolution and multi-scale pooling, and is used to solve the problem of the scale change of the target object in semantic segmentation; for the fused feature map that fuses the visual feature map and the structural feature map, due to the limited receptive field, it is difficult to capture small UI components (such as label) and large containers (nested multiple UI components) at the same time, and ResNet-50 requires multi-scale features for support, so calling the atrous spatial pyramid pooling module can not only enhance the fused feature map, but also capture small UI components and large containers at the same time; Specifically, the dilation rates in the atrous spatial pyramid pooling are set to [1, 6, 12, 18], the size of all convolution kernels is fixed at 3, the output channels are 128, and they are concatenated with the fused feature map to generate an enhanced feature map; Furthermore, pixel-level classification is performed based on the enhanced feature map to generate a UI component mask; Specifically, transposed convolution is used to enlarge the enhanced feature map from 16×16 to 512×512 as follows: Among them, is denoted as upsampling, is the output channel; Deconv is denoted as the transposed convolution operation, is denoted as the transposed convolution kernel; Specifically, the features of each pixel are mapped to class scores through a 1×1 convolutional classifier: Among them, is denoted as the classification convolution kernel, and K is the number of classes (container, icon, background); It should be noted that using a 1×1 convolutional classifier is equivalent to applying a fully connected layer to the feature vector of each pixel, and mapping to class scores is equivalent to outputting the class probability of each pixel. For example, the pixels in the button area are predicted as the "button" class, and the text box area is predicted as the "text box"; Specifically, the class probability of each pixel is output: Among them, represents the probability that the pixel (i, j) belongs to class k; Specifically, the generated UI component mask is denoted as: Among them, is a multi-channel mask; It should be noted that by generating the UI component mask, the boundaries of each pixel's UI components can be clearly identified; Furthermore, the UI component mask and layer metadata are combined to construct an interaction dependency graph; It should be explained that the interaction dependency graph (Interaction Dependency Graph, IDG) is a graph structure, which is the key output of the zero-code platform and can convert static UI design drafts into dynamic and interactive multi-terminal applications; Specifically, the interaction dependency graph is denoted as: Among them, V is denoted as the UI component instance (a button with functions), and E is denoted as the interaction logic (such as clicking a button to trigger form submission) and layout constraints (such as the button being on the right side of the text box) between UI components; Specifically, for the interaction logic between UI components, such as "clicking the button" triggering "form submission", an edge (from the button to the form) is added to the button and the form; It should be noted that the interaction logic is designed to add edges to UI components with functional relationships and not to those without functional relationships; Specifically, for layout constraints, the connected regions of each UI component category and the hierarchical information of layer metadata are identified through the UI component mask, the bounding box is calculated for each connected region, and the center point of the bounding box is obtained , thereby inferring the spatial relationship of the UI components, and its spatial relationship is calculated as follows: Among them, represents the distance threshold (in pixels); right represents the left side, left represents the right side, below represents the lower side, above represents the upper side, and include represents inclusion; It should be noted that by constructing the interaction dependency graph, not only can the interaction dependency relationship between UI components be clearly displayed, but also the spatial position relationship between UI components can be inferred, thereby improving the accuracy of the interaction relationship between UI components; S3. Automatically generate the control mapping and interaction code for the target platform according to the parameters and semantic representation of the target device, thereby realizing the construction of a cross-platform application with zero code; It needs to be explained that since the traditional method lacks the dynamic adaptation ability to the characteristics of the target device and is prone to problems such as UI component overlap or visual imbalance, in diverse device scenarios, the layout effect of UI components will be poor; the solution of the present invention overcomes the limitations brought by the traditional method by considering the characteristics of the target device (such as the aspect ratio of the output device display and the interaction method) and dynamically optimizing the layout parameters; Furthermore, obtain the resolution, pixel density, and interaction method of the device; It should be noted that this interaction method specifically refers to the interaction method of the user with the device, rather than the interaction method between UI components; Specifically, the resolution of the device refers to the width w and height h of the target device; Specifically, the pixel density refers to the pixel density of the target device, with the unit of pixels per inch (PPI); Specifically, the interaction method refers to the operation of the user on the device, such as the touch or keyboard and mouse method; Even further, generate a context vector based on the parameters of the target device and generate dynamic weights through a multi-layer perceptron; Specifically, the context vector C is expressed as: Among them, is the pixel density, and M is the interaction method; It should be noted that a multi-layer perceptron (MLP, Multi-Layer Perceptron) is a feedforward neural network composed of multiple fully connected layers, which is used to learn non-linear mappings from input features and output prediction values or weights; in the solution of the present invention, the MLP is used to process input features (i.e., context vectors) and output dynamic weights, which are used to adjust the layout constraints of the interaction dependency graph or the rendering parameters of UI components (such as button size); Specifically, the multi-layer perceptron has a three-layer structure, including an input layer, a hidden layer, and an output layer; Specifically, the input layer of the multi-layer perceptron receives the context vector, generates 128 hidden features, captures the pattern of "high PPI, small screen", and then the hidden layer further refines the 128 hidden features generated by the input layer into 64 features to identify the need to zoom in / out UI components. Finally, the output layer generates dynamic weights according to the refinement results of the hidden layer, where the dynamic weights include the total number of adjustable parameters in the UI component layout; Furthermore, parameterize the layout constraints in the interaction dependency graph, and adjust the initial layout constraints according to the context vector and dynamic weights to generate adjusted layout parameters; Specifically, parameterize each UI component, denoted as: , where, is the q-th UI component after parameterization, align is the alignment method of the UI component, and margin is the margin of the UI component; Specifically, the adjusted layout parameters are expressed as: where, represents the margin of the q-th UI component, is the margin of the q-th UI component after adjustment, is expressed as the dynamic weight, is expressed as the width of the target device, is expressed as the width of the UI component design draft; Even further, based on the adjusted layout parameters, construct a layout objective function to minimize the overlap rate and visual imbalance of UI components; It should be noted that the process of minimizing visual imbalance is a process of maintaining visual balance; Specifically, the constructed layout objective function L is expressed as: where, respectively represent the bounding boxes of the q-th UI component and the p-th UI component, q≠p; is expressed as and The intersection over union, which is used to measure the overlapping degree of UI components, is expressed as the central coordinates of , which is only recorded when the UI component is in a connected state, otherwise ; Mvf represents the visual center of gravity, which is the weighted average of the central coordinates of all UI components; is expressed as the balance factor, which is used to adjust the balance between the overlapping degree and the visual center of gravity; Specifically, through the simulated annealing algorithm, set the initial temperature to 100, the cooling rate to 0.95, and iterate 100 times. Continuously adjust the output result of L to gradually approach the global optimal solution, so as to achieve the minimization of L; Furthermore, based on the predefined control library and mapping rules, map the components in the UI component mask to the atomic controls of the target platform; It should be explained that the atomic control is the standard control, that is, the smallest UI element that cannot be further divided, such as <button>, Button in SwiftUI; Specifically, the predefined control library includes control categories (such as buttons, text boxes), control properties (text content, styles), event support (click, input), and platform adaptation (the implementation of each control on different platforms, such as HTML and Swift); Specifically, the mapping rules include the category matching of components (such as the UI component mask "button" mapping to< / button> <button>), the visual attributes of the UI component mask (such as blue, size 100×100) and semantic attributes (such as the "submit" function of the UI component mask "button"); It should be noted that the actual code actually mapped to each target platform is different. Taking the mapping rule as an example: Web:< / button> <button style=""width:" 479px;background-color:blue;">Submit< / button> ; iOS: Button(label:{Text("Submit").frame(width:479px)}); Android: <button android:width=""479px"android:text="Submit" / ">; Furthermore, convert the interaction logic in the interaction dependency graph into an event-action chain, and obtain the code for the target platform according to the syntax rules of the target platform; Specifically, event-action represents user operation events (such as clicks) and component operation response actions (such as submitting a form), and the chain represents the events and actions required to add an edge (such as from a button to a form, property "trigger submission" → event "click" + action "submit"); then converting the edges of the interaction dependency graph into an event-action chain can obtain the following form: Chain 1: {event = "click button", action = "submit form"}; Specifically, the following are three common types of syntax rules for target platforms: Web: HTML (structure), CSS (style), JavaScript (interaction); iOS: SwiftUI (structure + style + interaction), Swift (logic); Android: XML (structure + style), Kotlin (logic + interaction); It should be noted that by constructing an event-action chain, the event-action relationship between UI components can be obtained quickly, thereby improving the automation level of zero-code development and cross-platform consistency.

[0027] Those skilled in the art should understand that the embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware aspects. Moreover, this application can take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk memories, CD-ROMs, optical memories, etc.) containing computer-usable program code. The solutions in the embodiments of this application can be implemented in various computer languages, for example, object-oriented programming languages such as Java and interpreted scripting languages such as JavaScript.

[0028] This application is described with reference to the flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to the embodiments of this application. It should be understood that each process and / or block in the flowchart and / or block diagram, and the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to the processors of general-purpose computers, special-purpose computers, embedded processors, or other programmable data processing devices to generate a machine, so that the instructions executed by the processors of the computer or other programmable data processing devices generate for implementation in the process Figure 1 one process or multiple processes and / or blocks Figure 1 means for the functions specified in one or more blocks.

[0029] These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing apparatus to operate in a particular manner, such that the instructions stored in the computer-readable memory produce a manufacture including an instruction means that implements the functions specified in one Figure 1 one or more processes and / or blocks Figure 1 one or more blocks.

[0030] These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer-implemented process, so that the instructions executed on the computer or other programmable apparatus provide steps for implementing the functions specified in one Figure 1 one or more processes and / or blocks Figure 1 one or more blocks.

[0031] Although the preferred embodiments of the present application have been described, those skilled in the art can make additional changes and modifications once they learn the basic creative concepts. Therefore, the appended claims are intended to be construed to include the preferred embodiments and all changes and modifications falling within the scope of the present application.

[0032] Obviously, those skilled in the art can make various changes and modifications to the present application without departing from the spirit and scope of the present application. Thus, if these modifications and variations of the present application fall within the scope of the claims of the present application and their equivalent technologies, the present application is also intended to include these modifications and variations.< / button>

Claims

1. A zero-code multi-terminal application automatic construction method based on AI semantic understanding, characterized in that: include: Receive UI design draft images and layer metadata uploaded by users; Based on AI semantic understanding technology, the UI design draft image and layer metadata are parsed to generate semantic representations of UI component masks and interaction dependency graphs; According to the parameters of the target device and the semantic representation, the control mapping and interaction code of the target platform are automatically generated, thereby realizing zero-code cross-platform application construction.

2. The method for automatically building zero-code multi-terminal applications based on AI semantic understanding as claimed in claim 1, characterized in that: The layer metadata includes the ID, location, level, and interaction annotation of the UI component.

3. The method for automatically building zero-code multi-terminal applications based on AI semantic understanding as claimed in claim 2, characterized in that: Based on AI semantic understanding technology, the UI design draft image and layer metadata are parsed, including: A dual-branch feature extractor is designed to extract visual features from the UI design draft image to obtain a visual feature map, and to extract structural features from the layer metadata to obtain a structural feature map.

4. The method for automatically building zero-code multi-terminal applications based on AI semantic understanding as claimed in claim 3, characterized in that: Also includes: Designing a cross-modal attention module to fuse the visual feature map and the structural feature map to generate a fused feature map; A dilated spatial pyramid pooling module is applied to the fused feature map to generate an enhanced feature map.

5. The method for automatically building zero-code multi-terminal applications based on AI semantic understanding as described in claim 2 or 4, characterized in that: Generates semantic representations of UI component masks and interaction dependency graphs, including: Perform pixel-level classification based on the enhanced feature map to generate a UI component mask; Combining the UI component mask with the layer metadata to construct an interaction dependency graph; The nodes in the interaction dependency graph are represented as UI component instances, and the edges in the interaction dependency graph are represented as interaction logic and layout constraints between UI components.

6. The method for automatically building zero-code multi-terminal applications based on AI semantic understanding according to claim 1, characterized in that: The parameters of the target device include resolution, pixel density and interaction mode.

7. The method for automatically building zero-code multi-terminal applications based on AI semantic understanding as claimed in claim 6, characterized in that: Also includes: A context vector is generated based on the parameters of the target device, and a dynamic weight is generated through a multi-layer perceptron.

8. The method for automatically building zero-code multi-terminal applications based on AI semantic understanding as claimed in claim 7, characterized in that: The layout constraints in the interaction dependency graph are parameterized, and the initial layout constraints are adjusted according to the context vector and the dynamic weight to generate adjusted layout parameters.

9. The method for automatically building zero-code multi-terminal applications based on AI semantic understanding as claimed in claim 8, characterized in that: Also includes: Based on the adjusted layout parameters, a layout objective function is constructed to minimize the overlap rate and visual imbalance of UI components.

10. The method for automatically building zero-code multi-terminal applications based on AI semantic understanding according to claim 1, characterized in that: According to the parameters of the target device and the semantic representation, the control mapping and interaction code of the target platform are automatically generated, including: Based on the predefined control library and mapping rules, the components in the UI component mask are mapped to the atomic controls of the target platform; the interaction logic in the interaction dependency graph is converted into an event-action chain, and the code of the target platform is obtained according to the grammatical rules of the target platform.

Images