Front-end code generation method and device, equipment, storage medium and program product

Abstract

Embodiments of the present application provide a front-end code generation method and device, equipment, storage medium and program product, relating to the field of big data and financial technology, aiming to solve the problems of insufficient understanding of layout relationship, difficulty in balancing local details and overall structure, large calculation burden, and low accuracy and rationality of generated code in the process of converting the existing user interface image into code. The method obtains a target user interface image, divides the image by a fixed window to extract local area features, forms overlapping areas by sliding to extract cross-area correlation features, and then inputs the two types of features into a code generation model to output the corresponding front-end code sequence. In this way, the understanding of nested, parallel and cross-area layout relationships can be enhanced at a high feature extraction efficiency, and the accuracy of code generation, page structure consistency and actual deployment effect can be improved.

Description

Technical Field

[0001] This application relates to the fields of big data and financial technology, and in particular to a front-end code generation method, apparatus, device, storage medium and program product. Background Technology

[0002] In the field of web front-end development, the efficiency of the integration between user interface design and code implementation directly impacts product development cycle and quality. Currently, the front-end development process typically involves designers creating user interface design drafts and front-end developers manually writing code. The designer's visual design needs to be translated into code through complex communication, and developers need to spend a significant amount of time analyzing the layout, interaction logic, and style details in the design drafts, and then writing code line by line. Especially in agile development models, when product requirements change frequently, the inefficiency and high error rate of manual coding become core problems hindering rapid iteration.

[0003] In existing technologies, the traditional manual development model involves designers completing user interface design drafts, followed by front-end developers manually writing the code. This model relies heavily on the developers' expertise, has a long development cycle, and is prone to inconsistencies between the code and the design draft due to errors in the design draft parsing. While deep learning models are used for code generation, they lack the ability to jointly model local details and global layout of user interface images, suffer from low model training efficiency, lack structured features in the generated code, and fail to optimize code maintainability for actual development needs.

[0004] Therefore, existing technologies struggle to simultaneously capture both local details and global layout of user interface images. Automated tools lack structured features in the generated code and fail to optimize the maintainability of the generated front-end code to meet the actual needs of front-end development, resulting in low efficiency in front-end code generation. Summary of the Invention

[0005] This application provides a front-end code generation method, apparatus, device, storage medium, and program product to solve the technical problems of existing technologies that make it difficult to simultaneously capture local details and global layout of user interface images, lack of structured features in code generated by automated tools, and failure to optimize the maintainability of generated front-end code for actual front-end development needs, resulting in low efficiency in front-end code generation.

[0006] Firstly, this application provides a front-end code generation method, including:

[0007] Obtain the target user interface image;

[0008] Using a window of a preset size, multiple first window regions are divided on the target user interface image, and the local self-attention features of the target user interface image within the first window region are obtained;

[0009] A sliding window is used to divide the target user interface image into multiple second window regions. Cross-window self-attention features of the target user interface image within the second window region are obtained. The second window regions have at least partial overlap with the corresponding first window regions.

[0010] Local self-attention features and cross-window self-attention features are input into the code generation model to generate target code sequences corresponding to the target user interface image.

[0011] Secondly, this application provides a front-end code generation apparatus, comprising:

[0012] The image acquisition module is used to acquire the target user interface image;

[0013] The window segmentation module is used to divide the target user interface image into multiple first window regions using a window of a preset size, and to obtain the local self-attention features of the target user interface image within the first window region.

[0014] The window translation module is used to slide the window, divide the target user interface image into multiple second window regions, and obtain the cross-window self-attention features of the target user interface image within the second window region. The second window region and the corresponding first window region have at least partial overlap.

[0015] The code generation module is used to input local self-attention features and cross-window self-attention features into the code generation model to generate target code sequences corresponding to the target user interface image.

[0016] Thirdly, this application provides an electronic device, including: a processor and a memory communicatively connected to the processor;

[0017] The memory stores instructions that the computer executes;

[0018] The processor executes computer-executable instructions stored in memory to implement any of the methods of the first aspect.

[0019] Fourthly, this application provides a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, are used to implement the method of any one of the first aspects.

[0020] Fifthly, this application provides a computer program product, including a computer program that, when executed by a processor, implements the method of any one of the first aspects.

[0021] The front-end code generation method, apparatus, device, storage medium, and program product provided in this application acquire a target user interface image, extract local self-attention features within a first window region divided by a window of a preset size, and cross-window self-attention features within a second window region divided by a sliding window and at least partially overlapping with the corresponding first window region. The local self-attention features and cross-window self-attention features are then input into a code generation model. This approach can control the computational complexity of feature extraction while also considering the expression of local interface details and cross-region layout relationships, thereby improving the accuracy, structural rationality, and consistency with the design draft of the code sequence corresponding to the target user interface image. Attached Figure Description

[0022] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0023] Figure 1 A flowchart illustrating a front-end code generation method provided in an embodiment of this application;

[0024] Figure 2 A schematic diagram of the structure of a front-end code generation model network provided in an embodiment of this application;

[0025] Figure 3 This is a schematic diagram of the structure of a front-end code generation device provided in an embodiment of this application;

[0026] Figure 4 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application.

[0027] The accompanying drawings illustrate specific embodiments of this application, which will be described in more detail below. These drawings and descriptions are not intended to limit the scope of the concept in any way, but rather to illustrate the concept of this application to those skilled in the art through reference to particular embodiments. Detailed Implementation

[0028] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this application as detailed in any of the foregoing embodiments.

[0029] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties. Furthermore, the collection, storage, use, processing, transmission, provision, disclosure, and application of the relevant data all comply with the relevant laws, regulations, and standards of the relevant countries and regions, have taken necessary confidentiality measures, do not violate public order and good morals, and provide corresponding operation access points for users to choose to authorize or refuse.

[0030] Furthermore, the technical solution involved in this application, which involves big data analysis of user information (including but not limited to personal biometrics, identity data, consumption data, asset data, electronic terminal operation data, etc.) and the use of artificial intelligence technology for automated decision-making, and makes decisions that have a significant impact on personal rights based on the results of automated decision-making, provides users with corresponding operation entry points for users to choose to agree to or reject the results of automated decision-making; if the user chooses to reject, the process will proceed to the expert decision-making process.

[0031] It should be noted that the front-end code generation method, apparatus, device, storage medium and program product provided in this application can be used in the fields of big data and fintech, as well as in any field other than big data and fintech. The application fields of the front-end code generation method, apparatus, device, storage medium and program product in this application are not limited.

[0032] Front-end development involves transforming user interface designs into executable page code, commonly seen in the development processes of corporate websites, e-commerce pages, management backends, and mobile web pages. In such scenarios, it is usually necessary to first obtain the target user interface image, and then complete the page structure based on the navigation bar, forms, lists, buttons, and image areas in the image.

[0033] In existing implementations, one approach involves developers manually writing relevant script code based on the design drafts, while the other uses an image-to-code deep learning model to first extract interface image features and then output the corresponding code in sequence. The former mainly relies on manual understanding of the position, size, and hierarchical relationship of interface elements, while the latter often employs a unified feature extraction and sequence generation mechanism to map the visual information in the entire interface into code text.

[0034] However, manual coding relies heavily on development experience. When the design draft contains duplicate cards, nested containers, complex alignment relationships, and detailed elements, developers need to repeatedly identify the layout logic and manually adjust the tag hierarchy, resulting in a longer conversion cycle and a higher risk of structural deviations. Especially when requirements change frequently, repeatedly modifying the page structure further increases communication and implementation costs.

[0035] While some automatic code generation methods can perform the basic conversion from image to code, they often struggle to simultaneously represent both local details and the overall layout in complex user interface scenarios. For example, a model might be able to identify local components such as buttons and text boxes, but it may fail to accurately grasp the parallel, containment, or cross-regional correspondences between components. This can lead to discrepancies between the generated code and the design draft in terms of nested structure, layout order, or page semantics. Furthermore, some methods perform extensive correlation calculations on the entire image, resulting in a high computational burden, impacting training efficiency and actual deployment performance, ultimately requiring significant manual correction of the generated results.

[0036] In light of this, balancing feature extraction efficiency and layout understanding during the conversion from user interface images to code has become a pressing technical challenge. To address this issue, a front-end code generation method is provided. After acquiring the target user interface image, multiple first window regions are divided on the target user interface image using a window of a preset size to obtain local self-attention features of the target user interface image within each first window region. Then, multiple second window regions are divided on the target user interface image using a sliding window to obtain cross-window self-attention features of the target user interface image within each second window region, wherein the second window regions at least partially overlap with their corresponding first window regions. Subsequently, the local and cross-window self-attention features are input into a code generation model to generate a target code sequence corresponding to the target user interface image. This method can be deployed in front-end development tools or low-code platforms that include an image upload module, a model processing module, and a code output module to improve the accuracy and structural rationality of code generation.

[0037] Figure 1 This is a flowchart illustrating a front-end code generation method provided in an embodiment of this application, such as... Figure 2 As shown, the method includes:

[0038] S101: Obtain the target user interface image.

[0039] In this application, the target user interface image serves as the input object for the front-end code generation method, carrying the visual interface content required for subsequent window segmentation, attention feature extraction, and code sequence generation. The executing entity can be a front-end code generation device deployed on a local terminal, server, or low-code platform. Specifically, it can receive user-submitted interface design draft images, read existing interface screenshots, design export diagrams, or prototype diagrams, or obtain interface images through image acquisition. The target user interface image can be a single page image or a composite image containing multiple interface areas. In the case of a composite image, the executing entity treats the entire image as a unified input object for subsequent processing.

[0040] In one possible embodiment, after acquiring the target user interface image, the image can be preprocessed to meet the requirements of subsequent processing. After processing, target user interface image data is generated and written into the feature extraction process as input for subsequent steps.

[0041] Based on the above analysis, it can be seen that the target user interface image does not simply refer to the original static image file, but rather to the interface image data object that has already entered the processing chain and can be directly used for window partitioning and feature calculation. This step, by clearly defining the input object, enables the subsequent first and second window areas to perform attention calculations on a unified processing object. For interfaces containing navigation bars, buttons, text boxes, card lists, image areas, and nested containers, subsequent steps can extract structure-related information based on the interface image, thus enabling the code generation model to receive stable input data. It should be understood that the above example is merely illustrative and not limiting.

[0042] S102: Using a window of a preset size, divide the target user interface image into multiple first window regions and obtain the local self-attention features of the target user interface image within the first window region.

[0043] In this application, a window is used to define the scope of attention calculation. The first window region is the image region formed on the target user interface image when the window is not translated. Local self-attention features are used to characterize the pixel region relationships, component local layout relationships, and local nesting relationships within a single window. In specific implementation, the executing entity first determines a window of a preset size according to the model configuration. Subsequently, the executing entity uses a window of the preset size to divide multiple first window regions on the target user interface image.

[0044] In one specific implementation, the image preprocessing unit first inputs the target user interface image into the visual encoding front end to obtain a basic feature map. Then, the window partitioning unit divides the image into regions according to a preset size, forming multiple first window regions. For each first window region, the executing entity acquires the regional features within that region and performs attention calculations within the current first window region to generate a local context representation of that first window region. The local context representations at multiple locations are further processed through linear transformation, normalization, and residual connections to form the local self-attention features of that first window region. After repeating this process for multiple first window regions, a set of local self-attention features of the target user interface image is obtained.

[0045] The role of local self-attention features is to aggregate information within a single window, enabling the modeling of local visual relationships such as button text and button boundaries, form labels and input boxes, image areas and their titles, and icons and text within cards within a limited scope. If there are duplicate cards or rule lists in the interface, multiple first window regions will output the local self-attention features of their respective regions. Subsequent modules then aggregate or stitch these features together to form a relatively stable representation of the local structure. Since the first window regions do not perform cross-region attention calculations in this step, the scope of attention calculation is limited to within a single window, and the computational complexity is controlled by the window size, eliminating the need for global correlation calculations across all positions in the entire image. Based on the above analysis, it can be seen that by constructing multiple first window regions on the target user interface image using windows of preset sizes and extracting local self-attention features within each first window region, the computational complexity can be controlled while maintaining the ability to express local details of the interface. This allows buttons, icons, text, component boundaries, and local inclusion relationships to enter the subsequent code generation process in the form of structured features. It should be understood that the above example is for demonstration purposes only and is not a limitation.

[0046] S103: Sliding window, divides the target user interface image into multiple second window regions, and obtains the cross-window self-attention features of the target user interface image within the second window region.

[0047] In this embodiment of the application, the second window area overlaps at least partially with the corresponding first window area.

[0048] In this application, the second window region is a region formed on the target user interface image after sliding and translating a window of the same size as the first window region. This region is used to introduce inter-window relationship modeling within a local computation framework. Cross-window self-attention features are used to characterize the layout continuity, parallel relationships, containment relationships, and cross-region dependencies between different windows. In specific implementation, after extracting the local features of the first window region, the executing entity calls a sliding window unit to perform a displacement operation on the original window coordinates. The displacement amount is determined by the sliding step size. The sliding step size is smaller than the window size, causing each second window region to shift its position relative to the corresponding first window region, forming at least a partial overlap, while simultaneously covering image content originally located near the boundary of the adjacent first window region.

[0049] In one possible embodiment, the execution entity forms a second window region that at least partially overlaps with the first window region by sliding the window. After the window is translated, the execution entity acquires the cross-window self-attention features of the target user interface image within the second window region. Since the second window region spans the boundaries of multiple original first window regions, the attention calculation within the current second window region actually establishes feature associations between the original adjacent first window regions. Therefore, the obtained cross-window self-attention features not only include the local structural information within the second window region but also the mutual constraint relationships between elements on both sides of the boundary of the first window region.

[0050] For example, multiple buttons in a navigation bar may be located in adjacent first window regions, and the boundaries of card containers and their internal text areas may belong to different first window regions. If only local self-attention features within the first window region are used, information on both sides of the boundary may be truncated at the window segmentation. After forming a second window region that at least partially overlaps with the first window region by sliding the window, the executing entity can re-establish attention connections in the overlapping region and its adjacent regions, allowing text, icons, container outlines, and alignment lines located on both sides of the first window region boundary to enter the same attention calculation range, thereby obtaining cross-window self-attention features. The cross-window self-attention features obtained from multiple second window regions can be rearranged in spatial order and aligned with the corresponding local self-attention features of the first window region to form a joint representation sequence.

[0051] Based on the above analysis, this step, through the sliding window mechanism, allows the relationships between elements originally segmented in different first window regions to be expressed without expanding the scope of a single attention calculation. This addresses the problem of insufficient understanding of global layout and cross-regional structure when relying solely on local windows. The implementation that the second window region at least partially overlaps with the corresponding first window region enables the second window region to cover adjacent boundary regions. This allows the side-by-side arrangement of components, cross-column alignment, parent-child container coverage, and cross-regional semantic correspondences to be reflected in the cross-window self-attention features. For repeated cards, nested containers, and multi-column layouts in complex user interface scenarios, this step provides a cross-boundary association path, enabling subsequent code generation to recover a more accurate label hierarchy and layout order based on joint features. It should be understood that the above examples are for demonstration purposes only and are not limiting.

[0052] S104: Input local self-attention features and cross-window self-attention features into the code generation model to generate the target code sequence corresponding to the target user interface image.

[0053] In this application, the code generation model is used to map visual features into code results corresponding to the target user interface image. The target code sequence is a structured code text sequence output by the code generation model, which can be other front-end code sequences that can represent page hierarchy, component order, and region nesting relationships. In specific implementation, the executing entity first jointly organizes the local self-attention features obtained in S102 and the cross-window self-attention features obtained in S103. The joint method can be concatenation according to corresponding spatial positions, weighted fusion, or fusion after inputting them into different encoding branches of the model. If the concatenation method is used, the local self-attention features and cross-window self-attention features corresponding to each spatial position are connected in the channel dimension to form a joint feature vector sequence containing local details and cross-region layout relationships. If the dual-branch encoding method is used, the local self-attention features are input into the first encoding branch, and the cross-window self-attention features are input into the second encoding branch. The fusion layer aligns the two types of features according to position encoding and semantic similarity and outputs a unified encoded representation.

[0054] The code generation model can employ an encoder and decoder structure. The encoder receives a sequence of joint feature vectors and performs multi-level feature transformations to generate a high-dimensional representation reflecting the overall structure of the target user interface image. The decoder outputs various tags from the target code sequence progressively according to a predefined vocabulary. Specifically, during generation, the decoder starts with the initial tag and, combining the target user interface image structure representation output by the encoder with the already generated tag sequence, progressively predicts the next code tag until an end tag is output. The generated tags can include container labels, text tags, image tags, button tags, class name fragments, hierarchical control symbols, and sequence elements representing sibling relationships, child node relationships, or repetitive structures. The decoder can output syntactic fragments representing hierarchical nesting, parallel connections, repetition counts, and attribute appending; for direct front-end code sequences, the decoder can output structural tags and layout-related style reference information. After the model outputs the target code sequence, the result can be written to the code output module for display, editing, or further conversion into executable page code by front-end development tools.

[0055] In one possible embodiment, the code generation model employs an autoregressive generation approach during the decoding phase. This means that each time, the next tag is predicted based on previously generated code tags and the current target user interface image encoding representation. This ensures that container opening, container closing, child node insertion, and sibling node sorting follow a consistent generation logic. Local self-attention features provide stable identification criteria within components and local blocks, while cross-window self-attention features provide layout criteria for cross-column, cross-area, and cross-container relationships. Both together constrain the decoder's judgment of tag hierarchy and order. If the target user interface image contains structures such as nested cards, left-right columns, or separation of top navigation from content, the encoder can form a more complete page structure representation after fusing the two types of features, and the decoder outputs the corresponding target code sequence accordingly.

[0056] In one implementation scenario, to improve the efficiency of model training and the accuracy of generated code, this invention preprocesses the training data. The specific steps are as follows: 1) Data collection and annotation: Collect a large number of user interface images and their corresponding front-end code. This data can come from publicly available datasets or be generated by automated tools. Each user interface image is paired with corresponding code to ensure the diversity and richness of the training data. 2) Code parsing: Use a code parser to parse the collected code and extract key elements and attributes. The parser decomposes the code into a series of tags, attributes, and content to form a structured data representation. 3) Conversion to markup language code: Use a code conversion tool to convert the parsed code into markup language code. Markup language code can represent complex code structures with fewer characters. The conversion process includes: tag simplification, attribute simplification, nested structure simplification, data cleaning and verification, and removal of invalid or erroneous code fragments. At the same time, verify whether the converted markup language code can be correctly restored to the original format code to ensure the accuracy and integrity of the data. 4) Constructing a training dataset: Pair the preprocessed user interface images with the corresponding markup language code to form a training dataset. Each dataset contains a user interface image and a corresponding code sequence in markup language format, used for model training and validation. These preprocessing steps not only reduce the length of input and output text during model training, improving training efficiency, but also ensure the conciseness and accuracy of the generated code. The preprocessed dataset provides high-quality training samples for the model, contributing to improved model performance and the quality of the generated code.

[0057] A code generation model is used to generate target code sequences corresponding to user interface images. Specifically, the code generation model consists of an encoder and a decoder. The encoder is responsible for converting the input sequence (such as the feature representation of the user interface image) into a context representation. The encoder's input is an embedding representation of the sequence, and its output is a context representation used as input to the decoder. The decoder is responsible for converting the context representation generated by the encoder into the target code sequence. The decoder also consists of multiple identical stacked layers, each containing three main components: multi-head self-attention, encoder-decoder attention, and a feedforward neural network. The decoder's input is the encoder's output and the output of the previous time step, and its output is the next element of the target sequence.

[0058] To reduce the quadratic computational complexity of traditional global self-attention mechanisms while preserving the ability to extract fine local features when processing user interface images, the encoder employs a window-based multi-head self-attention module. The user interface image is divided into multiple non-overlapping windows, each containing a portion of the image. For example, suppose the image is divided into M×M windows, each containing h×w pixels. Within each window, attention scores are calculated only between regions within the window, significantly reducing computational and storage overhead. Specifically, this includes:

[0059] In-window attention calculation includes: dividing the input user interface image X into N windows, where each window contains a feature (pixel) matrix represented as follows: .

[0060] For each window Calculate its internal query (Q), key (K), and value (V) matrices:

[0061]

[0062] In the formula, , , These are the trainable parameter matrices.

[0063] The attention score for each region within the window is calculated as follows:

[0064]

[0065] In the formula, This refers to the dimension of the key-value vector, preventing gradient vanishing during model training. Through a window-based multi-head self-attention mechanism, computational efficiency is significantly improved while maintaining model performance. It is suitable for processing structured user interface images, not only reducing computational load but also preserving feature information of local regions in the image, providing a foundation for generating accurate front-end code.

[0066] To ensure interaction between different window regions and further enhance the model's understanding of component relationships in user interface images, a multi-head self-attention mechanism based on sliding windows is introduced. The concept of a sliding window refers to capturing the relationships between different components in an image by sliding a window across the image during self-attention computation to cover different regions. By shifting each window to the right and down by a certain step, each new window includes not only the region within the original window but also parts of adjacent windows. This design allows components originally belonging to different windows to perform local attention computations within the new window, thus capturing the relationships between these components. For example, in a list component, sliding windows can help the model identify hierarchical relationships between list items, even if these items were divided into different windows in the original image. This leverages the computational advantage of windowing methods while maintaining awareness of the global structure of the image. By adjusting the sliding step and window size, efficient capture and encoding of user interface image features can be achieved under different image sizes and complexities. A high-efficiency code generation model is then built to automatically generate target front-end code from user interface images.

[0067] Figure 2 This application provides a schematic diagram of the structure of a front-end code generation model network, as shown in the embodiments. Figure 2 As shown, the process of the front-end code generation model is as follows: The user interface image is first divided into tiles by a linear embedding layer, and then enters the encoder consisting of multiple stages for feature extraction; the encoded feature vector is used as context information and input into the decoder; the decoder combines the generated code sequence and predicts the markup language code in the next time step by step through self-attention mechanism and cross-attention mechanism.

[0068] For example, the detailed structure and steps of the encoder include: the encoder adopts a hierarchical structure, which is designed to efficiently capture local details (such as button text and icons) and global layout (such as component arrangement and nesting relationship) in user interface images.

[0069]

[0070] The detailed structure and steps of the decoder include: The core task of the decoder is to transform the image feature sequence extracted by the encoder into a linear target code sequence in markup language format. As the generator, it predicts the code word by word using an autoregressive approach, focusing on utilizing a cross-attention mechanism to map visual features to textual semantics.

[0071]

[0072] In another implementation scenario, in the field of front-end development, taking a product platform as an example, the system first acquires the target user interface image and analyzes its complexity before formally dividing the windows. The complexity evaluation model identifies and calculates the component density and nesting level in the target user interface image, outputs the corresponding complexity score, and then calls the preset window size decision network based on the complexity score to determine the window size that matches the target user interface image, thus forming the corresponding window division scheme.

[0073] During the training phase, the system collects historical user interface images and their corresponding code from the product platform. Several product display page design drafts and their code are selected as training data, and a parser is used to extract tags from the code. The code is then simplified into markup language format using a conversion tool. The converted markup language code is cleaned, and its reproducibility is verified. A training dataset containing user interface images and markup language code is then constructed.

[0074] After determining the window partitioning scheme, the system divides the target user interface image into multiple non-overlapping first window regions, each containing a preset number of pixels. In the platform's homepage example, windows can use 8×8 pixel blocks. The logo area of ​​the top navigation bar is divided into one first window region, and the image area of ​​the product cards is divided into another. The system calculates local self-attention features based on the pixel distribution, texture information, and component presentation within each first window region. It calculates a local self-attention score within each window and extracts detailed features of the components using a multi-head self-attention module. Specifically, the logo window captures color, shape, and position information, while the product card window identifies the layout relationship between images and text and characterizes button styles and product title font sizes.

[0075] After extracting local features from the first window region, the system performs a sliding operation on the window, translating the current window along the horizontal and vertical directions of the target user interface image to generate multiple second window regions. Each second window region at least partially overlaps with its corresponding first window region. In this example, the system translates the window to the right and downwards with a step size of 4 pixels, causing partial overlap between adjacent windows. For example, after translating the product card window, it covers part of the adjacent card's area. For each second window region, the system calculates cross-window self-attention features based on pixels within the region. Through a multi-head self-attention module based on the sliding window, the component relationships between adjacent windows, cross-region arrangement relationships, and overall interface layout information are simultaneously encoded. Specifically, in the product display area, the horizontal arrangement logic of multiple columns of cards is identified and encoded into a parallel structure in markup language code.

[0076] Local self-attention features and cross-window self-attention features are then jointly input into the code generation model, which includes an encoder and a decoder. The encoder alternately uses local and cross-window self-attention features to jointly extract local details and global layout of the target user interface image. Based on these two types of features, the encoder constructs a component relationship graph corresponding to the target user interface image, modeling the hierarchical relationships between components. The system uses the encoder to process user interface image features. The component relationship graph models the nesting relationships between product cards and containers to constrain the parent-child hierarchy and parallel relationships between components in the code structure.

[0077] The decoder receives the feature representation and component relationship graph semantic information output by the encoder, and fuses the local details, global layout, and hierarchical relationships between components in the target user interface image to generate a target code sequence corresponding to the target user interface image. The decoder generates the target code sequence word by word using an autoregressive approach. A cross-attention mechanism aligns the code generation with image features. For the current e-commerce platform homepage design draft, the system finally outputs a complete target code sequence in markup language format, which developers can directly convert into code and deploy.

[0078] After the target code sequence is generated, the system uses a reverse verification and feedback mechanism to check the generated results, feeds back the error information obtained from the verification to the code generation model, optimizes the model parameters, and adjusts the target code sequence based on the optimized model parameters. During the reverse verification process, the generated markup language format code is subjected to reproducibility verification, structural consistency checks, and alignment checks with the target user interface image. This ensures that the training data constructed from 1000 product display page design drafts and their corresponding codes during the training phase continuously participates in the model parameter updates, and that the output code maintains the hierarchical structure and arrangement corresponding to the logos, search box, and user login button in the top navigation bar, the multiple columns of product cards, images, titles, prices, and add-to-cart buttons in the product display area, and the organizational information and social media links in the bottom information bar.

[0079] The front-end code generation method provided in this embodiment includes: acquiring a target user interface image; dividing the target user interface image into multiple first window regions using a window of a preset size, and acquiring local self-attention features of the target user interface image within the first window regions; sliding the window to divide the target user interface image into multiple second window regions, acquiring cross-window self-attention features of the target user interface image within the second window regions, wherein the second window regions overlap with the corresponding first window regions at least partially; and inputting the local self-attention features and cross-window self-attention features into a code generation model to generate a target code sequence corresponding to the target user interface image. In this application, by combining the calculation of local self-attention in the window with the calculation of cross-window self-attention in the sliding overlapping window, the local structure of the interface is first extracted within a limited range, and then the layout relationship between windows is supplemented in the overlapping region. Subsequently, the code generation model maps the joint features into a code sequence, so that repeated cards, nested containers, complex alignment relationships, and local detail elements in complex UI scenes can be reflected in the generation result with a more consistent hierarchy and order, and the feature extraction process maintains a controlled computational scale. It should be understood that the above examples are only illustrative and not limiting.

[0080] In one possible implementation, a window of a preset size is used to divide the target user interface image into multiple first window regions, and the local self-attention features of the target user interface image within the first window regions are obtained, including: using the window to divide the target user interface image into multiple non-overlapping first window regions, each first window region containing a preset number of pixels; and calculating local self-attention features based on the pixels within the first window regions.

[0081] In the specific implementation, after the target user interface image is input into the feature extraction module, the image is first divided into grids according to a preset window width and height, so that each first window region corresponds to a set of pixel blocks at fixed positions, and the window boundaries do not overlap. Subsequently, a query vector, key vector, and value vector are constructed for the pixels in each first window region, and attention weights are calculated based on the similarity between the query vector and the key vector. Then, the value vectors are weighted and converged to obtain the local self-attention feature corresponding to that region. This feature can be output as a feature map with the same size as the window, or it can be output as a fixed-dimensional region representation after linear mapping for subsequent model reception.

[0082] In practical applications, the window size can be pre-set according to the resolution of the target user interface image and the density of interface elements. For example, each window can cover a local area suitable for recognizing buttons, text boxes, or icons. When the window size changes, adaptation can be achieved simply by synchronously adjusting the pixel segmentation rules and the attention calculation range. In practical applications, other window sizes can also be selected, and this application does not limit this. After completing the pixel relationship modeling within the region, the local self-attention features are output as multiple window-level features for subsequent cross-window self-attention feature fusion or code generation models.

[0083] By dividing the target user interface image into non-overlapping first window regions and calculating attention only within each region, feature modeling can be restricted to a local scope, allowing the model to focus on representing the element arrangement, edge relationships, and detailed structure within a single interface block. Since each window contains a fixed number of pixels, the computational scale of local features remains consistent, making the output more suitable for subsequent region alignment and structural combination, thus resulting in a more stable representation of the local content of the interface.

[0084] In one possible implementation, a sliding window is used to divide the target user interface image into multiple second window regions. The cross-window self-attention features of the target user interface image within the second window regions are obtained, including: translating the current window along the horizontal and / or vertical direction of the target user interface image to generate a second window region that at least partially overlaps with the current first window region; and calculating the cross-window self-attention features based on the pixels within the second window regions.

[0085] In practical implementation, the translation amount of the current window is less than the window's side length or height, causing the second window region to partially overlap with the current first window region. The current window can be translated independently in the horizontal direction, independently in the vertical direction, or simultaneously in both directions to cover adjacent interface regions. Pixels within the second window region are first positionally encoded and then fed into the self-attention calculation unit to calculate the weight distribution between the query vector, key vector, and value vector. The outputs of pixels at different positions are then weighted and converged to form a cross-window self-attention feature. This feature can be used as a representation of the association between windows and output to the subsequent code generation model. In practical applications, this feature extraction unit can also choose other model structures, which this application does not limit.

[0086] Because the second window region overlaps with the first window region, local elements at the window boundaries can be repeatedly perceived in adjacent windows. Cross-window relationships can be modeled within the local computational scope, thus preserving the correspondence between adjacent buttons, text boxes, list items, and container boundaries. Cross-window self-attention features calculated based on pixels within the second window region can provide more continuous layout association information to subsequent models, resulting in a more complete structural representation of the target user interface image.

[0087] Based on the aforementioned embodiments, before dividing the target user interface image into multiple first window regions using a window of a preset size, the method further includes: dynamically adjusting the window size according to the complexity of the target user interface image to generate a window division scheme corresponding to the target user interface image.

[0088] In the specific implementation, the target user interface image can be input to the complexity evaluation module. The complexity evaluation module analyzes the component density, nesting level, element boundary changes, and local alignment relationships in the image and outputs a complexity index. Based on this complexity index, the system dynamically sets the window size of the region to be segmented. When a region contains many nested buttons, text boxes, cards, or containers, the window size is set to a smaller value, allowing the region to be segmented with finer granularity. When a region has fewer visual elements and a simpler structure, the window size is set to a larger value, allowing the region to be segmented with coarser granularity. After the window size is determined, the target user interface image is then segmented using an overlay method according to the corresponding window segmentation rules, generating multiple first window regions that match the image. This segmentation result is used as the input for subsequent local self-attention feature calculation.

[0089] In another implementation, the complexity evaluation result can also be used to generate a uniform window size. This involves first calculating the overall complexity of the entire target user interface image, then determining the size of a single window based on this, and finally dividing the entire image according to that size. This method is suitable for scenarios where the overall interface structure is relatively consistent, ensuring that the window division rules are consistent with the overall layout of the target user interface image. In this application, the complexity evaluation can be performed by a pre-trained image analysis model, and the window size can be characterized by pixel side length, aspect ratio, or the number of pixels contained. In practical applications, other models or implementation methods can be selected for the model and parameter expression, and this application does not limit this.

[0090] By dynamically sensing the complexity of the target user interface image and adjusting the window size accordingly, the generated window partitioning scheme can adapt to the local structure of the interface, enabling complex areas to obtain finer partitioning granularity and simple areas to obtain a larger coverage area. This makes the subsequent partitioning results of the first window area more consistent with the interface layout characteristics and provides a more stable regional basis for local self-attention feature extraction.

[0091] In one possible implementation, the window size is dynamically adjusted based on the complexity of the target user interface image to generate a window partitioning scheme corresponding to the target user interface image. This includes: analyzing the component density and nesting level of the target user interface image using a preset complexity evaluation model to generate a complexity score for the target user interface image; determining the window size corresponding to the target user interface image based on the complexity score using a preset window size decision network; and dividing the target user interface image into multiple first window regions using a window of the preset size.

[0092] In practical implementation, the target user interface image is first processed by a component parsing module to extract bounding boxes for buttons, text boxes, list items, card containers, and navigation areas. The number of bounding boxes, the number of components per unit area, and the parent-child component containment relationships are then input into a complexity evaluation model. The complexity evaluation model can consist of a convolutional feature extraction unit and a regression output unit. The convolutional feature extraction unit extracts image edges, segmentation contours, and region clustering features, while the regression output unit outputs a continuous complexity score. The window size decision network can employ a multi-layer perceptron structure, mapping the window size to one of multiple discrete sizes based on the complexity score, or outputting continuous size parameters before quantization, to allow for smaller windows in complex interfaces and larger windows in simple interfaces. The determined window size is then used to rasterize the target user interface image, forming multiple non-overlapping first window regions. These first window regions are used to subsequently acquire local self-attention features.

[0093] This allows window partitioning to reflect the density and hierarchical nesting of interface elements, generating a window partitioning scheme adapted to the target user interface image. Therefore, in complex UI scenarios, finer-grained windows can be used to cover local structures, while in simple UI scenarios, larger windows can be used to reduce the number of blocks, thus ensuring consistency between subsequent feature calculations and layout representation.

[0094] With this approach, the window size is no longer fixed but can adaptively adjust according to the complexity of the interface. The partitioning result of the first window region better matches the interface structure characteristics, thereby improving the targeting of local feature extraction and reducing the redundancy caused by the number of invalid windows in subsequent calculations. For images with deep nesting levels or high component density, smaller windows can retain more local structural information; for images with simpler structures, larger windows can reduce partitioning overhead, thus maintaining a balance between accuracy and computational cost in the window partitioning scheme.

[0095] In one possible implementation, the code generation model includes an encoder and a decoder; local self-attention features and cross-window self-attention features are input into the code generation model to generate a target code sequence corresponding to the target user interface image, including: constructing a component relationship graph corresponding to the target user interface image in the encoder based on the local self-attention features and cross-window self-attention features, and modeling the hierarchical relationship between the components corresponding to the target user interface image; and fusing the semantic information of the component relationship graph in the decoder to generate the target code sequence corresponding to the target user interface image.

[0096] In its implementation, the encoder may include a feature projection layer, a relation modeling layer, and a graph construction layer. The feature projection layer maps local self-attention features and cross-window self-attention features to component representations of a unified dimension. The relation modeling layer generates candidate edges based on the similarity between component representations, spatial adjacency, and the correspondence of the second window region. The graph construction layer then builds a component relationship graph with type labels based on the candidate edges. The graph construction process can store node connection relationships in the form of an adjacency matrix or an edge list, and assign a relation label to each edge indicating nesting, parallelism, or inclusion. In practical applications, the encoder may also use graph neural networks, graph attention networks, or deep learning model structures with relation type embeddings; this application does not limit this. The decoder may include an initial state injection unit, a graph semantic fusion unit, and a code sequence output unit. The initial state injection unit writes the global structural representation of the component relationship graph into the decoding start state. The graph semantic fusion unit performs attention matching between the current code fragment to be generated and the graph node representation at each time step. The code sequence output unit generates the target code sequence in the order of HTML tags, hierarchical closure symbols, and attribute fields.

[0097] The structure works as follows: the encoder first organizes the local features obtained from the window and the cross-window self-attention features into a component relationship graph, so that the nodes and edges in the graph simultaneously carry visual semantics and layout hierarchy information. This structured information is then passed to the decoder. When generating code, the decoder uses the component relationship graph as a generation constraint, determining the label nesting order based on the parent-child connection relationships in the graph, and determining the sibling node arrangement order based on the parallel relationships, thus outputting front-end code consistent with the target user interface image. By incorporating local detail features and cross-window layout features into graph relationship modeling, the generated result can simultaneously maintain the accuracy of local component recognition and the consistency of the overall structure.

[0098] With this specific implementation, the hierarchical relationships between components are explicitly expressed in the form of a graph structure. The decoder does not need to rely solely on linear sequences to infer the page structure, and can generate target code sequences with clear nesting relationships. Since cross-window self-attention features are involved in relationship modeling, the code generation results are more accurate in representing duplicate cards, nested containers, and corresponding components across regions. The consistency between the generated code and the target user interface image is improved, and the amount of subsequent manual correction is correspondingly reduced.

[0099] Building upon the aforementioned embodiments, further, by inputting local self-attention features and cross-window self-attention features into the code generation model to generate a target code sequence corresponding to the target user interface image, the method further includes: alternately using local self-attention features and cross-window self-attention features in the encoder to extract local details and global layout of the target user interface image; generating the target code sequence in the decoder based on the local details and global layout of the target user interface image, combined with the semantic information of the component relationship graph; and optimizing the model parameters through a reverse verification and feedback mechanism to optimize the target code sequence.

[0100] In its implementation, the encoder can employ a multi-layer attention block concatenation structure. Each attention block first performs local self-attention computation on the feature tensors within the first window region, then fuses the output of the same layer with overlapping features within the second window region across windows. This process is repeated to form an encoded representation that considers both local texture and overall structure. The decoder initializes the generation state based on this encoded representation and, combined with node attributes, edge types, and parent-child constraints in the component relationship graph, outputs the target code according to a preset code syntax order. A reverse verification and feedback mechanism can restore the target code sequence to the page rendering result and perform pixel-level or structural-level comparison with the target user interface image. Based on the differences, feedback signals are generated to update the model parameters.

[0101] This approach enables the encoder to alternate between local and cross-window information modeling, while the decoder uses the component relationship graph to constrain code generation. Reverse verification then performs closed-loop correction on the output, thereby making the generated target code sequence more stable in terms of local element expression, overall layout consistency, and hierarchical semantic correctness.

[0102] Figure 3 This is a schematic diagram of the structure of a front-end code generation device provided in an embodiment of this application, such as... Figure 3 As shown, the front-end code generation device 30 provided in this embodiment includes:

[0103] Image acquisition module 301 is used to acquire the target user interface image;

[0104] The window segmentation module 302 is used to divide the target user interface image into multiple first window regions using a window of a preset size, and to obtain the local self-attention features of the target user interface image within the first window region.

[0105] The window translation module 303 is used to slide the window, divide the target user interface image into multiple second window regions, and obtain the cross-window self-attention features of the target user interface image within the second window region. The second window region and the corresponding first window region have at least partial overlap.

[0106] The code generation module 304 is used to input local self-attention features and cross-window self-attention features into the code generation model to generate target code sequences corresponding to the target user interface image.

[0107] In one possible implementation, the window segmentation module 302 is specifically used to: use windows to divide the target user interface image into multiple non-overlapping first window regions, each first window region containing a preset number of pixels; and calculate local self-attention features based on the pixels in the first window regions.

[0108] In one possible implementation, the window translation module 303 is specifically used to: translate the current window along the horizontal and / or vertical direction of the target user interface image to generate a second window region that at least partially overlaps with the current first window region; and calculate cross-window self-attention features based on the pixels within the second window region.

[0109] In one possible implementation, the front-end code generation device is further specifically used to: dynamically adjust the window size according to the complexity of the target user interface image, and generate a window partitioning scheme corresponding to the target user interface image.

[0110] In one possible implementation, the front-end code generation device is also specifically used to: analyze the component density and nesting level of the target user interface image through a preset complexity evaluation model, and generate a complexity score for the target user interface image.

[0111] Based on complexity scoring, a decision network with a preset window size is used to determine the window size corresponding to the target user interface image.

[0112] The window partitioning module 302 is also specifically used to: divide a window of the target user interface image into multiple first window regions using the window size.

[0113] In one possible implementation, the code generation model includes an encoder and a decoder; the code generation module 304 is specifically used to: construct a component relationship graph corresponding to the target user interface image in the encoder based on local self-attention features and cross-window self-attention features, and model the hierarchical relationship between the components corresponding to the target user interface image; and fuse the semantic information of the component relationship graph in the decoder to generate the target code sequence corresponding to the target user interface image.

[0114] In one possible implementation, the code generation module 304 is further specifically used to: alternately use local self-attention features and cross-window self-attention features in the encoder to extract local details and global layout of the target user interface image; in the decoder, generate a target code sequence based on the local details and global layout of the target user interface image, combined with the semantic information of the component relationship graph; and optimize the target code sequence by optimizing the model parameters through reverse verification and feedback mechanisms.

[0115] The front-end code generation device provided in this embodiment can execute the method provided in the above method embodiment. Its implementation principle and technical effect are similar, and will not be described in detail here.

[0116] Figure 4 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Figure 4 As shown, the electronic device 40 may include a memory 401 and a processor 402. Optionally, the electronic device may also include a transceiver 403, wherein the memory 401 and the processor 402 communicate with each other; for example, the memory 401, the processor 402 and the transceiver 403 may communicate via a communication bus 404, the memory 401 is used to store a computer program, and the processor 402 executes the computer program to implement the method of the above embodiments.

[0117] Optionally, the aforementioned processor can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), etc. The general-purpose processor can be a microprocessor or any conventional processor. The steps in the method embodiments disclosed in this application can be directly implemented by a hardware processor, or implemented by a combination of hardware and software modules within the processor.

[0118] This application also provides a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, implement the methods in any of the above method embodiments.

[0119] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the methods in any of the above method embodiments.

[0120] All or part of the steps in the above method embodiments can be implemented by hardware related to program instructions. The aforementioned program can be stored in a readable memory. When the program is executed, it performs the steps of the above method embodiments; and the aforementioned memory (storage medium) includes: read-only memory (ROM), RAM, flash memory, hard disk, solid-state drive, magnetic tape, floppy disk, optical disk, and any combination thereof.

[0121] This application describes embodiments with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processing unit of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processing unit of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0122] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0123] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1The steps of the function specified in one or more boxes.

[0124] Obviously, those skilled in the art can make various modifications and variations to the embodiments of this application without departing from the spirit and scope of this application. Therefore, if these modifications and variations to the embodiments of this application fall within the scope of this application and its equivalents, this application also intends to include these modifications and variations.

[0125] In this application, the term "comprising" and its variations can refer to non-limiting inclusion; the term "or" and its variations can refer to "and / or". The terms "first", "second", etc., in this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. In this application, "multiple" refers to two or more. "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, and B existing alone. The character " / " generally indicates that the preceding and following related objects have an "or" relationship.

[0126] It should be noted that, for the sake of simplicity, the foregoing method embodiments are all described as a series of actions. However, those skilled in the art should understand that this application is not limited to the described order of actions, as some steps may be performed in other orders or simultaneously according to this application. Furthermore, those skilled in the art should also understand that the embodiments described in the specification are all optional embodiments, and the actions and modules involved are not necessarily essential to this application.

[0127] It should be further noted that although the steps in the flowchart are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowchart may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the sub-steps or stages of other steps.

[0128] It should be understood that the above-described device embodiments are merely illustrative, and the device of this application can also be implemented in other ways. For example, the division of units / modules in the above embodiments is only a logical functional division, and there may be other division methods in actual implementation. For example, multiple units, modules, or components may be combined, or integrated into another system, or some features may be ignored or not executed.

[0129] Furthermore, unless otherwise specified, the functional units / modules in the various embodiments of this application can be integrated into one unit / module, or each unit / module can exist physically separately, or two or more units / modules can be integrated together. The integrated units / modules described above can be implemented in hardware or as software program modules.

[0130] When integrated units / modules are implemented in hardware, the hardware can be digital circuits, analog circuits, etc. The physical implementation of the hardware structure includes, but is not limited to, transistors, memristors, etc. Unless otherwise specified, the processor can be any suitable hardware processor, such as a CPU, GPU, FPGA, DSP, and ASIC, etc. Unless otherwise specified, the storage unit can be any suitable magnetic or magneto-optical storage medium, such as Resistive Random Access Memory (RRAM), Dynamic Random Access Memory (DRAM), Static Random Access Memory (SRAM), Enhanced Dynamic Random Access Memory (EDRAM), High-Bandwidth Memory (HBM), Hybrid Memory Cube (HMC), etc.

[0131] If the integrated unit / module is implemented as a software program module and sold or used as an independent product, it can be stored in a computer-readable storage device (CMD). Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a memory and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this application. The aforementioned memory includes various media capable of storing program code, such as a USB flash drive, read-only memory (ROM), random access memory (RAM), portable hard drive, magnetic disk, or optical disk.

[0132] In the above embodiments, the descriptions of each embodiment have their own emphasis. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments. The technical features of the above embodiments can be combined arbitrarily. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as the combination of these technical features does not contradict each other, it should be considered within the scope of this specification.

[0133] Other embodiments of this application will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of this application that follow the general principles of this application and include common knowledge or customary techniques in the art not disclosed herein.

[0134] It should be understood that this application is not limited to the precise structure described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope.

Claims

1. A front-end code generation method, characterized in that, The method includes: Obtain the target user interface image; Using a window of a preset size, multiple first window regions are divided on the target user interface image, and local self-attention features of the target user interface image within the first window regions are obtained; Slide the window to divide multiple second window regions on the target user interface image, and obtain the cross-window self-attention features of the target user interface image within the second window region. The second window region and the corresponding first window region have at least partial overlap. The local self-attention features and cross-window self-attention features are input into the code generation model to generate the target code sequence corresponding to the target user interface image.

2. The method according to claim 1, characterized in that, The method involves using a window of a preset size to divide the target user interface image into multiple first window regions, and obtaining local self-attention features of the target user interface image within each first window region, including: Using the window, the target user interface image is divided into multiple non-overlapping first window regions, each of which contains a preset number of pixels; The local self-attention features are calculated based on the pixels within the first window region.

3. The method according to claim 2, characterized in that, The process of sliding the window to divide the target user interface image into multiple second window regions and obtaining cross-window self-attention features of the target user interface image within the second window regions includes: The current window is translated along the horizontal and / or vertical direction of the target user interface image to generate a second window area that at least partially overlaps with the current first window area; The cross-window self-attention feature is calculated based on the pixels within the second window region.

4. The method according to claim 1, characterized in that, Before dividing the target user interface image into multiple first window regions using a window of a preset size, the method further includes: Based on the complexity of the target user interface image, the window size is dynamically adjusted to generate a window partitioning scheme corresponding to the target user interface image.

5. The method according to claim 4, characterized in that, The step of dynamically adjusting the window size based on the complexity of the target user interface image to generate a window partitioning scheme corresponding to the target user interface image includes: By using a pre-defined complexity evaluation model, the component density and nesting level of the target user interface image are analyzed to generate a complexity score for the target user interface image. Based on the complexity score, a preset window size decision network is used to determine the window size corresponding to the target user interface image; The window, using a preset size, divides the target user interface image into multiple first window regions, including: Using the window size, a plurality of first window regions are divided on the target user interface image.

6. The method according to claim 1, characterized in that, The code generation model includes an encoder and a decoder; the step of inputting the local self-attention features and cross-window self-attention features into the code generation model to generate the target code sequence corresponding to the target user interface image includes: Based on the local self-attention features and cross-window self-attention features, a component relationship graph corresponding to the target user interface image is constructed in the encoder to model the hierarchical relationship between the components corresponding to the target user interface image. The semantic information of the component relationship graph is fused in the decoder to generate the target code sequence corresponding to the target user interface image.

7. The method according to claim 6, characterized in that, The step of inputting the local self-attention features and cross-window self-attention features into the code generation model to generate the target code sequence corresponding to the target user interface image further includes: The encoder alternately uses local self-attention features and cross-window self-attention features to extract local details and global layout of the target user interface image; In the decoder, the target code sequence is generated based on the local details and global layout of the target user interface image, combined with the semantic information of the component relationship graph; The target code sequence is optimized by using reverse verification and feedback mechanisms to optimize model parameters.

8. A front-end code generation device, characterized in that, The device includes: The image acquisition module is used to acquire the target user interface image; The window segmentation module is used to divide the target user interface image into multiple first window regions using a window of a preset size, and to obtain the local self-attention features of the target user interface image within the first window regions. A window translation module is used to slide the window, divide multiple second window regions on the target user interface image, and obtain cross-window self-attention features of the target user interface image within the second window regions. The second window regions and the corresponding first window regions have at least partial overlap. The code generation module is used to input the local self-attention features and cross-window self-attention features into the code generation model to generate the target code sequence corresponding to the target user interface image.

9. An electronic device, characterized in that, include: A processor, and a memory communicatively connected to the processor; The memory stores computer-executed instructions; The processor executes computer execution instructions stored in the memory to implement the method as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-executable instructions, which, when executed by a processor, are used to implement the method as described in any one of claims 1 to 7.

11. A computer program product, characterized in that, Includes a computer program that, when executed by a processor, implements the method of any one of claims 1 to 7.

Images