A web page code generation method and device based on image block

Abstract

The application relates to a webpage code generation method and device based on image block segmentation. The device comprises a processor, which comprises an image segmentation module, a local code generation module and a global code assembly module. The image segmentation module performs feature scanning on a received webpage design diagram to extract an orthogonal layout isolation belt in the webpage design diagram, and divides the webpage design diagram into multiple image blocks along the orthogonal layout isolation belt to output each image block and corresponding boundary features. The local code generation module calls a multimodal large language model to perform layout-driven chain generation, and generates local codes corresponding to each image block in a preset standardized mark template. The global code assembly module receives the boundary features and the local codes, and performs dynamic strategy selection according to an adaptive relationship between a required context input length corresponding to the local codes and a context capacity threshold of the multimodal large language model, so as to output target webpage codes by using different assembly paths.

Description

Technical Field

[0001] This invention relates to the field of mobile communication technology, and in particular to a method and device for generating web page code based on image segmentation. Background Technology

[0002] Front-end developers typically write web page code based on graphical user interface (GUI) prototypes created by UI designers. This process involves translating visual components (such as elements, layouts, and functions) into Hypertext Markup Language (HTML), Cascading Style Sheets (CSS), and JavaScript code, which is often time-consuming and costly. Due to the heavy burden of generating large amounts of repetitive web page code, over 75.8% of front-end developers have adopted AI tools to improve development efficiency. Therefore, the demand for solutions that can automatically convert web page designs into code is constantly increasing.

[0003] Previously, some existing technologies attempted to generate UI code from simple stylized design images (such as hand-drawn sketches) using smaller models. Recently, the powerful capabilities of multimodal large language models (MLLMs), such as Pix2Struct, GPT-4V, and Claude, have made it possible to directly convert high-resolution web designs into code. Several works have built large-scale corpora for training, including WebSight, WebCode2M, and Web2Code. Notably, Si et al. established a high-quality benchmark and provided a new evaluation metric (i.e., visual scoring) specifically for evaluating the performance of multimodal large language models. Building on this, subsequent research has focused on fine-tuning task-specific multimodal large language models or using augmentation methods to instruct interactive multimodal large language models. Furthermore, Wan et al. proposed DCGen, which improves code generation by synthesizing code directly from the entire design and combining it with natural language descriptions of sub-regions.

[0004] Despite some progress in previous research, fully automated UI composition for complex real-world web pages remains a challenge in this field. Existing methods primarily rely on holistic generation, directly using a multimodal large language model to generate complete web page code from the design. Research has revealed inherent limitations in this approach, often resulting in the loss of some layout information during code generation. Therefore, when dealing with real-world web pages containing diverse styles and complex content, multimodal large language models struggle to preserve the original structure and layout.

[0005] Furthermore, in the image preprocessing stage, existing automatic generation solutions often rely solely on pixel color changes for indiscriminate segmentation when segmenting design drawings, without considering the inherent structural specifications of web front-ends (such as the CSS box model). This can easily lead to irreversible structural fragmentation before the code generation step begins.

[0006] Figure 1 It showcases a real-world example from the screenshot-to-code project, where a multimodal large language model uses prompt words to generate corresponding code from website screenshots. Figure 1 (a) is a web design diagram showing an image of a social media page. The top of the web design diagram contains an area marked by a dashed box: a circular image area on the left and an information area containing usernames and action buttons on the right, both presented in a horizontal side-by-side layout.

[0007] Figure 1 (b) Displays a webpage synthesized using GPT-4V. For example... Figure 1 As shown in (b), the generated webpage layout differs significantly from the original webpage design, especially in the elements marked with boxes. Even with the input instructions explicitly requiring "ensuring the layout is correct," the multimodal large language model (GPT-4V) still incorrectly arranges the originally horizontally distributed circular image areas, usernames, and operation buttons vertically instead of horizontally, and each element is abnormally centered.

[0008] The investigation and analysis show that multimodal large language models have limitations in the process of converting web page designs into web page code, and this phenomenon is not unique. This limitation may be due to the fragility of multimodal large language models in fact interpretation and numerical reasoning, which affects the accurate capture of the position, size, and micro-topology of elements in web page designs.

[0009] Furthermore, long code and context capacity bottlenecks can also lead to global assembly failure. When performing global assembly of locally generated code, the context window capacity of a large multimodal language model often becomes a bottleneck. Forcing the model to handle long code blocks, boundary features, and global layout constraints can easily lead to input length overflow, causing the global code assembly process to be interrupted.

[0010] CN105808628A discloses a webpage transcoding method, which includes: receiving a control data update request sent by a client; obtaining original control data from a pre-loaded original webpage; updating the original control data with new control data stored in a preset form, and submitting a new webpage access request corresponding to the new control data to a webpage server; transcoding the new webpage; and sending the transcoded new webpage to the client. However, this technical solution does not fully automate the UI composition of real webpages.

[0011] Currently, automatically converting graphical user interface (GUI) designs into front-end code using Multimodal Large Language Models (MLLMs) has become an important direction for improving development efficiency. However, CN109683871A often relies solely on simple control boundary detection, failing to incorporate the inherent CSS box model structure specifications of web front-ends. This can easily lead to layout information loss or structural fragmentation when processing complex real-world web pages, causing the front-end CSS box model structure to collapse. Furthermore, when processing high-resolution or long-sequence code, the limitations of MLLMs in spatial logic reasoning and context capacity often cause the global assembly process to be interrupted.

[0012] In summary, overcoming the damage to UI structure caused by traditional image segmentation, eliminating layout distortion caused by natural language description through layout-driven chain generation, and solving the crash problem of constrained models during global assembly of long code are technical challenges that urgently need to be addressed in this field.

[0013] Furthermore, on the one hand, there are differences in understanding among those skilled in the art; on the other hand, the applicant studied a large number of documents and patents when making this invention, but due to space limitations, not all details and contents were listed in detail. However, this does not mean that the present invention does not possess the features of these prior art. On the contrary, the present invention already possesses all the features of the prior art, and the applicant reserves the right to add relevant prior art to the background art. Summary of the Invention

[0014] To address the shortcomings of existing webpage transcoding technologies, which primarily rely on overall generation or local descriptions in natural language, resulting in the vulnerability of multimodal large-scale language models in numerical reasoning and factual interpretation to accurately capture the spatial location, size, and hierarchy of the design, leading to layout information loss and structural fragmentation, this invention aims to solve a series of core technical problems faced by multimodal large-scale language models in the process of generating real webpages. First, it addresses the problem of excessive code generation burden on multimodal large-scale language models due to the complexity and large amount of code required for real webpages, preventing the model from missing details when processing complex areas or skipping the generation of some code blocks due to model output truncation and omission. Second, it solves the problem of constrained models being prone to failure in the global assembly stage, overcoming the problem of numerous assembly interruptions caused by models like DeepSeek-VL2 with limited context windows, which cannot simultaneously handle all code blocks and global layout constraints. Finally, by overcoming the limitations of multimodal large-scale language models in terms of weak layout preservation and limited context carrying capacity, it solves the fundamental problem of insufficient automation in the synthesis of complex, high-resolution real-world webpage UIs, a problem that existing technologies have consistently failed to fully address.

[0015] To address the shortcomings of existing technologies, this invention provides a webpage code generation device based on image segmentation, comprising a processor, an image segmentation module, a local code generation module, and a global code assembly module. The image segmentation module performs feature scanning on a received webpage design image to extract orthogonal layout isolation bands and segments the webpage design image into multiple image blocks along these isolation bands, outputting each image block and its corresponding boundary features. The local code generation module invokes a multimodal large-scale language model to perform layout-driven chained generation, generating local code corresponding to each image block within a pre-set standardized tag template. The global code assembly module receives the boundary features and local code, and dynamically selects strategies based on the compatibility relationship between the required context input length corresponding to the local code and the context capacity threshold of the multimodal large-scale language model, to output target webpage code using different assembly paths.

[0016] The device of this invention breaks down the task of generating long code all at once into a process of generating local code independently for each image block, and then assembling it globally according to a strategy. This approach effectively reduces the amount of data processed by a large multimodal language model in a single operation, and determines the accurate position of each image block in the overall webpage through boundary features. This avoids layout chaos that occurs when a large multimodal language model processes complex pages, as well as the risk of device interruption due to excessive data processing.

[0017] According to a preferred embodiment, the global code assembly module performs dynamic strategy selection in the following ways: when the required context input length is greater than the context capacity threshold, coordinate constraint assembly based on boundary features is performed to splice the local code; when the required context input length is not greater than the context capacity threshold, coordinate constraint assembly and semantic adaptive assembly are performed respectively to generate candidate code, and a verification operator is called to evaluate the candidate code and perform automatic optimization to output the target webpage code.

[0018] Preferably, the global code assembly module calls the verification operator to perform feature comparison and automatic optimization of candidate code to output the target webpage code. This dynamic strategy can flexibly switch the assembly method according to the model's processing capability. When the processing capability of the multimodal large language model is limited, only coordinate constraint assembly is used to ensure that the code can be successfully assembled and output; when the model's processing capability is sufficient, both assembly methods are used simultaneously and automatically compared and scored to select the webpage code with the most accurate page layout and the best visual effect.

[0019] According to a preferred embodiment, the image segmentation module is further configured to: identify initial image blocks with an area lower than a preset area threshold before outputting each image block; merge the initial image block with spatially adjacent image blocks, and redetermine the boundary features of the merged region.

[0020] Before generating code, the image segmentation module merges fragmented image blocks that are too small. This prevents the complete web page components from being broken up due to overly dense segmentation lines, making the segmented image block areas more consistent with the actual structure of the web page, thus laying a good foundation for accurate code generation and stitching in the future.

[0021] According to a preferred embodiment, the image segmentation module uses an adaptive segmentation threshold when extracting orthogonal layout isolation bands. The image segmentation module is also configured to: acquire spacing distribution features characterizing the current web page design or historical design data; and set the segmentation threshold used to determine whether a candidate isolation band in the web page design is a valid orthogonal layout isolation band as a dynamic parameter determined based on the spacing distribution features, so that the segmentation threshold is adaptively adjusted according to the layout density of the web page design.

[0022] The image segmentation module no longer uses fixed values ​​for image segmentation, but can automatically adjust the judgment distance based on the density of elements on the webpage design. This avoids erroneously merging different elements in densely populated areas or creating unnecessary blank blocks in sparsely populated areas, thus more accurately segmenting along the actual edges of webpage components.

[0023] According to a preferred embodiment, the local code generation module is configured to perform layout-driven chained generation through the following steps: executing the principle of prioritizing the output of spatial layout relationship structure code to establish the layout elements corresponding to the image block and the hierarchical relationships between the elements; generating content detail code of the image block based on the layout elements and the hierarchical relationships between the elements; and calling a multimodal large language model to perform consistency verification and repair on the generated content detail code.

[0024] This local code generation module guides the multimodal large language model to execute step by step, first determining the spatial location and then generating the internal details. This approach not only reduces the difficulty of a single computation for the multimodal large language model, but also compensates for the shortcomings of large language models in handling spatial relationships, preventing incorrect stacking or misalignment of generated web page components.

[0025] According to a preferred embodiment, the local code generation module calls a multimodal large language model to perform consistency verification and repair on the generated content detail code. This includes comparing the text, color, and / or background style in the content detail code with the visual features of the image block, and refining the content detail code based on the comparison results.

[0026] The partial code generation module adds an automatic verification process after generating the partial code. This allows the module to detect and correct text, color, or background errors in the partial code before concatenating all the code together, thus ensuring that each individually generated code snippet is highly consistent with the original webpage design.

[0027] According to a preferred embodiment, the global code assembly module is configured to calculate the required context input length by: obtaining a first input length corresponding to all local codes to be assembled; obtaining a second input length corresponding to the boundary features corresponding to all local codes; obtaining global assembly constraint information for indicating spatial location and hierarchical structure, and determining its corresponding third input length; and determining the required context input length based on the first input length, the second input length, and the third input length.

[0028] The global code assembly module calculates the length of all information that needs to be processed, transforming the abstract assessment of whether a large multimodal language model can handle the input data into an objective standard that can be measured by specific numerical values. This effectively prevents generation interruptions caused by the amount of data exceeding the capacity of the large multimodal language model during runtime.

[0029] According to a preferred embodiment, the global code assembly module calls the verification operator to evaluate candidate code in the following ways: calling the rendering engine to perform page rendering on the candidate code and obtaining the corresponding rendered page; comparing the features of the rendered page with the original web page design, and calculating the page pixel difference measure and semantic similarity measure between the rendered page and the original web page design respectively; and calculating the verification operator based on the page pixel difference measure and semantic similarity measure.

[0030] The global code assembly module establishes an objective evaluation mechanism through verification operators. This module simultaneously integrates the difference features of page pixels and the semantic similarity of high-dimensional visual data for scoring, avoiding the instability and inaccuracy that may occur when relying solely on subjective evaluation using multimodal large-scale language models.

[0031] The present invention provides a webpage code generation method based on image segmentation from a second aspect. The method includes: performing feature scanning on a received webpage design image to extract orthogonal layout isolation bands in the webpage design image, and segmenting the webpage design image into multiple image blocks along the orthogonal layout isolation bands to output each image block and its corresponding boundary features; calling a multimodal large language model to perform layout-driven chained generation to generate local code corresponding to each image block within a preset standardized tag template; receiving the boundary features and local code, and performing dynamic strategy selection based on the adaptability relationship between the required context input length corresponding to the local code and the context capacity threshold of the multimodal large language model to output target webpage code using different assembly paths.

[0032] The method of this invention breaks down the webpage code conversion task into a process of first extracting features in blocks, then gradually generating local code step by step and dynamically assembling it. This approach solves the problem of layout deviation that easily occurs when traditional methods directly generate long code, improves the accuracy of multimodal large language models in generating every detail of the code, and enables multimodal large language models with different computing capacities to successfully complete complex automated webpage generation tasks.

[0033] According to a preferred embodiment, the dynamic strategy selection method includes: when the required context input length is greater than the context capacity threshold, performing coordinate constraint assembly based on boundary features to splice the local code; when the required context input length is not greater than the context capacity threshold, performing coordinate constraint assembly and semantic adaptive assembly respectively to generate candidate code, and calling a verification operator to evaluate the candidate code and perform automatic optimization to output the target webpage code.

[0034] This dynamic strategy selection step can avoid the risk of losing code blocks in a large multimodal language model due to excessive data volume based on the required context length. While ensuring the success rate of assembly, it also ensures that the generated web page code maintains a high degree of consistency in overall layout and visual effect. Attached Figure Description

[0035] Figure 1 This is a schematic diagram of a real-world counterexample from the well-known project "screenshot-to-code" provided by the present invention; Figure 2 This is a schematic diagram of the workflow of the LaTCoder provided by the present invention; Figure 3 This is a schematic diagram illustrating an example of orthogonal layout isolation zone detection provided by the present invention; Figure 4 This is a schematic diagram of the simplified instruction set (including standardized marker templates) used by the local code generation module provided by this invention to perform layout-driven chained generation; Figure 5 This is a schematic diagram of the visual rating decomposition on the CC-HARD dataset provided by the present invention; Figure 6 This is a schematic diagram provided by the present invention, which uses GPT-4o as the backend to perform pairwise human preference evaluation on the baseline method and LaTCoder on the CC-HARD dataset, employing majority voting by six annotators. Figure 7 This is a schematic diagram illustrating the LaTCoder provided by this invention in a case study of sample examples generated using a large multimodal language model in the backend, compared with other baseline methods. Figure 8 This is a schematic diagram of the complete constraint rules for local code generation provided by the present invention; Figure 9 This is a schematic diagram of the module connection of the web page code generation device based on image segmentation according to the present invention.

[0036] List of reference numerals 100: Processor; 110: Image segmentation module; 111: Image reading unit; 112: Optical character recognition unit; 113: Segmentation unit; 114: Block merging unit; 120: Local code generation module; 130: Global code assembly module; 140: Orthogonal layout isolation zone; 150: Multimodal large language model interface. Detailed Implementation

[0037] The following is a detailed explanation with reference to the accompanying drawings.

[0038] This invention provides explanations for some terms and nouns.

[0039] LaTCoder: This is an abbreviation for the technical architecture of the webpage code generation method based on image segmentation of the present invention. This architecture aims to overcome the deficiencies of layout distortion and computational resource overload that occur in the synthesis of long-sequence code using multimodal large language models. Its complete closed-loop execution logic mainly includes three progressive stages. First, spatial boundary establishment: the device extracts orthogonal layout isolation bands through the image segmentation module 110, dividing the webpage design into multiple image blocks and accurately anchoring the boundary features of each image block. Second, local independent reasoning: the local code generation module 120 calls the multimodal large language model to perform layout-driven chain generation within a pre-set standardized tag template; during this process, the device prioritizes outputting spatial layout relationship structure code, thereby completing the block-by-block output of local code. Finally, global dynamic restoration: the global code assembly module 130 triggers a dynamic strategy selection mechanism based on the required context input length, flexibly scheduling and executing coordinate constraint assembly or semantic adaptive assembly, and finally calling the verification operator to perform feature comparison and automatic optimization of all candidate codes to output the final target webpage code.

[0040] Orthogonal Layout Separator 140: A feature pixel band extending horizontally or vertically in a web design. The relative distance between any separator and its spatially closest adjacent separator is limited by a segmentation threshold determined based on spacing distribution characteristics, and it does not cross any identified text areas. It is used to divide a web design into structured image blocks that conform to the front-end CSS box model specification. The orthogonal layout separator 140 can be a background gap band of various colors in a continuous rectangular blank or white space area, or a graphic dividing line of various types with horizontal or vertical straight lines of specified pixel width, or a color block boundary extension band extending along the horizontal or vertical contour boundary of the interface between two color blocks.

[0041] Spacing distribution features refer to the feature quantities extracted after statistical analysis of the spatial distribution of background gaps, margins, or padding between UI elements in a web page design. Spacing distribution features reflect the density of page layout and its orthogonal topological regularity in the horizontal and vertical dimensions. Spacing distribution features include: the frequency distribution of candidate isolation band widths in the page, the mean spacing, the median spacing, and the representative layout step size. Spacing distribution features provide a quantitative benchmark for determining the segmentation threshold of effective orthogonal layout isolation bands, enabling devices to distinguish between the physical separation space between components and the subtle visual gaps within components.

[0042] Boundary features: A data structure that records the original spatial topological position of each image block in the web page design. It usually includes coordinate values ​​such as top, bottom, left, and right, and is used to accurately anchor the image blocks to the overall layout during the local code generation and global code assembly stages.

[0043] Layout-driven chain generation: This is a sequential processing mechanism based on the "layout as thinking" approach. It breaks down web page design into multiple independent thinking units (image blocks) and executes them block by block by a multimodal large language model. The execution principle is to first output the spatial layout relationship structure code to establish the layout elements and hierarchical relationships, then generate the content detail code based on the relationships, and finally perform consistency checks and polishing on the generated text, colors and / or background styles.

[0044] Standardized tag templates: Pre-defined fixed frame structures (such as HTML templates using the Tailwind framework) are used to impose mandatory tag organization constraints on the output stream of multimodal large language models, ensuring that the generated local code is consistent in structure, tag organization, and code style, reducing the risk of style conflicts during splicing.

[0045] Coordinate-constrained assembly, also known as absolute positioning assembly, is the fundamental path strategy for restoring the overall layout in the "layout as thinking" approach. Based on the boundary features output during image segmentation, coordinate-constrained assembly encapsulates the local code of each image block within a parent container node, directly sets its position and size style attributes, and then performs the concatenation of the local code.

[0046] Semantic adaptive assembly: This is assembly based on a multimodal large-scale language model, and is a more flexible advanced path strategy within the "layout as thinking" approach. Semantic adaptive assembly combines prompt words with the logical reasoning capabilities of a multimodal large-scale language model, comprehensively analyzes the layout information provided by all boundary features, and merges and assembles local code to ensure that the generated content and position match the original webpage design.

[0047] Validation operator: An automatically calculated metric with no-reference characteristics. It executes page rendering on candidate code by calling the rendering engine, compares the rendered page with the original web page design, and fuses the calculation of page pixel difference and semantic similarity metrics to quantify and score different assembly results. The validation operator includes data such as validation score, validation index, validation ratio, and validation degree.

[0048] The Mean Absolute Error (MAE) is an objective metric used to quantify the pixel-level differences between the rendered page and the original web page design. Its score reflects the performance of the local code generation and assembly stages in terms of the alignment accuracy of image block content features and spatial coordinate offset.

[0049] Semantic Similarity Measure (CLIP): One of the calculation parameters of the verification operator, it measures the coherence and structural fidelity of the target webpage code in the overall visual performance of the page after global assembly and rendering by capturing similarity features at the high-dimensional semantic level.

[0050] To alleviate the performance limitations of large multimodal language models in capturing layout information, this invention provides a webpage code generation method and device based on image segmentation. For example... Figure 9 As shown, the processor 100 of the device includes an image segmentation module 110, a local code generation module 120, and a global code assembly module 130. Unlike the traditional one-time overall code generation paradigm, this invention decouples the code synthesis task into sequentially processable logical steps through a layout-driven chain generation mechanism, achieving block-by-block generation of web page code. During execution, the web page design is segmented into multiple image blocks, each serving as an independent processing unit. Specifically, the image segmentation module 110 first divides the web page design into image blocks and extracts the boundary features corresponding to each image block; then, the local code generation module 120 reads each image block, calls a multimodal large-scale language model to perform layout-driven chain generation, and outputs the local code corresponding to each image block within a preset standardized tag template; next, under the constraints of the boundary features, the global code assembly module 130 performs coordinate constraint assembly or semantic adaptive assembly according to a dynamic strategy, integrating the generated local codes into complete web page code; finally, by calling a verification operator, the code generated by different assembly paths is evaluated and optimized, and the target web page code is output. Since each image patch is spatially anchored within the overall layout through boundary features, this mechanism reduces the limitations of large multimodal language models in spatial dependency modeling. Furthermore, decoupling the overall generation into block-level code synthesis reduces the computational load on large multimodal language models when processing long sequences of code in a single operation, and improves the accuracy of content-detail code generation.

[0051] To verify the effectiveness and generalization ability of the proposed webpage code generation method, this invention sets up a test dataset containing highly complex layout features. Specifically, webpage samples are collected from public network datasets, and image-code pairing data is constructed to form a complex page test benchmark. In the quantitative evaluation, the device uses multiple multimodal large language models as the generation framework, and compares the method provided by this invention with existing benchmark models on test sets of varying complexity. The test results show that the method of this invention improves the performance of objective evaluation metrics when combined with different large language models, especially in the quantitative metric representing the similarity of HTML Document Object Model (DOM) tree substructures. Simultaneously, comparative evaluation experiments confirm that the webpage code generated by this method has advantages in front-end rendering fidelity and structural coherence, verifying the stability of this method in maintaining page structural correctness.

[0052] As described above, the method and apparatus provided by this invention enhance the structural fidelity of web page design conversion into code through a layout-driven chain generation mechanism and controlled assembly logic. Verification based on a constructed complex layout dataset demonstrates the effectiveness of the apparatus in overcoming the layout loss problem caused by the weak numerical reasoning capabilities of large multimodal language models, thus expanding the engineering applicability of large multimodal language models in the field of automated front-end engineering code synthesis.

[0053] The present invention aims to convert web page design images (such as website screenshots or design sketches) into corresponding web page code. A typical web page consists of the following core components: an HTML structure configured to define structural elements (e.g., ... 、 <button>and The invention focuses on generating static webpage code from webpage design drafts. The visual appearance of the rendered static webpage is determined by both the HTML structure and CSS styles. Therefore, webpages with the same HTML structure will exhibit significant visual deviations during rendering due to differences in CSS properties. The core technology for executing layout-driven code generation tasks using a multimodal large language model lies in accurately extracting the element size and spatial position information from the webpage design drafts and precisely mapping this information to the target HTML structure and CSS style code.

[0054] The webpage code generation device based on image segmentation of the present invention, such as... Figure 9 As shown, in the processor, the image segmentation module 110 is connected to the local code generation module 120 and the global code assembly module 130, respectively. The image segmentation module 110, together with the local code generation module 120 and the global code assembly module 130, constitutes an automated web page transcoding processing structure. This structure enables it to prevent layout fragmentation and eliminate the interruption of multimodal large language models due to computational overload when processing high-resolution complex web page designs through spatial decoupling and dynamic strategy selection.

[0055] The image segmentation module 110 performs feature scanning on the received webpage design image to extract the orthogonal layout isolation band 140 in the webpage design image, and segments the webpage design image into multiple image blocks along the orthogonal layout isolation band 140 to output each image block and its corresponding boundary features. The image segmentation module 110 sends each image block to the local code generation module 120 and sends the corresponding boundary features to the global code assembly module 130, such as... Figure 9 As shown.

[0056] The local code generation module 120 is connected to the image segmentation module 110 and is used to receive image blocks from the image segmentation module 110. The local code generation module 120 calls a multimodal large-scale language model to perform layout-driven chained generation, generating local codes corresponding to each image block within a preset standardized tag template. The local code generation module 120 is connected to the global code assembly module 130, so that the local code generation module 120 sends the generated local codes to the global code assembly module 130, such as... Figure 9 As shown.

[0057] The global code assembly module 130 receives boundary features from the image segmentation module 110. The global code assembly module 130 receives local code from the local code generation module 120. Based on the adaptability relationship between the required context input length corresponding to the local code and the context capacity threshold of the multimodal large language model, the global code assembly module 130 performs dynamic strategy selection to output target webpage code using different assembly paths.

[0058] To support the data flow and computing power of the aforementioned modules, the device of this invention also includes a storage module communicatively connected to the processor 100 and a multimodal large-scale language model interface 150. The storage module persistently stores images, boundary features, and generated local code using a document-oriented database, and uses an index query mechanism to achieve efficient data retrieval across modules. For example... Figure 9 As shown, the multimodal large language model interface 150 is communicatively connected to the local code generation module 120 and the global code assembly module 130, respectively, and is configured to access the multimodal large language model to provide logical reasoning computing power support for the generation of image blocks into local code and the assembly of global code.

[0059] Furthermore, the device of the present invention also includes a service module communicatively connected to the processor 100. This service module is configured to provide a Web API interface, enabling users or external clients to upload web page design drafts and obtain online previews of the rendered target web page code.

[0060] The device of the present invention also provides a terminal, which is communicatively connected to the above-mentioned web page code generation device based on image segmentation, and is configured to realize the automated conversion and rendering display from high-resolution web page design to front-end code by calling the Web API interface provided by the device of the present invention.

[0061] The present invention also provides a computer-readable storage medium having computer instructions stored thereon. When executed by a processor, the computer instructions implement the webpage code generation method based on image segmentation of the present invention.

[0062] According to a preferred embodiment, the dynamic strategy executed by the global code assembly module 130 of the present invention includes: when the required context input length is greater than the context capacity threshold, performing coordinate constraint assembly based on boundary features to splice the local code; when the required context input length is not greater than the context capacity threshold, performing coordinate constraint assembly and semantic adaptive assembly respectively to generate candidate code, and calling a verification operator to evaluate the candidate code for automatic optimization to output the target webpage code.

[0063] When the processing power of a multimodal large language model is limited, the global code assembly module 130 only uses coordinate constraint assembly to ensure that each local code can be successfully spliced ​​and output; when the processing power of a multimodal large language model is sufficient, the global code assembly module 130 uses both coordinate constraint assembly and semantic adaptive assembly and automatically performs scoring and comparison to select the target webpage code with the most accurate page layout and the best visual reproduction effect.

[0064] For the high-resolution webpage design, the embodiments of the present invention aim to automatically generate the corresponding HTML structure and CSS style code. The conversion task from the webpage design to the target webpage code can be represented as a mapping function F: I→(H,S), where, It is the size of Input webpage design, This represents the generated HTML element. CSS rules (property value pairs) are specified. The goal is to optimize the rendered output. Minimize the visual difference between input I and input I: .

[0065] In the above formula, I represents the input webpage design. This represents the generated HTML code element. This represents a CSS style rule containing attribute-value pairs. The optimization objective of this transformation task is to minimize the perceptual distance between the rendered output R and the input I. .

[0066] like Figure 2 As shown, the workflow of the method provided by this invention is implemented through three functional execution stages, corresponding to three components: (a) image segmentation module 110, (b) local code generation module 120, and (c) global code assembly module 130. In the execution process, firstly, the image segmentation module 110 segments the webpage design image, converting it into a list of image blocks containing multiple sub-regions, and outputs boundary features containing top, bottom, left, and right coordinate values. Subsequently, the local code generation module 120 receives each image block from the image segmentation module 110, and, in conjunction with the input prompts, calls a multimodal large-scale language model to perform layout-driven chained generation, outputting local codes corresponding to each image block (e.g., Block 1 to Block n correspond to...). (Code snippet). Finally, the global code assembly module 130 receives boundary features and local code, and evaluates them in two parallel assembly paths according to a dynamic strategy selection mechanism: one is a semantic adaptive assembly strategy that combines prompt words with the reasoning capabilities of a multimodal large-scale language model, and the other is a coordinate constraint assembly strategy that directly injects positional style attributes into the local code based on boundary features. The global code assembly module 130 uses either coordinate constraint assembly or semantic adaptive assembly strategies to assemble the generated local code, and outputs a complete HTML structure (such as...) based on the dynamic strategy selection mechanism.<!DOCTYPE html> The target webpage code (declaration). By utilizing boundary features to anchor each image block to its original spatial topology position in the webpage design drawing, the method of this invention achieves high-fidelity restoration of the webpage structure while reducing the computational load of this multimodal large language model when generating long sequence codes in a single process.

[0067] The image-block-based webpage code generation device of the present invention executes the image-block-based webpage code generation method. The specific steps of the image-block-based webpage code generation method are as follows.

[0068] S100: Image segmentation module 110 segments the web page design and outputs boundary features.

[0069] S101: The image segmentation module 110 performs feature scanning on the received web page design drawing and extracts the orthogonal layout isolation band 140 in the web page design drawing by performing a global scan in the horizontal and vertical directions.

[0070] Segmenting the webpage design into image blocks of appropriate dimensions is a crucial preliminary stage for performing local code synthesis. Traditional image segmentation methods (such as Mask R-CNN and U-Net models) are typically configured to extract irregular object boundaries through instance segmentation or semantic segmentation. However, front-end webpage layouts strictly adhere to the CSS box model specification, with their underlying logic based on structured rectangular containers as fundamental building blocks. Based on this structural characteristic, this invention provides an isolation band extraction algorithm adapted to the characteristics of front-end engineering. This algorithm is configured to divide the webpage design into spatially aligned image blocks by detecting orthogonal layout isolation bands 140, ensuring that the block boundaries match the webpage typography characteristics of HTML and CSS layout devices.

[0071] In the specific execution logic of the image segmentation module 110, to achieve accurate segmentation of the webpage design into gridded image blocks, the image segmentation module 110 identifies a set of effective orthogonal layout isolation bands 140 that can completely cover and segment the webpage design. To adapt to the design intent of the layout-driven chain generation mechanism, the image segmentation module 110 adopts isolation band recognition rules that combine webpage layout features: the orthogonal layout isolation band 140 is defined as a feature pixel band extending along a horizontal or vertical direction, and the relative distance between any orthogonal layout isolation band 140 and its spatially closest adjacent isolation band is subject to a preset segmentation threshold. The limitations of this method are as follows: In the specific execution process, the image segmentation module 110 calls a preset recursive scanning algorithm, configured to perform a row-by-row or column-by-column linear feature scan on the input image along a single orthogonal dimension (i.e., only along the horizontal direction or only along the vertical direction) to capture candidate isolation bands. Furthermore, this recursive scanning algorithm is triggered cyclically within the global view of the web page design and each recursively generated sub-image region until no segmentation threshold can be extracted within the currently traversed region. The process continues until the newly added isolation zones are added, and then the complete set of orthogonal layout isolation zones 140 is output.

[0072] To reduce the computational resource consumption of high-resolution design drafts, the image segmentation module 110 employs a grid sampling technique, performing discrete sampling on the web design draft at fixed intervals of a preset value (e.g., 5 pixels), rather than scanning pixel by pixel. Simultaneously, the image segmentation module 110 filters a preset number (e.g., 10) of pixels at the edges of the design draft to eliminate boundary noise. To avoid the orthogonal layout isolation band 140 truncating complete text paragraphs, the optical character recognition unit 112 (e.g., EasyOCR) in the image segmentation module 110 extracts text region features. The optical character recognition unit 112 extracts the font family metadata corresponding to each text region by comparing it with a preset feature glyph library, and transmits this metadata to the local code generation module 120 as constraint parameters for local code generation. Through this pre-identification, the local code generation module 120 can accurately specify Google Fonts names when calling a multimodal large language model. The optical character recognition unit 112 merges the coordinates of adjacent text boundary features with a spacing within a preset tolerance range (e.g., 20 pixels) in the horizontal or vertical direction to establish a complete text paragraph region. When scanning is performed, the image segmentation module 110 determines, based on the coordinate exclusivity condition, that the coordinates of the candidate isolation band do not overlap with the coordinates of any recognized text region, so that the extracted orthogonal layout isolation band 140 avoids the recognized text region and ensures the integrity of the text structure.

[0073] During the recognition process, an effective orthogonal layout isolation band 140 needs to satisfy the feature pixel band condition and not cross any text region. To this end, the image segmentation module 110 employs the following optimization method: (1) Ensure the integrity of the text region. The orthogonal layout isolation band 140 may segment the text region, especially when it spans line or paragraph gaps. To avoid this, the optical character recognition unit 112 (OCR) in the image segmentation module 110 detects the text region (e.g., ...). Figure 3 The area marked by the dashed box in the middle), and ensure that the orthogonal layout isolation band 140 is considered valid only if it does not cross any text area.

[0074] (2) Improve efficiency through grid sampling. Pixel-by-pixel scanning on a high-resolution web design is both inefficient and computationally expensive. In contrast, this invention uses grid sampling technology to sample pixels on the web design at fixed intervals when determining feature pixel bands, instead of scanning pixel by pixel.

[0075] (3) Ignore edge pixels. Pixels at the edges of the web page design may hinder the determination of feature pixel bands. To address this issue, during the determination process, the image segmentation module 110 ignores the first few pixels of the edge (e.g., ...). Figure 3 (As shown by the hollow circles at the beginning and end of each scan line).

[0076] After detecting all orthogonal layout isolation bands 140, the web page design is segmented into a series of image blocks, which are represented by boundary features. Image blocks that are too small after segmentation are merged with adjacent image blocks. These boundary features are further recorded for local code generation and global code assembly; the specific logic of this extraction algorithm is described below.

[0077] This algorithm aims to extract a set S of effective orthogonal layout isolation bands 140 that conform to the front-end CSS box model specification, protect text integrity, and meet typographic density requirements from a web page design image I. The algorithm inputs are the image I to be processed and a dynamically adaptive minimum distance threshold determined based on spacing distribution characteristics. .

[0078] In the initialization phase, the algorithm first sets the initial horizontal reference anchor point h, used to control row spacing, to the top edge of the image, and sets the initial vertical reference anchor point v, used to control column spacing, to the left edge of the image. Simultaneously, an empty set S is created to store the final qualified dividing lines. Subsequently, the algorithm enters the main phase of prioritizing the search for horizontal dividing lines. In this phase, the algorithm uses a grid sampling technique instead of pixel-by-pixel scanning, continuously scanning line by line from top to bottom along the image. For the currently scanned candidate line h1, the algorithm determines whether it simultaneously meets the following three conditions: First, after ignoring a preset number of edge pixels at both ends of the design drawing, the line must exhibit characteristic pixel attributes such as blank background gaps or dividing lines at the grid sampling points; second, the coordinates of the line cannot cross any text areas pre-identified by OCR to ensure the integrity of the underlying text structure; third, the distance between the line and the previous reference anchor point h must be greater than or equal to an adaptive threshold. If candidate row h1 satisfies all the above conditions, the algorithm adds it to set S and updates the horizontal reference anchor point h to the current h1, serving as the new starting point for calculating the spacing of the next line. This search process will be executed cyclically until all rows of the image have been scanned.

[0079] After completing all horizontal scans, the algorithm checks the state of set S. If set S is not empty at this point, it means that the system has successfully extracted at least one qualified horizontal isolation strip. The algorithm will then return to set S and terminate its operation prematurely, without initiating further vertical searches.

[0080] If set S is empty, the algorithm performs a vertical segmentation line search. In this stage, the image is scanned column by column from left to right using a grid sampling method. For the currently scanned candidate column v1, the algorithm uses the same composite logic as the horizontal search: that is, the column must present a characteristic pixel band after ignoring edge pixels, not cross any identified text regions, and have a distance from the previous vertical reference anchor point v greater than or equal to an adaptive threshold. Candidate columns that meet all conditions will be added to set S, and the vertical reference anchor point v will be updated synchronously. After traversing all columns, the algorithm completes and outputs the final set S of orthogonal layout isolation bands 140.

[0081] Figure 3 An example of detecting orthogonal layout isolation bands 140 is shown. The image segmentation module 110 first forms a grid by sampling points at fixed intervals across the entire web page design (e.g., ...). Figure 3 The background is shown as a uniformly distributed array of light-colored dots, which improves detection efficiency by replacing pixel-by-pixel scanning with a grid sampling technique. The image segmentation module 110 then scans the grid row by row or column by column along the horizontal or vertical direction to identify potential orthogonal layout isolation zones 140. In this process, a valid orthogonal layout isolation zone 140 must be a feature pixel band that does not cross any text area. To minimize the influence of edges, one edge pixel is ignored during the determination process. The image segmentation module 110 ultimately identifies the horizontal and vertical scan lines as orthogonal layout isolation zones 140, as shown below. Figure 3 The sequence of green solid dots marked in the image is shown. After identifying a valid orthogonal layout isolation band 140, the image segmentation module 110 performs a segmentation action at the corresponding position (e.g., ...). Figure 3 (As shown by the scissors icon with dividing labels), to divide the web design into multiple image blocks. It is worth noting that although edge pixels are ignored, they are still considered part of the orthogonal layout isolation band 140.

[0082] According to a preferred embodiment, the image segmentation module 110 extracts the orthogonal layout isolation bands 140 using an adaptive segmentation threshold. This segmentation threshold is configured to conform to the structural constraint boundaries of the front-end CSS box model specification, rather than simply using empirical image color parameters. The image segmentation module 110 acquires the spacing distribution characteristics characterizing the current webpage design or historical design data. The image segmentation module 110 sets the segmentation threshold used to determine whether a candidate isolation band in the webpage design is a valid orthogonal layout isolation band 140 as a dynamic parameter determined based on the spacing distribution characteristics, so that the segmentation threshold adaptively adjusts with the layout density of the webpage design.

[0083] In the image segmentation stage, the segmentation threshold used by the image segmentation module 110 is not a fixed empirical constant, but a dynamic parameter that is adaptively adjusted according to the overall layout density and layout characteristics of the web page design to be processed.

[0084] Specifically, the image segmentation module 110 acquires the spacing distribution features based on statistics from a preset reference webpage sample set. Preferably, to ensure the breadth and representativeness of the statistical basis, the reference webpage sample set can use mainstream webpage datasets including WebCode2M, Design2Code, or WebSight. Subsequently, the image segmentation module 110 extracts the distribution data of the main horizontal and vertical spacing in the pages of the reference webpage sample set, and uses the statistical characteristic parameters of this distribution data (e.g., taking the median or dominant frequency value) as the basic spacing reference scale.

[0085] Preferably, the image segmentation module 110 can perform global histogram analysis on the current webpage design to statistically analyze the pixel grayscale change frequency or the mean of the blank spacing as a spacing distribution feature.

[0086] Based on this, the image segmentation module 110 sets the segmentation threshold used to perform image segmentation as a dynamic scaling parameter based on the basic spacing reference scale, and determines whether the candidate isolation band is a valid orthogonal layout isolation band 140 accordingly. Preferably, the image segmentation module 110 can set the initial threshold for this adaptive adjustment to 50 pixels, while limiting the maximum depth of the recursive search for the orthogonal layout isolation band 140 (e.g., set to 3), so that the segmentation threshold can be adaptively matched strictly according to the layout density of the design drawing. This allows the segmentation threshold to adapt to the page design style. For pages with tighter spacing, the segmentation threshold adaptively determined by the image segmentation module 110 will be reduced accordingly to avoid excessive merging; for pages with looser spacing, the segmentation threshold adaptively determined by the image segmentation module 110 will be increased accordingly to avoid generating too many fragmented blocks.

[0087] Compared to fixed thresholds set based on human experience, the image segmentation module 110, through the aforementioned dynamic mechanism, can more stably align the orthogonal layout isolation band 140 with the actual boundaries of web page components, avoiding missegmentation due to local pixel gradients. This reduces the likelihood of incorrectly segmenting the same structured image block or incorrectly merging multiple structured image blocks. Therefore, by setting a segmentation threshold, the image segmentation module 110 establishes a correspondence between the segmentation threshold and the actual layout spacing of the page, making the segmentation results more consistent with the container structure organization of the front-end CSS box model.

[0088] Preferably, such as Figure 9 As shown, the image segmentation module 110 includes an image reading unit 111, an optical character recognition unit 112, a segmentation unit 113, and a block merging unit 114. The image reading unit 111 is connected to the optical character recognition unit 112. The image reading unit 111 is connected to the segmentation unit 113. The optical character recognition unit 112 is connected to the segmentation unit 113. The segmentation unit 113 is connected to the block merging unit 114.

[0089] like Figure 9 As shown, the image reading unit 111 receives a webpage design from an external source. The image reading unit 111 sends the webpage design to the optical character recognition unit 112 and the segmentation unit 113. The optical character recognition unit 112 receives the webpage design from the image reading unit 111. The optical character recognition unit 112 performs feature extraction before image segmentation, allowing the image segmentation module 110 to accurately identify text region features in the webpage design, thereby preventing structural fragmentation due to incorrect segmentation. Furthermore, when the webpage design has a complex mixed text and image layout, the optical character recognition unit 112 identifies text boundary coordinates during feature scanning, ensuring that the orthogonal layout isolation band 140 extracted by the segmentation unit 113 establishes a spatially mutually exclusive relationship with the text region coordinates in terms of coordinate occupancy, thus ensuring that the orthogonal layout isolation band 140 does not cross the text content during its extension, maintaining the integrity of the text structure. The optical character recognition unit 112 sends the text region features to the segmentation unit 113.

[0090] like Figure 9 As shown, segmentation unit 113 receives a web page design from image reading unit 111. Segmentation unit 113 receives text region features from optical character recognition unit 112. Segmentation unit 113 is used to identify orthogonal layout isolation bands 140 and perform cropping operations. When the web page design has uneven spacing distribution, segmentation unit 113 performs discrete grid sampling according to an adaptive segmentation threshold determined based on spacing distribution features, so that the extracted orthogonal layout isolation bands 140 can dynamically shift their position with local changes in layout density. Segmentation unit 113 sends the initial image block to block merging unit 114.

[0091] like Figure 9 As shown, the block merging unit 114 receives initial image blocks from the segmentation unit 113. The block merging unit 114 is used to identify initial image blocks with an area lower than a preset area threshold and merge them with spatially adjacent image blocks. The block merging unit 114 sends the final output image block to the local code generation module 120 and sends the corresponding boundary features to the global code assembly module 130.

[0092] Unlike the global code assembly module 130, the block merging unit 114 within the image segmentation module 110 differs fundamentally in its processing object and execution stage. The global code assembly module 130 processes the local code already generated by the multimodal large language model, focusing on resolving the post-processing splicing and assembly of long code sequences at the end of device execution. In contrast, the block merging unit 114 processes the raw image regions before code generation, focusing on pre-aggregating physical coordinates during image preprocessing. This block merging unit 114 aims to prevent excessive segmentation and structural fragmentation caused by dense scan lines before entering the code chain generation step, ensuring that the boundary features ultimately sent to the multimodal large language model conform to the container organization logic of the front-end CSS box model. This provides a structurally consistent basic physical unit for subsequent code generation and global assembly.

[0093] S102: Before outputting each image block, the image segmentation module 110 optimizes the boundary features.

[0094] The image segmentation module 110 identifies initial image blocks with an area smaller than a preset area threshold (e.g., 300x300 pixels). Based on the adjacency relationship of spatial coordinates, the image segmentation module 110 merges the initial image block with spatially adjacent image blocks and redefines the boundary features of the merged region to conform to the container structure of the front-end CSS box model. This ensures that the boundary features of the final output correspond to the structured components in the web page design drawing, making the segmentation results structurally consistent and reducing component fragmentation and layout deviations in the subsequent code generation process.

[0095] Specifically, when redefining the boundary features of the merged region, the image segmentation module 110 extracts the original coordinate values ​​of all initial image blocks involved in the merging process. The image segmentation module 110 calculates the minimum top coordinate, minimum left coordinate, maximum bottom coordinate, and maximum right coordinate of each initial image block. Subsequently, the image segmentation module 110 combines these four extreme coordinates to generate the minimum bounding rectangle enclosing all initial image blocks. The image segmentation module 110 outputs the coordinates of this minimum bounding rectangle as the new boundary features of the merged region.

[0096] S103: The image segmentation module 110 segments the web page design into multiple image blocks along the orthogonal layout isolation band 140 to output each image block and its corresponding boundary features.

[0097] Specifically, such as Figure 9 As shown, the image segmentation module 110 sends each image block to the local code generation module 120 so that the local code generation module 120 can generate local code based on the image blocks; at the same time, the image segmentation module 110 sends the corresponding boundary features to the global code assembly module 130 as spatial coordinate reference information when the global code assembly module 130 performs code splicing.

[0098] S200: Local code generation module 120 calls a multimodal large language model to perform chained code generation.

[0099] The local code generation module 120 invokes the multimodal large-scale language model to perform layout-driven chained generation based on boundary features. To ensure consistency in the output code of each image patch in terms of structure, tag organization, and code style, the local code generation module 120 generates local code corresponding to each image patch within a pre-set standardized tag template, eliminating potential code structure incompatibility and style conflicts during subsequent global stitching. The standardized tag template uses a fixed Tailwind framework HTML template to enforce a mandatory tag organization format constraint on the output stream of the multimodal large-scale language model. After generation, the local code generation module 120 sends the generated local code to the global code assembly module 130.

[0100] The local code generation module 120 adapts the instruction set based on the hardware performance parameters of the multimodal large language model and the required context input length for the current image patch. For multimodal large language models with limited context capacity, the local code generation module 120 is configured to call a simplified instruction set to perform code generation, ensuring that the code generation task completes the execution loop within a limited context window and avoiding task truncation and abnormal termination. This simplified instruction set reduces the computational load on the multimodal large language model when processing complex image patches by reducing the length of the prompt text and focusing on core layout instructions, thereby significantly reducing the probability of the multimodal large language model missing details or skipping code blocks in long sequence generation.

[0101] The goal of the local code generation module 120 is to generate local code for image patches. First, this invention uses boundary features to crop all image patches from the webpage design drawing. Then, each image patch is sequentially input into a multimodal large-scale language model for local code generation. This invention carefully designs constraint rules for generating accurate and high-quality webpage code. These constraint rules (see...) Figure 8 The following principles are mainly followed: (1) Use standardized markup templates: Generate Tailwind-style HTML / CSS code within a fixed standardized markup template. Standardized markup templates ensure that the code style of all image blocks remains consistent after global assembly.

[0102] (2) Layout first: Execute the principle of prioritizing the output of spatial layout relationship structure code, first focus on the consistency of appearance and layout, and then strive to maintain the consistency of content.

[0103] (3) Layout-driven chained generation: Step-by-step execution of steps including analyzing image blocks, generating initial content detail code, checking consistency (including text content, color, background and other styles), and polishing the code to complete local code generation. For weaker models (such as DeepSeek-VL2), due to the limited context capacity threshold of multimodal large language models, this invention provides a simplified instruction set. Figure 4 A simplified version of this constraint rule is shown.

[0104] Specifically, such as Figure 4 As shown, this simplified instruction set is configured in the local code generation module 120 to standardize the code generation logic of a multimodal large language model. Figure 4 As can be seen from the layout, this simplified instruction set uses a global dotted box at the top to set the system prompt, assigning a specific developer role to the multimodal large language model. Subsequently, the main body of the instructions is divided into two main core areas, left and right, each with a detailed functional breakdown.

[0105] The left-hand region represents static constraints, corresponding to... Figure 4 The Basic Requirements section outlines the mandatory requirements. Within the dashed boxes of static constraints, it provides an unmodifiable, standardized markup template code structure and specifies, in comment form, the precise anchor points where code snippets generated by the multimodal large language model must be embedded. Below this section, additional supplementary rules, independent of the code structure, are listed. These rules prioritize outputting spatial layout relationship structure code, meaning that consistency between appearance and layout has the highest priority. They also include detailed rules to ensure content consistency, disable comments or placeholders, and specify dependency libraries, thereby preventing code fragmentation from the outset.

[0106] The right-hand area contains the dynamic execution logic, corresponding to... Figure 4 The execution steps (Process Steps) are outlined in this section. This area, with its detailed text description, issues the core task of layout-driven chained generation, guiding the multimodal large-scale language model to complete the generation in a step-by-step manner. The instruction boxes within the dynamic execution logic guide the multimodal large-scale language model to first deeply analyze the image blocks of a specific region, breaking down the UI elements within that region and their relative hierarchical positions; subsequently, based on the analysis results, it initially generates detailed content code. After the initial code generation, a rigorous verification loop is defined below this area, sequentially issuing consistency verification instructions such as text content comparison, color comparison, background and style comparison, and finally instructing the multimodal large-scale language model to complete the final integrated output. Through this process-oriented instruction guidance and step-by-step execution path, the computational load of the multimodal large-scale language model for a single inference iteration when handling complex front-end structures is significantly reduced.

[0107] According to a preferred embodiment, the local code generation module 120 performs layout-driven chain generation in the following manner: it executes the principle of prioritizing the output of spatial layout relationship structure code to establish the layout elements corresponding to the image block and the hierarchical relationship between each element; it generates the content detail code of the image block based on the layout elements and the hierarchical relationship between each element; and it calls a multimodal large language model to perform consistency verification and repair on the generated content detail code.

[0108] Specifically, the local code generation module 120 has layout priority determination logic. When the multimodal large language model is receiving specific image blocks, it establishes a reasoning order of "spatial structure first, content details later," allowing the local code generation module 120 to prioritize filling container code representing the outer contour of visual components in the initially generated HTML structure. When generating content detail code, for image elements, the local code generation module 120 uses placeholder image Uniform Resource Locators with specified size parameters from an image placeholder service website during the initial reasoning phase.

[0109] According to a preferred embodiment, the local code generation module 120 performs consistency verification and repair on the generated content detail code by calling a multimodal large language model. This includes comparing the text, color, and / or background styles in the content detail code with the visual features of the image blocks, and refining the content detail code based on the comparison results. This allows the local code generation module 120 to detect and correct text, color, and / or background style errors generated by the multimodal large language model during code generation before final assembly, thereby ensuring that each individually generated code fragment is highly consistent with the visual features of the original webpage design.

[0110] S300: Global code assembly module 130 dynamically selects the assembly path based on context adaptability.

[0111] The global code assembly module 130 receives boundary features from the image segmentation module 110. The global code assembly module 130 receives local code from the local code generation module 120. Based on the adaptability relationship between the required context input length corresponding to the local code and the context capacity threshold of the multimodal large language model, the global code assembly module 130 performs dynamic strategy selection to output target webpage code using different assembly paths.

[0112] According to a preferred embodiment, the global code assembly module 130 obtains a first input length corresponding to all local codes to be assembled. The global code assembly module 130 obtains a second input length corresponding to the boundary features corresponding to all local codes. The global code assembly module 130 obtains global assembly constraint information indicating spatial location and hierarchical structure, and determines its corresponding third input length. The global code assembly module 130 determines the required context input length based on the first input length, the second input length, and the third input length.

[0113] Preferably, the global code assembly module 130 sums the first input length, the second input length, and the third input length to obtain the required context input length. Through explicit addition, the global code assembly module 130 can predict in advance whether the amount of input data will exceed the capacity of the multimodal large-scale language model. This prevents generation interruptions or task termination due to token overflow during long code generation or global assembly.

[0114] Preferably, the global code assembly module 130 calculates the sum of the first input length, the second input length, and the third input length, and multiplies this sum by a preset redundancy scaling factor to obtain the final required context input length. The redundancy scaling factor takes into account the dynamic token reservation space required by the multimodal large language model when performing complex inference tasks, ensuring that the assembly process maintains high stability within the limited context window.

[0115] Preferably, the global code assembly module 130 assigns corresponding weight coefficients to the first input length, the second input length, and the third input length, respectively. The required context input length is obtained by multiplying each input length by its corresponding weight coefficient and then summing the results. By setting differentiated weight coefficients, the contribution of code text, structured boundary data, and pre-defined template instructions to context occupancy under different word segmentation algorithms can be accurately evaluated. The global code assembly module 130 transforms the evaluation of whether a large multimodal language model can handle input data into an objective standard that can be measured by specific numerical values ​​through explicit addition of information lengths. This effectively prevents generation interruptions during operation and avoids the risk of data exceeding the capacity of the large multimodal language model.

[0116] Preferably, before proceeding to global assembly, the global code assembly module 130 first estimates the required context input length for the current page to perform global assembly of a multimodal large language model. This input mainly includes three parts: local codes corresponding to all image patches, boundary features and location descriptions associated with each image patch, and standardized marker templates and global constraint information for the assembly stage. Assume the page contains N image patches, and the average length of the local code for each image patch is... The average length of the boundary features and location descriptions attached to each image patch is [missing information]. The fixed input length during the assembly stage is The required context input length can be approximately expressed as: .

[0117] In the above formula, It can be obtained from the statistics of the local code of the existing page, or it can be obtained directly by generating the local code of each image block in the local code generation module 120; It consists of the coordinates, dimensions, and necessary identifiers of the boundary features, with a fixed format and a basically stable length; The corresponding standardized markup templates, system instructions, and assembly constraint information are fixed inputs in the local code generation module 120, and their lengths can be directly calculated. Therefore, the parameters here are not set by human experience, but are derived from existing code length statistics and fixed input structures, making them reproducible.

[0118] Global code assembly module 130 simultaneously reads the maximum context window that the current model can support. Because the assembly stage requires reserving output space and inference redundancy, the maximum context window is not used directly. Instead of setting a threshold, a safety limit is set. For example, take: .

[0119] In the above formula, This is the scaling factor for configuring context safety margin. It aims to balance anti-truncation security with model computational efficiency. The value range is preferably limited to between 0.7 and 0.9; in a specific numerical example, for example, it can be taken as... It is 0.8.

[0120] when When the global code assembly module 130 determines that the current model cannot stably complete the global assembly based on a multimodal large language model, it directly adopts coordinate constraint assembly based on boundary features. At that time, the global code assembly module 130 determines that both assembly paths can be executed, and enters the second stage of parallel generation and automatic optimization.

[0121] Through this mechanism, the question of "whether to allow multimodal large language models to participate in global assembly" is transformed into a computable and implementable judgment process, thereby ensuring that the selection of dynamic strategies has a clear basis.

[0122] When the required context input length is greater than the context capacity threshold, the global code assembly module 130 performs coordinate constraint assembly based on boundary features to splice the local code. When the required context input length is not greater than the context capacity threshold, it performs coordinate constraint assembly and semantic adaptive assembly to generate candidate code, and calls the verification operator to evaluate the candidate code for automatic selection, thereby outputting the target webpage code.

[0123] Regarding the processing of the global code assembly module 130, this invention provides two different strategies for assembling the local code of all image blocks while maintaining the overall layout of the web page design: coordinate constraint assembly and semantic adaptive assembly. The dynamic strategy selection mechanism of this invention is not a simple empirical switch between the two assembly paths, but a control mechanism designed to address the applicability boundaries of the two assembly paths under different model conditions. Coordinate constraint assembly has the advantages of high positional accuracy, stable implementation, and low requirements for the context capacity threshold of the multimodal large language model. It achieves structural restoration by directly mapping boundary features to the spatial position parameters of the local code. Semantic adaptive assembly is more flexible, with advantages of more natural transitions between local code blocks and more coherent overall page semantics. However, it requires a multimodal large language model to support a longer context to effectively process the local code of all image blocks; otherwise, it is prone to failure in the final assembly step. Therefore, this invention does not simply address the trade-off between superior and inferior page rendering effects, but rather the differences in executability and code structure restoration logic between different assembly paths. Based on these differences, the global code assembly module 130 retains both strategies where applicable and triggers a dynamic strategy selection mechanism to output the target webpage code using different assembly paths.

[0124] Strategy 1: Coordinate Constraint Assembly. This invention utilizes the coordinates from the boundary features obtained during the image segmentation stage for coordinate constraint assembly. The local code of each image block is encapsulated in spatial positioning parameters, and its position and size are set according to the boundary features.

[0125] Specifically, when performing coordinate constraint assembly based on boundary features, the global code assembly module 130 extracts the top, bottom, left, and right coordinate values ​​from the boundary features corresponding to each image block. The global code assembly module 130 configures the root node of the target webpage code with a relative positioning attribute. Simultaneously, the global code assembly module 130 configures the position parameters of each local code segment with an absolute positioning style, directly injecting the extracted coordinate values ​​into the absolute positioning attributes of each outer frame tag. Finally, the global code assembly module 130 converts the extracted coordinate values ​​into Cascading Style Sheets (CSS) pixel units and directly injects them into the absolute positioning attributes of each parent container node, completing the physical position anchoring. Before performing the splicing, the global code assembly module 130 cleanses the code structure, removing redundant HTML frame tags carried by each local code segment, and only extracting the core DOM nodes inside the tags for assembly. Furthermore, before performing feature comparison, the global code assembly module 130 performs image resource mapping based on boundary features, automatically replacing image placeholders in the local code with the storage addresses of the actual image materials corresponding to the original webpage design.

[0126] Strategy 2: Semantic Adaptive Assembly. For large multimodal language models, constraints can be applied by adjusting constraint rules to generate more accurate and high-quality web page code. This invention designs constraint rules to guide code assembly based on the following principles: (1) Maintain layout: Based on the original web page design and the boundary features of each image block, merge the provided local code to ensure that the position matches the position in the design; (2) Layout-driven chain assembly: Analyze layout information using all boundary features, assemble code, compare the generated web page with the web page design to ensure consistency of content, layout and style, and make corrections and refinements as necessary, and finally determine the complete target web page code.

[0127] According to a preferred embodiment, the global code assembly module 130 calls the verification operator to evaluate candidate code in the following ways: calling the rendering engine to perform page rendering on the candidate code and obtaining the corresponding rendered page; comparing the features of the rendered page with the original web page design, and calculating the page pixel difference measure and semantic similarity measure between the rendered page and the original web page design respectively; and calculating the verification operator based on the page pixel difference measure and semantic similarity measure.

[0128] The global code assembly module 130 integrates the difference features of page pixels for scoring. Simultaneously, it also incorporates semantic similarity from high-dimensional vision. This avoids relying solely on subjective evaluation using a large multimodal language model, resolving issues such as instability and inaccuracy that may result from subjective evaluation.

[0129] The dynamic strategy executed by the global code assembly module 130 is configured to output the optimal target webpage code when multiple assembly strategies are available. When the multimodal large language model has a sufficiently long context capacity threshold to handle all local code, two assembly strategies are executed in parallel; when the context capacity of the multimodal large language model is limited (e.g., using a lightweight model such as DeepSeek-VL2), only coordinate constraint assembly is performed.

[0130] To evaluate candidate code generated by a multimodal large language model, the global code assembly module 130 invokes a verification operator to quantify the similarity between the rendered page and the webpage design. This evaluation mechanism is reference-free, meaning it does not rely on the target's source code but verifies solely by comparing the image features of the rendered page with those of the original webpage design. To avoid output instability in image similarity verification based on the Large Language Model-as-a-Judge paradigm (MLLM-as-a-Judge), the verification operator employs an automatically calculated metric. To overcome the neglect of high-dimensional semantic information by single pixel-level metrics (such as mean squared error, MAE, or normalized Waterman distance, NEMD), the verification operator's computational model integrates the Page Pixel Difference Measure (MAE) and the Semantic Similarity Measure (CLIP). The formula for calculating the verification operator (verification score) is as follows: .

[0131] In the above formula, the Page Pixel Difference Measure (MAE) and the Semantic Similarity Measure (CLIP) are both configured with equal weight coefficients (0.5).

[0132] Before calculating the pixel difference measure for each page, the global code assembly module 130 extracts the resolution parameters of the rendered page and the original web page design. If the resolutions are determined to be inconsistent, the global code assembly module 130 uses a bilinear interpolation algorithm to scale the rendered page to the same pixel size as the original web page design. Before performing feature comparison, the global code assembly module 130 performs image resource mapping based on boundary features, replacing the image placeholders generated by the prompt words in the rendered page with the corresponding real local image blocks in the original web page design. Subsequently, the global code assembly module 130 calculates the sum of the absolute errors of the corresponding color channels of the two images pixel by pixel in the same coordinate system. The global code assembly module 130 calculates the mean of the absolute error of the total number of pixels in the entire image and outputs an objective measure of the page pixel difference.

[0133] When the first stage determines that the current model has sufficient contextual capabilities, the global code assembly module 130 simultaneously generates two sets of candidate webpage code: one set is the coordinate constraint assembly result based on boundary features, and the other set is the semantically adaptive assembly result. Subsequently, the global code assembly module 130 calls the background automated rendering engine to render the two sets of candidate code respectively. The background automated rendering engine preferably adopts a headless browser implementation based on Playwright.

[0134] The global code assembly module 130 obtains the corresponding rendered page and compares it with the original webpage design. This comparison process is uniformly completed by the global code assembly module 130. The global code assembly module 130 simultaneously calculates the page pixel difference measure (denoted by MAE) and the semantic similarity measure (CLIP), and then weights the two to obtain the verification operator. .

[0135] In the above formula, To verify the operator, the weighting coefficients and It can be configured as a fixed weight or an adaptive weight based on page content features.

[0136] Preferably or alternatively, the weighting coefficient and A preset balanced weight (e.g., all weights set to 0.5) is adopted. Preferably or alternatively, the global code assembly module 130 identifies the proportion of component categories in the web page design and dynamically adjusts the weights: for pages where the text area proportion is higher than a preset proportion threshold, the weight coefficient is adjusted accordingly. The value is greater than the weight coefficient. The value of the weight coefficient is determined for pages where the proportion of an image region exceeds a preset threshold. The value is greater than the weight coefficient. The value of is determined by the global code assembly module 130, which ultimately selects the set of results with the higher verification operator score as the output, forming a closed-loop process of "candidate code generation - automatic rendering - image feature comparison - quantization scoring - automatic optimization".

[0137] Regarding the performance evaluation data configuration for this device: During the performance verification phase, the webpage code generation test uses a test dataset containing highly complex layout features, including the Design2Code-HARD benchmark and the more challenging CC-HARD benchmark. The Design2Code-HARD benchmark contains 80 long-sequence webpage samples. For the CC-HARD benchmark, the specific verification parameters for the test process are configured as follows: (1) The maximum depth parameter of the relevant comparison algorithm (such as the DCGen algorithm) is configured to be 1; (2) Because this device adopts different local code combination strategies, the processing branch that performs coordinate constraint assembly and the processing branch that performs semantic adaptive assembly are used as independent basic verification benchmarks for comparative experiments.

[0138] In addition to using the Design2Code-HARD benchmark, which contains 80 long sequence samples, this embodiment also manually collected samples from the Common Crawl dataset and generated image-code pairing data to construct a more challenging CC-HARD benchmark (containing 128 complex page samples). This CC-HARD benchmark has higher layout complexity, with an average Document Object Model (DOM) tree depth of 16 levels and a denser distribution of front-end tag types and quantities (an average of 27 unique tags), placing extremely high demands on the model's spatial resolution capabilities.

[0139] The specific configuration of this invention is shown below.

[0140] (1) Experimental parameters and operating environment configuration To improve the reproducibility of the verification results, the temperature parameter of the application programming interface (API) was configured to 0, and the random seed of the pre-built packages (Python, torch, and numpy) was fixed at 2026. All verification experiments were performed on a server configured with a GPU computing array (e.g., four NVIDIA A800 80GB GPUs).

[0141] During the image segmentation stage, the optical character recognition unit 112 acquires the boundary features of all text regions and merges adjacent boundary features with a spacing within a preset tolerance range (20 pixels) in the horizontal or vertical directions to establish complete text paragraphs and prevent the orthogonal layout isolation band 140 from crossing adjacent text. Furthermore, the specific parameter configurations are as follows: the grid sampling interval is set to 5 pixels, the minimum spacing threshold for the effective orthogonal layout isolation band 140 is set to 50 pixels, the number of edge pixels to be ignored in the design drawing is configured to 10, the maximum depth limit for recursive scanning is 3, and the minimum area threshold for image block merging is configured to 300×300 pixels. This parameter combination ensures the balance of the final image segmentation granularity to achieve a stable visual reconstruction effect.

[0142] (2) Validation of webpage generation method Regarding the technical effectiveness of the method provided by this invention: According to performance test data, when using GPT-4o as a multimodal large-scale language model, compared with the comparison algorithm, this invention achieves performance improvements of 17.65%, 1.27%, 3.8%, and 37.41% in TreeBLEU, semantic similarity metric (CLIP), validation operator score, and page pixel difference metric (MAE) on the Design2Code-HARD benchmark, respectively; and 60%, 2.53%, 2.56%, and 43.23% in the CC-HARD benchmark, respectively. When using the Gemini model, performance improvements of 2.38%, 5.19%, and 40.09% in CLIP, validation operator score, and MAE on the Design2Code-HARD benchmark, respectively; and 30%, 2.56%, 2.63%, and 43.03% in TreeBLEU, CLIP, validation operator score, and MAE on the CC-HARD benchmark, respectively. When using the DeepSeek-VL2 model, on the Design2Code-HARD benchmark, performance improvements of 58.33%, 12.5%, and 25.31% were achieved in TreeBLEU, validation operator score, and MAE, respectively (with a 4.94% reduction in CLIP). On the CC-HARD benchmark, performance improvements of 66.67%, 12.5%, and 38.53% were achieved in TreeBLEU, validation operator score, and MAE, respectively (with a 1.33% reduction in CLIP). Test data demonstrate that the method of this invention can achieve technical gains in web page code synthesis tasks across various multimodal large-scale language model architectures and exhibits robustness in lightweight models with limited context capacity.

[0143] (3) Performance analysis of high-complexity test benchmarks Performance verification was conducted on high-complexity benchmarks. Test data shows that for the selected models (DeepSeek-VL2, Gemini, and GPT-4o), all comparison methods exhibited a downward trend in various metrics on the CC-HARD benchmark, primarily concentrated in TreeBLEU and MAE. Taking GPT-4o as an example, compared to the Design2Code-HARD benchmark, the average TreeBLEU, CLIP, and validation operator scores for all tested methods on the CC-HARD benchmark decreased by 48.97%, 8.03%, and 1.24%, respectively, while the MAE score increased by 9.12%. This data characteristic can be attributed to the higher layout complexity of the CC-HARD benchmark, which has a deeper HTML Document Object Model (DOM) tree hierarchy and a denser distribution of front-end tag types and quantities.

[0144] To further analyze the restoration mechanism of the method of the present invention under complex layouts and to provide micro-level technical support for the fluctuation of the above-mentioned overall indicators, this embodiment introduces a fine-grained visual scoring decomposition system.

[0145] Figure 5 This is a schematic diagram illustrating the visual scoring decomposition and comparison when performing webpage transcoding tasks on the CC-HARD dataset according to the present invention. Figure 5 The quantitative demonstration shows the technical gains achieved by the image-block-based web page code generation method (LaTCoder) proposed in this invention in terms of front-end code visual fidelity compared to existing baseline algorithms. Figure 5 The horizontal axis represents the underlying multimodal large-scale language model called when performing webpage transcoding tasks, specifically including weaker models with limited context capacity thresholds (DeepSeek-VL2) and higher-performance mainstream models (Gemini, GPT-4o). The vertical axis represents the percentage score for each validation dimension; higher values ​​indicate a higher structural fidelity between the generated webpage code rendering result and the original webpage design. Figure 5 In the diagram, the yellow bars represent the webpage code generation method based on image segmentation (LaTCoder) of this invention, the sky blue bars represent the direct prompting method that relies on the native capabilities of a multimodal large language model (Direct Prompting), the mint green bars represent text enhancement (Text-Augmented), the yellow-green bars represent self-revision (Self-Revision), and the light grass green bars represent the contrast algorithm (DCGen).

[0146] Figure 5 Five sub-indicators constituting the accuracy of front-end visual reconstruction are presented, used to evaluate the comparison results of features between the rendered page and the original web page design. Specifically, Text Color Match is used to examine the consistency of color and background style attributes in the content details code output by the local code generation module. Position Match measures the accuracy of the two-dimensional spatial topological coordinates of each component after code rendering; its score reflects the technical effect of the present invention in extracting orthogonal layout isolation zones through the image segmentation module and using boundary features to achieve precise spatial anchoring. Semantic Similarity Measure (CLIP), as one of the core calculation parameters of the verification operator, is used to capture the similarity at the high-dimensional semantic level to measure the overall visual coherence of the page after global code assembly. Text Content Match quantifies the matching degree between the character content in the generated HTML code and the web page design, verifying the advantages of this solution in avoiding text region segmentation to ensure the integrity of the text structure. Block Match examines the anti-truncation capability and local reconstruction accuracy of each independent processing unit in the assembly stage after the web page design is divided into multiple image blocks.

[0147] like Figure 5 As shown, under the constrained model architecture of the weaker model (DeepSeek-VL2), the image-block-based web page code generation method (LaTCoder) exhibits significant technical gains in all five dimensions. By triggering coordinate constraint assembly through a dynamic strategy selection mechanism, this scheme overcomes the global assembly collapse problem caused by the overflow of long sequence code context capacity in multimodal large language models. Furthermore, under the Gemini and GPT-4o architectures, even with enhanced baseline model capabilities, the method of this invention still maintains its advantage in text color matching and position matching metrics. This confirms that the layout-driven chain generation mechanism of this invention can compensate for the vulnerability of multimodal large language models in numerical reasoning and spatial location modeling, and eliminate the structural fragmentation and layout distortion defects caused by direct prompting methods.

[0148] (4) Performance impact mechanism of assembly strategy Regarding the performance comparison mechanism of global code assembly strategies, when using Gemini and GPT-4o as the base models, on both test benchmarks, the quantitative differences in TreeBLEU, CLIP, and verification operator scores between the processing branch performing coordinate constraint assembly and the processing branch performing semantic adaptive assembly are less than the preset evaluation threshold. This data indicates that different assembly paths have a limited impact on the overall code structure and component content similarity of the target webpage code. However, the processing branch performing coordinate constraint assembly outperforms the processing branch performing semantic adaptive assembly in terms of MAE. The technical reason is that the coordinate constraint assembly mechanism strictly anchors the position based on boundary features, while the semantic adaptive assembly mechanism may cause local spatial coordinate offset; coordinate constraint assembly can improve the alignment accuracy of image blocks with similar content features, thereby reducing the page pixel difference between the original webpage design and the original webpage design. On the other hand, the test verifies that the processing branch performing semantic adaptive assembly performs better in terms of global layout smoothness. Therefore, in this embodiment, the above two assembly strategies are deployed in parallel in the global code assembly module 130, and the verification operator is called to perform filtering and dynamic evaluation on the candidate code to ensure that the best target webpage code is finally output.

[0149] The most direct technical effect of this invention is reflected in its corrective effect on context-constrained models. Taking DeepSeek-VL2 as an example, if the global code assembly module 130 directly uses a multimodal large language model for the final step of single semantic adaptive assembly, the assembly failure rate is approximately 85% because it cannot simultaneously input all image block codes and layout information. However, after introducing the method of this invention, the global code assembly module 130 will automatically determine in the first stage that it is not suitable for performing semantic adaptive assembly, and instead use coordinate constraint assembly, making the assembly success rate close to 100%. This shows that the method does not bring about a general performance improvement, but rather an improvement from "the final step cannot be completed" to "being able to stably complete the assembly".

[0150] On models with strong contextual capabilities, the global code assembly module 130 does not degenerate into fixed coordinate-constrained assembly. Instead, it retains semantically adaptive assembly as a candidate and automatically selects the better result through validation operators. Therefore, this method simultaneously addresses two scenarios: ensuring process execution on weak models and further improving visual consistency and overall assembly quality on strong models. The generation experiment results of different assembly methods on the CC-HARD dataset are shown in the comparison table of the effects of different assembly strategies.

[0151] Table 1: Comparison of Technical Effects of Different Assembly Strategies

[0152] The above comparison shows that this method solves the pain point of "the last step cannot be completed" in weak model scenarios by using the global code assembly module 130, and obtains better overall results in strong model scenarios through automatic optimization.

[0153] (5) Subjective visual assessment based on human preferences.

[0154] Pairwise human preference comparisons were performed on the method of this invention and a baseline comparison method. This evaluation used a multimodal large language model (such as GPT-4o) as a foundation and was performed on a test benchmark (CC-HARD) containing highly complex layout features. When presenting the out-of-order rendered page and the original webpage design, testers used a majority voting strategy based on visual fidelity and structural fidelity to determine preference results and eliminate individual subjective evaluation bias.

[0155] Figure 6 This is a pairwise comparison of quantitative data between the method of this invention (LaTCoder) and four baseline methods in the above-mentioned subjective visual evaluation of human preferences. Figure 6 The vertical axis lists the four existing baseline methods used for comparison: Direct Prompting, Text-Augmented, Self-Revision, and DCGen. The horizontal axis represents the percentage of evaluation results. The sky-blue area (LaTCoderWin) represents the percentage of samples where the method in this embodiment achieves better visual preference; the mint-green area (Tie) represents the percentage of samples where there is no significant difference in visual effect between the two methods; and the light grass-yellow area (LaTCoder Lose) represents the percentage of samples where the baseline method achieves a better evaluation.

[0156] Combination Figure 6 The specific quantitative labels and comparative data for each evaluation dimension are presented in full as follows: In the comparison with the Direct Prompting method, the method of this invention has a win rate of 61.7%, a tie of 10.9%, and a disadvantage of 27.3%; in the comparison with Text-Augmented, the method of this invention has a win rate of 67.2%, a tie of 14.8%, and a disadvantage of 18.0%; in the comparison with Self-Revision, the method of this invention has a win rate of 68.8%, a tie of 10.9%, and a disadvantage of 20.3%; in the comparison with the DCGen algorithm, the method of this invention achieved the highest win rate of 79.7%, a tie of 7.0%, and a disadvantage of 13.3%.

[0157] The detailed data distribution described above objectively verifies the significant technical advantages of the method of this invention in front-end visual fidelity. The precise distribution of tie and latitude loss data further demonstrates the high robustness of the technical solution of this invention: even in the most basic direct prompting comparison group, the extreme case where the method of this invention is at a disadvantage is strictly compressed to below 27.3%. This confirms from the opposite perspective of human perception that the mechanism of using an image segmentation module to divide orthogonal layout isolation bands and applying coordinate constraint assembly in the later stage can greatly suppress the spatial numerical reasoning imbalance that is prone to occur when multimodal large language models process complex front-end HTML structures, and overcome the technical defect of traditional generation methods that are prone to layout fragmentation.

[0158] (6) Analysis of positive technical features of target webpage code generation.

[0159] Figure 7 This diagram illustrates a comparison of the rendering results of the method (LaTCoder) of this invention with various baseline methods for generating target webpage code. Figure 7 As shown in (a), the original web page design includes a top navigation bar, a centered image area, and sidebar components located on both sides of the main content, presenting a three-column side-by-side layout structure.

[0160] Comparison of the rendered results shows that the baseline method caused changes to the layout features when processing the webpage design. Specifically, such as... Figure 7 (c) Direct Prompting and Figure 7 As shown in (d) Self-Revision, the generated rendering result does not maintain the parallel layout structure of the original webpage design. The sidebar components and the main content area are vertically stacked, resulting in layout fragmentation. Figure 7 As shown in (e), the contrast algorithm (DCGen) produces a rendering result with structural deviation and loses the document object model (DOM) node corresponding to the centered image region.

[0161] like Figure 7 As shown in (b), the rendering result generated by the method of the present invention (LaTCoder) retains the three-column layout structure of the original webpage design. The image placeholders in the main content area (e.g., 600×300 pixels) and the sidebar components on both sides correspond spatially to the coordinates of the original webpage design, and the block components at the bottom of the sidebar remain horizontally arranged. It should be noted that... Figure 7 The image placeholders displayed in the rendering result of (b) are only used to visualize the physical boundary information of each image block. In the final generated production-grade web page code, the placeholders have been restored to the original image visual effect according to the image resource mapping mechanism, thereby maintaining the consistency of the overall visual appearance.

[0162] The above structural feature comparison shows that the method of the present invention extracts orthogonal layout isolation bands through the image segmentation module and performs layout-driven chain generation and coordinate constraint assembly. This mechanism reduces the probability of spatial coordinate offset when multimodal large language models process the global view, overcomes the layout fragmentation problem, and makes the structural features of the output target webpage code match the original webpage design.

[0163] like Figure 7 As shown, a comparison of the rendering results of the target webpage code generated using the method of this invention and the baseline method is presented. Test cases show that the code generated by directly executing the global code and the code that introduces self-checking and revision are similar in structural features, and both have layout fragmentation problems; while the comparison algorithm using local natural language description causes structural deviation and performance degradation when processing the global view. In contrast, the method of this invention extracts boundary features through the image segmentation module 110 and performs layout-driven chain generation, so that the output target webpage code achieves a significant improvement in similarity in both the overall front-end structure system and fine-grained view performance. The final output target webpage code includes code encapsulated by the global code assembly module 130.<!DOCTYPE html> This includes declarations and all local code containers embedded within the `<script>` tag. These local code containers establish hierarchical relationships through absolute positioning styles or flexbox semantic associations, ensuring that the final generated front-end code maintains the visual topology of the original web page design while possessing structured semantic characteristics that conform to modern web development standards.

[0164] (7) Analysis of abnormal boundaries caused by inherent limitations of multimodal large language models.

[0165] During the verification phase, two typical error boundaries were identified due to the limitations of the underlying computational capabilities of the multimodal large language model: (1) Spatial coordinate offset during the local code generation phase: In certain scenarios, the multimodal large language model may incorrectly anchor the UI elements of local image blocks (e.g., abnormally centering or bottoming within the parent container), exposing the computational reasoning defects of the multimodal large language model in parsing micro-topological structures and extracting precise coordinate system transformation values. Although the method of the present invention greatly suppresses this problem at the macro page level through the orthogonal layout isolation band 140, it is still limited by the visual parsing accuracy of the large model when generating fine-grained local code; (2) Model computation truncation defects during the code assembly phase: During long sequence context processing, the multimodal large language model (e.g., the Gemini model) may occasionally skip the processing action of specific image block code when performing code splicing, resulting in some UI blocks being missing in the final rendering output. In this regard, the dynamic strategy selection mechanism adopted by the global code assembly module 130 of the present invention can effectively reduce the risk of such truncation caused by the overflow of the context capacity of the large model.

[0166] (8) Comparison and summary of webpage transcoding technology mechanisms based on neural language models.

[0167] Early UI transcoding technologies were limited by underlying computing power bottlenecks, mostly employing recurrent neural networks (RNNs) or convolutional neural network (CNN) structures to generate domain-specific languages ​​from simple design images. With the development of large-scale language modeling technologies, various models for general code logic synthesis (such as InCoder, StarCoder, Code Llama, WizardCoder, Qwen-Coder, and DeepSeek-Coder) have improved the overall coding efficiency of software engineering. However, unlike general-purpose mind trees (ToT) or text-based reasoning and verification paradigms, this invention's "layout-driven chain generation" mechanism overcomes the technical challenges of layout distortion and computational resource overload in multimodal large-scale language models when processing extremely long and complex real-world front-end code by pre-applying image block features and injecting absolute spatial boundary feature constraints during the code synthesis stage.

[0168] Table 2: Statistical Comparison of Design2Code HARD and CC-HARD

[0169] Table 3: Overall performance comparison of LaTCoder and different backbone MLLM baseline models on two datasets.

[0170]

[0171] The following is supplementary information regarding the evaluation parameters for the performance verification data: (1) For specific multimodal large language models with limited context capacity (such as DeepSeek-VL2), only two comparative test paths are executed in the test configuration, and coordinate constraint assembly is used in the global code assembly stage to avoid assembly failure caused by context overflow.

[0172] (2) The method of the present invention sets up three comparison branches in performance verification, which correspond to different code assembly paths: a single semantic adaptive assembly branch, a single coordinate constraint assembly branch, and a complete process branch that executes the above two assembly paths in parallel and calls the verification operator to perform dynamic strategy selection.

[0173] (3) The performance change magnitude parameter (Δ) is configured as follows: it represents the improvement or attenuation ratio of the best quantization performance parameter obtained by the method of this embodiment in the above three control branches relative to the best quantization performance parameter obtained by all baseline methods.

[0174] It should be noted that the specific embodiments described above are exemplary. Those skilled in the art can devise various solutions inspired by the disclosure of this invention, and these solutions all fall within the scope of this invention and its protection. Those skilled in the art should understand that this specification and its accompanying drawings are illustrative and not intended to limit the scope of the claims. The scope of protection of this invention is defined by the claims and their equivalents. This specification contains multiple inventive concepts; terms such as "preferredly," "according to a preferred embodiment," or "optionally" indicate that the corresponding paragraph discloses an independent concept. The applicant reserves the right to file divisional applications based on each inventive concept. < / button>

Claims

1. A webpage code generation device based on image segmentation, characterized in that, Includes a processor (100), said processor (100) comprising: The image segmentation module (110) is used to perform feature scanning on the received web page design to extract the orthogonal layout isolation band (140) in the web page design and segment the web page design into multiple image blocks along the orthogonal layout isolation band (140) to output each image block and its corresponding boundary features. The local code generation module (120) is used to call a multimodal large language model to perform layout-driven chain generation and generate local codes corresponding to each image block within a preset standardized marker template; The global code assembly module (130) is used to receive the boundary features and the local code, and to perform dynamic strategy selection based on the adaptability relationship between the required context input length corresponding to the local code and the context capacity threshold of the multimodal large language model, so as to output the target web page code using different assembly paths.

2. The device according to claim 1, characterized in that, The global code assembly module (130) performs dynamic strategy selection in the following ways: If the required context input length is greater than the context capacity threshold, coordinate constraint assembly based on the boundary features is performed to splice the local code. If the required context input length is not greater than the context capacity threshold, the coordinate constraint assembly and semantic adaptive assembly are executed respectively to generate candidate code, and the verification operator is called to evaluate the candidate code and perform automatic optimization to output the target webpage code.

3. The device according to claim 1 or 2, characterized in that, Before outputting each of the image blocks, the image segmentation module (110) is further configured to: Identify initial image blocks with an area smaller than a preset area threshold; The initial image block is merged with spatially adjacent image blocks, and the boundary features of the merged region are redefined.

4. The device according to any one of claims 1 to 3, characterized in that, The image segmentation module (110) uses an adaptive segmentation threshold to extract the orthogonal layout isolation band (140), wherein, Obtain the spacing distribution characteristics that represent the current webpage design or historical design data; The segmentation threshold used to determine whether a candidate isolation band in the web page design is a valid orthogonal layout isolation band (140) is set as a dynamic parameter determined based on the spacing distribution characteristics, so that the segmentation threshold is adaptively adjusted according to the layout density of the web page design.

5. The device according to any one of claims 1 to 4, characterized in that, The local code generation module (120) is configured to perform the layout-driven chained generation through the following steps: The principle of prioritizing the output of spatial layout relationship structure codes is applied to establish the layout elements corresponding to the image blocks and the hierarchical relationships between the elements. The content detail code of the image block is generated based on the layout elements and the hierarchical relationships between the elements. The multimodal large language model is invoked to perform consistency verification and repair on the generated content detail code.

6. The device according to any one of claims 1 to 5, characterized in that, The local code generation module (120) calls the multimodal large language model to perform consistency verification and repair on the generated content detail code in the following ways: The text, color, and / or background style in the content detail code are compared with the visual features of the image block, and the content detail code is refined based on the comparison results.

7. The device according to any one of claims 1 to 6, characterized in that, The global code assembly module (130) is configured to calculate the required context input length in the following manner: Obtain the first input length corresponding to all the local codes to be assembled; Obtain the second input length corresponding to the boundary feature corresponding to all the local codes; Obtain global assembly constraint information that indicates spatial location and hierarchical structure, and determine its corresponding third input length; The required context input length is determined based on the first input length, the second input length, and the third input length.

8. The device according to any one of claims 1 to 7, characterized in that, The global code assembly module (130) calls the verification operator to evaluate the candidate code in the following ways: The rendering engine is invoked to render the candidate code and obtain the corresponding rendered page. The rendered page is compared with the original web page design by feature comparison, and the page pixel difference measure and semantic similarity measure between the rendered page and the original web page design are calculated respectively. The verification operator is calculated based on the page pixel difference metric and the semantic similarity metric.

9. A webpage code generation method based on image segmentation, characterized in that, The method includes: The received web page design is subjected to feature scanning to extract the orthogonal layout isolation band (140) in the web page design, and the web page design is divided into multiple image blocks along the orthogonal layout isolation band (140) to output each image block and its corresponding boundary features; A multimodal large language model is invoked to perform layout-driven chained generation, generating local code corresponding to each image patch within a pre-set standardized tag template; The boundary features and the local code are received, and dynamic strategy selection is performed based on the adaptability relationship between the required context input length corresponding to the local code and the context capacity threshold of the multimodal large language model, so as to output the target web page code using different assembly paths.

10. The method according to claim 9, characterized in that, The methods for performing dynamic strategy selection include: If the required context input length is greater than the context capacity threshold, coordinate constraint assembly based on the boundary features is performed to splice the local code. If the required context input length is not greater than the context capacity threshold, the coordinate constraint assembly and semantic adaptive assembly are executed respectively to generate candidate code, and the verification operator is called to evaluate the candidate code and perform automatic optimization to output the target webpage code.

Images