Method for automatically converting UI design diagram into front-end code based on multi-modal deep learning

Abstract

The application discloses a method for automatically converting a UI design diagram into front-end code based on multi-modal deep learning, which comprises the following steps: obtaining design image pixel matrix data; performing UI component detection and pixel-level segmentation through an improved Mask R-CNN framework to obtain the boundary box and segmentation mask of each component; solving the relative depth value of each component by using a depth estimation module, and constructing a hierarchical tree in combination with the overlapping relationship of the boundary box; generating structured description data by fusing the pixel-level visual attribute and the semantic label output by a visual-language model; mapping the structured data into basic page source code according to the syntax rules of a selected front-end framework; capturing the triggering operation of a user on an unbound event element in an interactive preview environment, querying the pre-constructed UI interaction knowledge graph to infer the intention and generating target page code; and packaging each page code and resource file into a front-end project. The application realizes the full-process automation of UI design, improves the UI restoration degree and shortens the development cycle.

Description

Technical Field

[0001] This invention relates to the intersection of artificial intelligence and software interface development, and in particular to a method for automatically converting UI design drafts into front-end code based on multimodal deep learning. Background Technology

[0002] In the traditional process of converting design drafts to UI, developers typically manually code based on the design drafts. Developers need to carefully examine the design drafts, identify elements, layouts, styles, and other information, then select a suitable development framework, write the corresponding UI code, and manually collect and associate relevant images, icons, and other resources. Some existing technologies attempt to achieve initial conversion through traditional image parsing algorithms and preset rule libraries, but still require manual or third-party plugin involvement in the parsing process. After generating the initial code, secondary processing such as online editing and data reorganization is required.

[0003] Technical problems with existing technologies: (2) Manual coding takes a lot of time, especially for complex design drawings, which often require a long development cycle; existing automation solutions are cumbersome, with many manual intervention steps, and the overall efficiency is limited. (2) Professional developers need to invest a lot of energy in coding, debugging and secondary processing of code, which increases the manpower cost of the project. (3) Traditional image analysis algorithms can only restore the static layer, cannot accurately identify the component hierarchy, and the restoration effect is greatly affected by human experience, with an average restoration degree of only 50%-70%. (4) Traditional prototyping tools only support static demonstrations or predefined interactions. User operations without business logic cannot trigger page flow, and the target page needs to be developed manually.

[0004] Therefore, there is an urgent need for a technical solution that can overcome the problems of low efficiency and high labor costs in the traditional process of converting design drafts to UI, while also addressing the industry pain points of difficulty in restoring component hierarchy and the inability to intelligently interpret interactive intents, in order to automatically convert UI design drafts into front-end code. Summary of the Invention

[0005] The purpose of this invention is to overcome the problems of low efficiency and high labor costs in the traditional process of converting design drafts into UI, and to provide a method for automatically converting UI design drafts into front-end code based on multimodal deep learning. This method enables intelligent parsing of user intent and real-time generation of target pages, thereby improving development efficiency, ensuring UI fidelity, reducing labor costs and error probability, and expanding the ability to generate dynamic interactions.

[0006] The technical solution adopted in this invention is:

[0007] A method for automatically converting UI design drafts into front-end code based on multimodal deep learning, executed by a computing device, includes the following steps:

[0008] Obtain the pixel matrix data of the user interface design draft and preprocess it to obtain standardized input image data; the pixel matrix data is a design draft in raster image format or a parsed design draft page image;

[0009] The input image data is fed into a UI component detection framework based on an improved Mask R-CNN. The UI component detection framework uses a convolutional neural network to perform multi-layer convolution and pooling on the input image data to extract multi-scale feature maps of the design drawing. A region proposal network generates candidate bounding boxes for UI components based on the multi-scale feature maps. The ROIAlign layer performs bilinear interpolation to align the feature map regions corresponding to each candidate bounding box and outputs a fixed-scale aligned feature map. Based on the aligned feature map, bounding box regression and pixel-level mask prediction are performed in parallel to identify the bounding box positions of UI components (buttons, text boxes) and generate pixel-level masks to obtain the component regions, achieving high-precision separation of UI components from the background and adjacent UI components.

[0010] A depth estimation enhancement module is used to perform cross-layer fusion of multi-scale visual features of different convolutional layers of image blocks corresponding to each component region to solve the relative depth value of each UI component; the stacking order of each UI component along the Z-axis is determined based on the relative depth value, and the parent-child containment relationship is inferred by combining the overlapping relationship of the bounding box positions of each component in the XY plane, and a hierarchical tree data structure representing the nesting level and stacking order of components is constructed.

[0011] Pixel-level visual attributes are extracted from the segmentation mask region corresponding to each component node in the hierarchical tree. Simultaneously, the component region image patch is input into a visual-language embedding model to obtain the semantic embedding vector of the component region. This vector is then matched with a pre-defined set of UI component semantic labels to obtain the semantic component type label for the corresponding UI component. The hierarchical relationship, spatial layout coordinates, pixel-level visual attributes, and semantic component type labels are fused to generate full-page structured description data. This structured description data uses the hierarchical tree as its skeleton to record the type, bounding box position, style attributes, and parent-child relationships of each UI component. The pixel-level visual attributes include color distribution, pixel size ratio, font texture features, and position coordinates.

[0012] Based on the user-selected front-end framework type, the structured description data is mapped and converted into basic page source code according to the syntax rules and componentization specifications of the front-end framework, resulting in basic page source code files. The basic page source code includes the declaration code of each UI component, the layout positioning code based on bounding box coordinates and hierarchy tree, and the style description code based on style attributes.

[0013] The generated base page source code file is compiled and rendered into an interactive preview page. Event listeners monitor user actions on interactive elements in the preview page that are not bound to event handlers, capturing the location and visual semantic information of the triggered interactive elements. Candidate interaction intentions and target page types are obtained by querying a pre-built UI interaction knowledge graph through visual semantic information. Based on the context of the current page, structured description data of the target page is automatically constructed and mapped to generate the target page source code file according to the syntax rules of the front-end framework. The UI interaction knowledge graph stores the mapping relationship between typical UI component types and associated interaction mode types, page flow relationships, and target page templates in graph nodes.

[0014] The source code files of the base page, the source code files of the target page, and the resource files extracted from the design data and optimized by format conversion and compression are mapped and associated according to the directory structure of the front-end framework, and output as a runnable UI project file package.

[0015] Furthermore, when the obtained UI design drawing is in Figma prototype file format, the system parses the layer tree structure and vector node data of the Figma file, rasterizes the vector nodes into the pixel matrix data, and retains the semantic meaning of the layer name as auxiliary annotation information to participate in semantic tag matching.

[0016] Furthermore, the UI component category set includes at least one or more of the following: buttons, input boxes, drop-down selection boxes, text labels, image containers, list containers, card containers, and navigation bar areas; pixel-level mask prediction outputs the soft mask corresponding to each candidate area, and obtains the foreground segmentation mask by binarization through a preset threshold, so as to achieve high-precision separation of components and background and reduce pixel-level missegmentation of overlapping component areas.

[0017] Furthermore, the implementation of the depth estimation enhancement module includes:

[0018] The depth estimation enhancement module extracts first-scale feature maps and second-scale feature maps from at least two different layers of the convolutional neural network, and obtains a fused feature map by upsampling and channel-wise weighted fusion.

[0019] The fused feature map is input into the depth regression head to obtain the relative depth value corresponding to each component region. The stacking relationship of UI components from top to bottom is determined based on the depth value from large to small. That is, the component with the smaller depth value is determined to be located in the lower layer of the stack or is occluded, and the component with the larger depth value is determined to be located in the upper layer of the stack.

[0020] When the overlap area of ​​the bounding boxes of any two components in the XY plane is less than a set threshold and the relative depth of the first component is much smaller than the relative depth of the second component, the two corresponding nodes of the two components are marked as sibling nodes in the hierarchy tree. When the overlap area is greater than the set threshold and the relative depth of the first component is smaller than the relative depth of the second component, the node of the first component is set as a child node of the node of the second component. That is, when the bounding box of the first component is contained by the bounding box of the second component with a preset area ratio and the relative depth of the first component is smaller than the relative depth of the second component, the node of the first component is set as a child node of the node of the second component.

[0021] Furthermore, the visual-language embedding model is either a CLIP model or a visual-language pre-trained model based on contrastive learning. The matching process for semantic component type labels is as follows: the component region image patch is encoded into a visual embedding vector, and the cosine similarity between the visual embedding vector and the text embedding vector of each preset UI component semantic label is calculated; the label with the highest similarity is taken as the semantic component type label of the corresponding component region; and the pixels in the segmented mask region are clustered by color, and the main color value is extracted as the background color or fill color attribute and written into the structured description data.

[0022] Furthermore, the front-end framework type includes React or Vue. When using React, the generated component declaration code is a .tsx / .jsx file extending React.Component or a functional component, and the layout and positioning code is implemented based on Flexbox or Grid style descriptions. When using Vue, the generated component declaration code is a single-file component in .vue format, and the layout and positioning code is written to... <style>块的CSS描述中。

[0023] 进一步地，交互知识图谱的构建过程包括：收集典型UI设计系统中的交互模式样本，抽取三元组形式的关系事实（即组件类型—交互模式—目标页面 / 动作类型），以图数据库或邻接表结构存储。

[0024] 进一步地，候选交互意图的推理过程包括：以被触发交互元素的语义组件类型标签与周围上下文组件的布局关系作为查询键，在交互知识图谱中检索匹配度最高的交互模式三元组，当匹配度超过预设阈值时，将对应的目标页面类型作为推理结果输出。

[0025] 进一步地，当系统捕获用户连续多次对未绑定事件的交互元素的触发操作时，将连续操作的各交互元素的语义组件类型标签序列与对应页面上下文组织为多步操作上下文序列，在UI交互知识图谱中以子图匹配方式进行复合意图推理，生成对应于多级页面流转路径的多个目标页面结构化描述数据，并分别映射生成对应的多级目标页面源代码文件。

[0026] 进一步地，对资源文件中内嵌的图片与图标进行有损 / 无损压缩优化，并将压缩后资源文件重映射到前端框架的静态资源路径下。

[0027] 本发明采用以上技术方案，与现有技术相比具有如下有益效果：

[0028] （1）本发明实现了从设计图输入到可运行前端项目的全流程自动化，无需开发人员手动编码、调试及资源整理，缩短开发周期，极大释放了人力资源，使团队可将更多精力投入业务逻辑与用户体验优化。

[0029] （2）本发明通过创新的图像理解层，能够精准识别各组件的像素级边界、空间叠放顺序与父子嵌套关系，从二维图像中还原出完整的三维层级结构，使最终生成的代码在组件位置、间距、颜色、字体等属性上与原始设计图的像素级吻合度提升，有效避免了因人工误判导致的返工。

[0030] （3）本发明独创的交互式意图推理机制，能够在用户点击未绑定事件的UI元素时，自动捕获操作意图并即时生成对应的目标页面代码，使静态设计图具备动态交互能力，打通了从原型设计到可交互应用的最后壁垒，特别适用于产品快速原型验证与敏捷开发场景。

[0031] （4）本发明支持PNG、JPG、Figma等多种设计图格式，并兼容React、Vue等主流前端框架，通过插件化设计还可扩展至Angular、Svelte等新兴框架，覆盖当前主流前端开发场景。无论是政企数字化转型中的批量规范处理，还是创业团队的快速MVP验证，均可无缝接入，无需改变现有工作流。附图说明

[0032] 以下结合附图和具体实施方式对本发明做进一步详细说明；

[0033] 图1为本发明基于多模态深度学习的UI设计图自动转前端代码的方法的流程示意图；

[0034] 图2为本发明结构化描述数据的生成流程示意图；

[0035] 图3为本发明目标页面源代码文件的生成流程示意图；

[0036] 图4为本发明推理用户意图的流程示意图。具体实施方式

[0037] 为使本申请实施例的目的、技术方案和优点更加清楚，下面将结合本申请实施例中的附图对本申请实施例中的技术方案进行清楚、完整地描述。

[0038] 如图1至图4所示，本发明具体实施公开了基于多模态深度学习的UI设计图自动转前端代码的方法，获取用户界面设计图的像素矩阵数据并预处理得到标准化的输入图像数据；像素矩阵数据为光栅图像格式的设计稿或经解析得到的设计稿页面图像；

[0039] 输入图像数据输入至基于Mask R-CNN改进的UI组件检测框架；UI组件检测框架通过卷积神经网络对输入图像数据进行多层卷积与池化以提取设计图的多尺度特征图；通过区域提议网络基于多尺度特征图生成UI组件的候选边界框；通过ROIAlign层对各候选边界框对应的特征图区域进行双线性插值对齐输出固定尺度的对齐特征图；基于对齐特征图并行执行边界框回归与像素级掩码预测，识别UI组件（按钮、文本框）的边界框位置并生成像素级掩码得到组件区域，实现UI组件与背景及相邻UI组件的高精度分离；具体地，除提取组件、布局、颜色、字体信息及重建层级结构外，大模型还对原型图中的交互逻辑提示进行识别和解析，如按钮的点击效果说明等，并将交互逻辑信息纳入结构化数据；针对高复杂度图层设计图，通过多模态融合技术进一步提升信息提取精度，避免信息丢失。

[0040] 通过一深度估计增强模块对各组件区域对应的图像块的不同卷积层的多尺度视觉特征进行跨层融合求解各UI组件的相对深度值；根据相对深度值确定各UI组件沿Z轴的堆叠先后顺序，并结合各组件边界框位置在XY平面的交叠关系推理父子容纳关系，构建表征组件嵌套层级与叠放顺序的层级树数据结构；

[0041] 提取层级树中的每个组件节点对应分割掩码区域内的像素级视觉属性，同时将组件区域图像块输入至视觉-语言嵌入模型，获得组件区域的语义嵌入向量并与预设UI组件语义标签集合进行相似度匹配得到对应UI组件的语义组件类型标签；将层级关系、空间布局坐标、像素级视觉属性与语义组件类型标签进行融合，生成全页结构化描述数据，结构化描述数据以层级树为骨架记录每个UI组件的类型、边界框位置、样式属性与父子隶属关系；其中，像素级视觉属性包括颜色分布、像素尺寸比例、字体纹理特征以及位置坐标；具体地，结合大模型（CLIP 模型）对图像进行深度分析，大模型通过自注意力机制捕捉图像中的全局上下文信息，提取组件布局结构、颜色值、字体类型和大小等细粒度属性。整个图像解析过程通过多模态融合（结合图像识别和语义理解）确保信息提取的全面性和准确性，将层级数据与基础属性信息整合为结构化数据。

[0042] 基于用户选定的前端框架类型按照前端框架的语法规则与组件化规范，将结构化描述数据映射转换为基础页面源代码，得到基础页面源代码文件；基础页面源代码包括各UI组件的声明代码、基于边界框坐标与层级树的布局定位代码、以及基于样式属性的样式描述代码；

[0043] 如图2所示，将生成的基础页面源代码文件编译渲染为交互预览页面，通过事件监听器监控用户对预览页面中未绑定事件处理函数的交互元素的触发操作，捕获被触发交互元素的位置信息与视觉语义信息；通过视觉语义信息查询预构建的UI交互知识图谱得到候选交互意图及目标页面类型，基于当前页面的上下文自动构造目标页面的结构化描述数据，并按照前端框架的语法规则映射生成目标页面源代码文件；其中，UI交互知识图谱以图谱节点存储典型UI组件类型与关联的交互模式类型、页面流转关系及目标页面模板的映射关系；

[0044] 将基础页面源代码文件、目标页面源代码文件以及从设计图数据中提取并经过格式转换与压缩优化的资源文件，按照前端框架的目录结构进行路径映射与依赖关联，输出为可运行的UI项目文件包。具体地，系统将生成的基础页面代码、交互式生成的目标页面代码及资源文件打包，输出一个完整可运行的 React UI 项目，开发人员可直接将该项目用于开发或测试。

[0045] 进一步地，本发明支持多种格式设计图及原型图导入。当获取的UI设计图为Figma原型文件格式时，系统解析Figma文件的图层树结构与矢量节点数据，将矢量节点栅格化为所述像素矩阵数据，同时保留图层名称语义作为辅助标注信息参与语义标签匹配。具体地，开发人员导入Figma格式的原型图，系统支持该格式的直接解析，无需额外转换，同时智能适配其图层复杂度与分辨率特性。

[0046] 进一步地，UI组件类别集合至少包括按钮、输入框、下拉选择框、文本标签、图像容器、列表容器、卡片容器、导航栏区域中的一种或多种；像素级掩码预测输出每个候选区域对应的软掩码，并通过预设阈值二值化得到前景分割掩码，以实现组件与背景的高精度分离，减少重叠组件区域的像素级误分割。

[0047] 进一步地，深度估计增强模块的实现包括：

[0048] 深度估计增强模块从卷积神经网络的至少两个不同层级分别提取第一尺度特征图与第二尺度特征图，通过上采样与逐通道加权融合得到融合特征图；

[0049] 将融合特征图输入深度回归头得到各组件区域对应的相对深度值，依据深度值由大到小确定UI组件从上至下的堆叠关系，即深度值较小的组件判定为位于堆叠下层或被遮挡，深度值较大的组件判定为位于堆叠上层；

[0050] 任意两个组件区域边界框在XY平面的交叠面积占小于设定阈值且第一个组件的相对深度值远小于第二组件的相对深度值时，在在层级树中将两个组件对应的两个节点标记为兄弟节点；交叠面积占大于设定阈值且第一个组件的相对深度值小于第二组件的相对深度值时，将第一组件节点设为第二组件节点的子节点。即当第一组件边界框被第二组件边界框以预设面积占比包含且第一个组件的相对深度值小于第二组件的相对深度值时，将第一组件节点设为第二组件节点的子节点。

[0051] 进一步地，视觉-语言嵌入模型为CLIP模型或基于对比学习的视觉-语言预训练模型，语义组件类型标签的匹配过程为：将组件区域图像块编码为视觉嵌入向量，分别计算视觉嵌入向量与各预设UI组件语义标签的文本嵌入向量的余弦相似度；取相似度最大的标签作为对应组件区域的语义组件类型标签；并且，对分割掩码区域内的像素进行颜色聚类，提取主色值作为背景色或填充色属性写入结构化描述数据。

[0052] 本发明通过结合改进的Mask R-CNN模型和深度估计模块，能够实现UI组件的精准检测和分割，尤其对于重叠组件或复杂布局的设计图，能有效区分组件层级，避免误检和漏检。大模型的引入增强了语义理解能力，能够识别设计图中的文本内容和视觉样式，从而准确提取组件属性。本发明技术方案的应用场景包括 Web 应用、移动应用等UI开发，尤其在快速原型设计和批量处理中，能显著提高设计图还原度（从70%-80%提升至95%以上），减少人工干预。

[0053] 进一步地，前端框架类型包括React框架或Vue框架；当为React框架时，生成的组件声明代码为扩展自React.Component或函数式组件形式的.tsx / .jsx文件，布局定位代码基于Flexbox或Grid的样式描述实现；当为Vue框架时，生成的组件声明代码为单文件组件.vue格式，布局定位代码写入<style>块的CSS描述中。

[0054] 具体地，开发人员选择React / Vue开发框架，大模型依据上一步得到的结构化数据，按照React / Vue框架的语法规则和组件规范，生成对应的UI代码，包括组件定义、样式设置和布局实现等部分，其中包含基本的交互实现代码，如按钮点击事件的处理函数等。系统生成初始UI原型后自动激活交互式环境，当用户点击原型中未绑定onClick事件的"查看详情”按钮时，模块捕获该交互行为；大模型调用UI交互知识图谱，结合按钮语义信息推理用户意图为"跳转详情页”，基于当前页面上下文设计详情页UI布局；经用户确认设计匹配后，即时生成详情页UI代码文件（如DetailPage.tsx）。

[0055] 进一步地，交互知识图谱的构建过程包括：收集典型UI设计系统中的交互模式样本，抽取三元组形式的关系事实（即组件类型—交互模式—目标页面 / 动作类型），以图数据库或邻接表结构存储。

[0056] 进一步地，候选交互意图的推理过程包括：以被触发交互元素的语义组件类型标签与周围上下文组件的布局关系作为查询键，在交互知识图谱中检索匹配度最高的交互模式三元组，当匹配度超过预设阈值时，将对应的目标页面类型作为推理结果输出。

[0057] 如图4所示，基于微调后的Qwen3-1.7B意图推理大模型，优化意图识别精度；当用户连续触发多个未绑定事件的操作（如点击"设置”图标后选择"账号安全”选项），系统结合多步操作上下文及UI交互知识图谱，推理复合意图并生成对应的多级页面；支持用户对生成的目标页面进行在线微调，调整后系统即时更新代码。

[0058] 进一步地，当系统捕获用户连续多次对未绑定事件的交互元素的触发操作时，将连续操作的各交互元素的语义组件类型标签序列与对应页面上下文组织为多步操作上下文序列，在UI交互知识图谱中以子图匹配方式进行复合意图推理，生成对应于多级页面流转路径的多个目标页面结构化描述数据，并分别映射生成对应的多级目标页面源代码文件。

[0059] 进一步地，对资源文件中内嵌的图片与图标进行有损 / 无损压缩优化，并将压缩后资源文件重映射到前端框架的静态资源路径下。具体地，在处理图片和图标资源时，系统不仅进行格式转换，还对资源进行压缩优化，以减小UI项目的体积，并关联到Vue代码中；自动收集生成的页面代码所需的依赖资源，完成资源路径的智能映射。

[0060] 本发明可解决传统设计图转UI过程中效率低、还原度差、人力成本高的技术问题，同时突破静态原型交互局限，实现用户意图的智能解析与目标页面即时生成，达到提升开发效率、保障 UI 还原度、降低人力成本与出错概率、拓展动态交互生成能力的技术效果。

[0061] 本发明适用于Web应用、移动应用、桌面软件等各类产品的UI开发环节，尤其适配政企数字化转型项目的批量规范处理场景、产品迭代初期的快速原型验证场景。涉及的主要元素包括设计图文件、图像识别模块、大模型模块、代码生成模块、交互式生成模块、资源整合模块以及输出的UI项目。各模块协同工作，实现从设计图到可用UI的全流程自动化转换，其中图像识别模块与大模型模块协同完成设计图解析及层级重建，交互式生成模块负责意图推理与目标页面生成，代码生成模块与资源整合模块处理代码及资源关联，最终输出完整UI项目用于开发或测试。

[0062] 本发明采用以上技术方案，与现有技术相比具有如下有益效果：

[0063] （1）本发明实现了从设计图输入到可运行前端项目的全流程自动化，无需开发人员手动编码、调试及资源整理。传统方式中，一名熟练工程师完成一个中等复杂度页面的编码平均需要8小时，且需反复比对设计稿修正偏差；而采用本发明后，整个过程可在45分钟内完成，开发周期缩短80%以上，极大释放了人力资源，使团队可将更多精力投入业务逻辑与用户体验优化。

[0064] （2）本发明通过创新的图像理解层，能够精准识别各组件的像素级边界、空间叠放顺序与父子嵌套关系，从二维图像中还原出完整的三维层级结构，使最终生成的代码在组件位置、间距、颜色、字体等属性上与原始设计图的像素级吻合度提升至95%以上，有效避免了因人工误判导致的返工。

[0065] （3）本发明独创的交互式意图推理机制，能够在用户点击未绑定事件的UI元素时，自动捕获操作意图并即时生成对应的目标页面代码，使静态设计图具备动态交互能力，打通了从原型设计到可交互应用的最后壁垒，特别适用于产品快速原型验证与敏捷开发场景。

[0066] （4）本发明支持PNG、JPG、Figma等多种设计图格式，并兼容React、Vue等主流前端框架，通过插件化设计还可扩展至Angular、Svelte等新兴框架，覆盖当前95%以上的前端开发场景。无论是政企数字化转型中的批量规范处理，还是创业团队的快速MVP验证，均可无缝接入，无需改变现有工作流。

[0067] 显然，所描述的实施例是本申请一部分实施例，而不是全部的实施例。在不冲突的情况下，本申请中的实施例及实施例中的特征可以相互组合。通常在此处附图中描述和示出的本申请实施例的组件可以以各种不同的配置来布置和设计。因此，本申请的实施例的详细描述并非旨在限制要求保护的本申请的范围，而是仅仅表示本申请的选定实施例。基于本申请中的实施例，本领域普通技术人员在没有作出创造性劳动前提下所获得的所有其他实施例，都属于本申请保护的范围。< / style>

Claims

1. A method for automatically converting UI design drafts into front-end code based on multimodal deep learning, executed by a computing device, characterized in that... The method includes the following steps: Obtain the pixel matrix data of the user interface design draft and preprocess it to obtain standardized input image data; the pixel matrix data is a design draft in raster image format or a parsed design draft page image; The input image data is fed into a UI component detection framework based on Mask R-CNN; the UI component detection framework uses a convolutional neural network to perform multi-layer convolution and pooling on the input image data to extract multi-scale feature maps of the design drawing; and a region proposal network generates candidate bounding boxes of UI components based on the multi-scale feature maps. The ROIAlign layer performs bilinear interpolation to align the feature map regions corresponding to each candidate bounding box, outputting an aligned feature map of a fixed scale. Based on the aligned feature map, bounding box regression and pixel-level mask prediction are performed in parallel to identify the bounding box position of UI components and generate pixel-level masks to obtain the component region; A depth estimation enhancement module is used to perform cross-layer fusion of multi-scale visual features of different convolutional layers of image blocks corresponding to each component region to solve the relative depth value of each UI component; the stacking order of each UI component along the Z-axis is determined based on the relative depth value, and the parent-child containment relationship is inferred by combining the overlapping relationship of the bounding box positions of each component in the XY plane, and a hierarchical tree data structure representing the nesting level and stacking order of components is constructed. Extract pixel-level visual attributes from the segmentation mask region corresponding to each component node in the hierarchical tree. Simultaneously, input the component region image patch into the visual-language embedding model to obtain the semantic embedding vector of the component region and perform similarity matching with the preset set of UI component semantic labels to obtain the semantic component type label of the corresponding UI component. The hierarchical relationship, spatial layout coordinates, pixel-level visual attributes, and semantic component type tags are integrated to generate full-page structured description data. The structured description data records the type, bounding box position, style attributes, and parent-child relationship of each UI component with a hierarchical tree as the skeleton. Among them, the pixel-level visual attributes include color distribution, pixel size ratio, font texture features, and position coordinates. Based on the user-selected front-end framework type, the structured description data is mapped and converted into basic page source code according to the syntax rules and componentization specifications of the front-end framework, resulting in basic page source code files. The basic page source code includes the declaration code of each UI component, the layout positioning code based on bounding box coordinates and hierarchy tree, and the style description code based on style attributes. The generated base page source code file is compiled and rendered into an interactive preview page. Event listeners monitor user actions on interactive elements in the preview page that are not bound to event handlers, capturing the location and visual semantic information of the triggered interactive elements. Candidate interaction intentions and target page types are obtained by querying a pre-built UI interaction knowledge graph using visual semantic information. The structured description data of the target page is automatically constructed based on the context of the current page, and the target page source code file is generated according to the syntax rules of the front-end framework. The UI interaction knowledge graph stores the mapping relationship between typical UI component types and associated interaction mode types, page flow relationships, and target page templates in graph nodes. The source code files of the base page, the source code files of the target page, and the resource files extracted from the design data and optimized by format conversion and compression are mapped and associated according to the directory structure of the front-end framework, and output as a runnable UI project file package.

2. The method for automatically converting UI design drafts into front-end code based on multimodal deep learning according to claim 1, characterized in that, When the obtained UI design drawing is in Figma prototype file format, the system parses the layer tree structure and vector node data of the Figma file, rasterizes the vector nodes into the pixel matrix data, and retains the semantic meaning of the layer name as auxiliary annotation information to participate in semantic tag matching.

3. The method for automatically converting UI design drafts into front-end code based on multimodal deep learning according to claim 1, characterized in that, The UI component category set includes at least one or more of the following: buttons, input boxes, drop-down selection boxes, text labels, image containers, list containers, card containers, and navigation bar areas; pixel-level mask prediction outputs the soft mask corresponding to each candidate area, and obtains the foreground segmentation mask by binarization through a preset threshold, so as to achieve high-precision separation of components and background and reduce pixel-level missegmentation of overlapping component areas.

4. The method for automatically converting UI design drafts into front-end code based on multimodal deep learning according to claim 1, characterized in that, The implementation of the depth estimation enhancement module includes: The depth estimation enhancement module extracts first-scale feature maps and second-scale feature maps from at least two different layers of the convolutional neural network, and obtains a fused feature map by upsampling and channel-wise weighted fusion. The fused feature map is input into the depth regression head to obtain the relative depth value corresponding to each component region. The stacking relationship of UI components from top to bottom is determined based on the depth value from large to small. That is, the component with the smaller depth value is determined to be located in the lower layer of the stack or is occluded, and the component with the larger depth value is determined to be located in the upper layer of the stack. When the overlap area of ​​the bounding boxes of any two components in the XY plane is less than a set threshold and the relative depth of the first component is much smaller than the relative depth of the second component, the two corresponding nodes of the two components are marked as sibling nodes in the hierarchy tree. When the overlap area is greater than the set threshold and the relative depth of the first component is smaller than the relative depth of the second component, the node of the first component is set as a child node of the node of the second component. That is, when the bounding box of the first component is contained by the bounding box of the second component with a preset area ratio and the relative depth of the first component is smaller than the relative depth of the second component, the node of the first component is set as a child node of the node of the second component.

5. The method for automatically converting UI design drafts into front-end code based on multimodal deep learning according to claim 1, characterized in that, The visual-language embedding model is either the CLIP model or a visual-language pre-trained model based on contrastive learning. The matching process of semantic component type labels is as follows: the component region image patch is encoded into a visual embedding vector, and the cosine similarity between the visual embedding vector and the text embedding vector of each preset UI component semantic label is calculated. The label with the highest similarity is taken as the semantic component type label of the corresponding component region. Furthermore, the pixels in the segmented mask region are clustered by color, and the main color value is extracted as the background color or fill color attribute and written into the structured description data.

6. The method for automatically converting UI design drafts into front-end code based on multimodal deep learning according to claim 1, characterized in that, The front-end framework type includes React or Vue. When using React, the generated component declaration code is a .tsx / .jsx file extending React.Component or a functional component, and the layout and positioning code is implemented based on Flexbox or Grid style descriptions. When using Vue, the generated component declaration code is a single-file component in .vue format, and the layout and positioning code is written to... <style>块的CSS描述中。7.根据权利要求1所述的基于多模态深度学习的UI设计图自动转前端代码的方法，其特征在于，交互知识图谱的构建过程包括：收集典型UI设计系统中的交互模式样本，抽取三元组形式的关系事实，以图数据库或邻接表结构存储。8.根据权利要求1所述的基于多模态深度学习的UI设计图自动转前端代码的方法，其特征在于，候选交互意图的推理过程包括：以被触发交互元素的语义组件类型标签与周围上下文组件的布局关系作为查询键，在交互知识图谱中检索匹配度最高的交互模式三元组，当匹配度超过预设阈值时，将对应的目标页面类型作为推理结果输出。9.根据权利要求1所述的基于多模态深度学习的UI设计图自动转前端代码的方法，其特征在于，当系统捕获用户连续多次对未绑定事件的交互元素的触发操作时，将连续操作的各交互元素的语义组件类型标签序列与对应页面上下文组织为多步操作上下文序列，在UI交互知识图谱中以子图匹配方式进行复合意图推理，生成对应于多级页面流转路径的多个目标页面结构化描述数据，并分别映射生成对应的多级目标页面源代码文件。10.根据权利要求1所述的基于多模态深度学习的UI设计图自动转前端代码的方法，其特征在于，对资源文件中内嵌的图片与图标进行有损 / 无损压缩优化，并将压缩后资源文件重映射到前端框架的静态资源路径下。< / style>

Images