An off-line user interface code automatic generation method based on image recognition

Abstract

This invention provides an offline user interface code automatic generation method based on image recognition, including offline recognition and structuring of UI elements and layouts, conversion of UI description data to intermediate representation, and a Qt code generator. It ensures operation in offline environments, making it particularly suitable for development environments with high security requirements. The core lies in combining computer vision, OCR, and a native Qt toolchain, using the cooperation of multiple tools to achieve the conversion from screenshots to executable programs.

Description

Technical Field

[0001] This invention relates to the field of simulation system technology, and in particular to an automatic offline user interface code generation method based on image recognition. Background Technology

[0002] In the field of equipment software development, the research and development of equipment simulation software needs to meet the security and confidentiality requirements of equipment software prototype data. Existing development models typically rely on the following methods for converting UI design into code: Manual coding: Developers manually write code based on UI design drafts, which is inefficient and prone to errors.

[0003] Online UI design tools, such as Figma and Adobe XD, can generate some front-end code (such as HTML / CSS), but they usually cannot directly generate Qt Widgets or QML code and require an internet connection to use.

[0004] Template-based code generation: requires predefined templates, lacks flexibility, and cannot handle arbitrary screenshots.

[0005] Existing end-to-end research: There is some research on "graph-to-code" in academia and industry (such as pix2code), but they generally have the following limitations: Online dependencies: Most rely on cloud APIs for image recognition or model inference.

[0006] Domain limitations: Primarily designed for web front-end development, it lacks versatility and is not suitable for complex controls in desktop applications (such as QTableView and QTreeView in Qt).

[0007] Low layout fidelity: The generated code layout differs significantly from the original image, making it impossible to accurately reproduce the position and size of controls.

[0008] Lack of offline solutions: Cannot be used in internal networks or closed environments with high confidentiality requirements.

[0009] Therefore, there is an urgent need for an automated UI code generation solution that can run in offline environments, reproduce high-fidelity code, and is specifically designed for cross-platform application frameworks (such as the Qt framework). Summary of the Invention

[0010] The purpose of this invention is to address the shortcomings of the prior art by providing an automatic offline user interface code generation method based on image recognition.

[0011] To achieve the above objectives, the present invention adopts the following technical solution: This invention provides a method for automatically generating offline user interface code based on image recognition, comprising the following steps: S1, UI element detection and recognition, including: image preprocessing stage, element detection and recognition stage, layout analysis and hierarchy inference stage, and structured output stage; S2, Data Transformation and Layout Reasoning, including: data parsing and tree construction, layout strategy reasoning, layout parameter calculation, and intermediate representation generation; S3 and Qt code generation, including: template selection and configuration, recursive code generation, and output and formatting.

[0012] Furthermore, the image preprocessing stage includes: size normalization, contrast or brightness enhancement, and grayscale conversion; The element detection and recognition stage includes: target detection branch and OCR text recognition branch; The OCR text recognition branch includes the following steps: Input: Original image or preprocessed image; Text detection: Locate all regions containing text in an image; Text recognition: Identifies specific text content within the detection area; Tools: Use the offline version of the Tesseract OCR engine or the CRNN deep learning model; Output: A list of texts, each containing: {"content": "Username:", "bbox": [x,y, w, h]}.

[0013] Furthermore, in the layout analysis and hierarchy inference stage, a tree structure with hierarchical relationships is constructed, including: Data association: Matching the text recognized by OCR to the controls recognized by object detection; Control Relationship Analysis: Calculate the relative positional relationships between all controls; Calculate overlapping area: Determine if elements are nested; Calculate relative position: Determine if an element is left-aligned, right-aligned, top-aligned, horizontally centered, or vertically aligned; The tree structure algorithm for the hierarchical relationship uses heuristic rules to infer parent-child relationships, specifically: Nesting rule: If the bounding box of element A completely contains the bounding box of element B, then element B is a child element of element A. Grouping rule: If multiple elements are close in space and aligned, they are wrapped by an invisible container. The algorithm needs to infer and create this virtual parent container. Alignment rule: Adjacent elements with the same alignment are in the same layout manager and are sibling nodes; The structured output includes: metadata and a UI element tree; The metadata describes overall information about the identification task, including: Version number: Used to identify the version of the JSON schema, facilitating future format expansion and compatibility; Source image information: including the filename and original size of the source screenshot; Timestamp: Records the time when the recognition task was completed; Overall confidence level: A value between 0 and 1, representing the average confidence level of the entire recognition process; Target Qt version: Identifies the Qt framework version for which the generated code is intended; The UI element tree includes a root-level container and attribute descriptions for individual UI elements; The root-level container is an element that represents the window and the top-level dialog box, containing the size and position information of the entire interface, and all UI elements exist as child nodes or descendant nodes. The attribute descriptions of the UI elements: Each identified UI element is described in detail as a node in the tree, including the following information: Unique Identifier: A unique ID within a file, used for precise referencing within the tree; Element type: The type of QT control identified; Bounding box: Describes the precise location and size of an element in an image; Attribute set: including text content, visual style, state, and QT-specific attributes; Detection confidence: A value between 0 and 1, representing the accuracy of intelligent element recognition; Relationship pointer: includes parent node ID and list of child nodes; Layout tips: Provide key clues for what to do next, including suggested layout types, element alignment within the layout, element sizing strategies, and stretch factors.

[0014] Furthermore, the data transformation and layout reasoning includes data parsing and tree construction, layout strategy reasoning, layout parameter calculation, and intermediate representation generation; The data parsing and tree construction are specifically as follows: Parsing JSON: Use a standard JSON parsing library to read the file; Rebuild the control tree: Recursively create control nodes based on the children field in the JSON and establish parent-child pointer relationships; Geometric information enhancement: In addition to the absolute coordinates already present in the JSON, derived geometric information is calculated; Layout strategy reasoning includes analyzing the spatial relationships between sub-elements, applying rules for inference, and recursive processing. Analyze the spatial relationships of sub-elements, including: Alignment: Checks whether the left, right, top, or bottom center lines of child elements are aligned; Uniformity: Check if the spacing between child elements is equal; Grid mode: Checks whether child elements can be arranged into a non-incomplete grid; Inference based on applied rules includes: Horizontal layout: If most of the child elements are arranged horizontally and have similar heights and alignments, then choose QHBoxLayout; Vertical layout: If most of the child elements are arranged vertically and have similar widths and alignment, then choose QVBoxLayout; Grid layout: If the child elements can be clearly divided into rows and columns, choose QGridLayout. You need to calculate the row index and column index of each element. No layout or absolute positioning: If there is a lot of overlap between child elements, or the arrangement is irregular, or it is a complex control drawn by a custom artist, then it is determined that no layout manager is used and absolute coordinates are retained; Recursive processing: Once the layout type is inferred for the current parent container, the same layout reasoning process needs to be recursively performed on each child element under this parent container.

[0015] Furthermore, the calculation of the layout parameters is specifically as follows: For BoxLayout: Stretch Factor: Calculates the stretch factor for each child element. If an element is significantly wider or taller than other elements in the same row or column, it is assigned a larger stretch value. The formula is: stretch_i = (element_size_i / average_size) * 10; Spacing: Calculates the average spacing between sibling elements and sets it as the spacing value for the layout; For GridLayout: Row or column span: If an element's bounding box covers multiple grid cells, then rowSpan and colSpan need to be calculated; For all layouts: Margin: Calculates the gap between the outermost child element and the boundary of the parent container, and sets it as the layout's margin value; The intermediate representation generation involves appending all calculated layout information to the control tree nodes in memory and finally serializing it into a format that is easy for the code generator to process, specifically: Additional information: Each node now contains not only its own attributes, but also layout attributes and a reference to the layout manager applied to its corresponding child nodes; Serialization: Converts the control tree object into a final intermediate representation, which is a more complex JSON, an XML, or a memory structure that maps directly to the Qt meta-object system.

[0016] Furthermore, the template selection and configuration specifically include: Determine the output target: Read the configuration to decide whether to generate code based on C++ Widgets or declarative QML code; Load templates: The system has two built-in template libraries, which allow for the embedding of variables and logic control; Initialize context: Create a context dictionary or object to store and pass all variables that need to be populated into the template during code generation; The recursive code generation specifically involves: Traversing nodes: Start traversing from the root node of the IR tree; Node type identification and processing: For each node, first determine whether it is a layout container node or a UI control node; The layout container node includes: Generate the code that creates the layout manager object; Generate code to set layout parameters, filling in the margin, spacing, and stretch values ​​calculated in stage three; Recursively process all child nodes: Recursive calls ensure the correct generation of nested layouts; Generate code that sets the layout to the parent control; The UI control nodes include: Generate code to create control objects; Instantiate the corresponding Qt class based on the type field in the IR; Generate code to set control properties, including geometric properties, text properties, and other properties; Generate style sheet code: If the IR contains the identified color and font style information, convert the style information into a QSS string and apply it through the setStyleSheet method; Establish parent-child relationship: Add the currently generated control or layout to the layout of the parent node.

[0017] Furthermore, the output and formatting specifically include: file generation, resource processing, code formatting, and output; The file generation includes: C++ mode: generating header files containing class declarations and control pointer member variables; generating source files containing the specific implementation of the setupUi function; QML pattern: Generate one or more .qml files to describe the entire UI using declarative syntax; Resource processing: If icon resources are extracted during the UI recognition phase, they will be saved to the project directory and a .qrc file will be generated to register the resource file with Qt's resource system; The code formatting involves calling Clang-Format or a custom formatter to format the generated raw code. Output: Finally, a complete Qt project directory is generated, containing all program files, with a clear structure and standardized format. Users can directly import it into QtCreator for compilation and further development.

[0018] The beneficial effects of this invention are: offline recognition and structuring of UI elements and layouts, utilizing a locally deployed lightweight deep learning model (a cropped and optimized YOLO, SSD object detection model, or U-Net segmentation model) to perform pixel-level analysis on the input user interface screenshots. The input is a software screenshot image, such as JPG, BMP, PNG, etc., and the output is a structured JSON file containing the type, absolute / relative coordinates, size, text attributes, and parent-child hierarchy relationships of each UI element. UI description data to intermediate representation conversion: Design a dedicated conversion engine to convert the JSON structured data generated in the previous step into an intermediate representation compatible with the Qt layout system. The input is a structured JSON file output by the offline recognition and structuring module for UI elements and layouts, and the output is an intermediate data structure rich in layout information corresponding to Qt Widgets or QML object models; Qt code generator: Based on an intermediate representation, it uses code template technologies (such as Jinja2) to generate Qt code that can be directly compiled and runnable. The input is the intermediate data structure transformed from UI description data to the intermediate representation; the output is a complete, compilable .cpp / .h file or .qml file. By implementing these three steps, it is possible to work in a network-free environment, making it particularly suitable for development environments with high security requirements. The core lies in combining computer vision, OCR, and the local Qt toolchain, using the cooperation of multiple tools to convert screenshots into executable programs. Attached Figure Description

[0019] Figure 1 A diagram illustrating the components of an offline user interface code automatic generation method based on image recognition; Figure 2This is an offline workflow diagram; Figure 3 Workflow diagram for the offline recognition module of UI elements and layout; Figure 4 Workflow diagram for the data transformation and layout reasoning module; Figure 5 Workflow diagram for Qt code generation module. Detailed Implementation

[0020] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.

[0021] Please see Figures 1 to 5 An offline user interface code automatic generation method based on image recognition includes the following steps: S1, UI element detection and recognition, including: image preprocessing stage, element detection and recognition stage, layout analysis and hierarchy inference stage, and structured output stage; S2, Data Transformation and Layout Reasoning, including: data parsing and tree construction, layout strategy reasoning, layout parameter calculation, and intermediate representation generation; S3 and Qt code generation, including: template selection and configuration, recursive code generation, and output and formatting.

[0022] The image preprocessing stage includes: size normalization, contrast or brightness enhancement, and grayscale conversion; The element detection and recognition stage includes: target detection branch and OCR text recognition branch; The OCR text recognition branch includes the following steps: Input: Original image or preprocessed image; Text detection: Locate all regions containing text in an image; Text recognition: Identifies specific text content within the detection area; Tools: Use the offline version of the Tesseract OCR engine or the CRNN deep learning model; Output: A list of texts, each containing: {"content": "Username:", "bbox": [x,y, w, h]}.

[0023] The layout analysis and hierarchy inference stage constructs a tree structure with hierarchical relationships, including: Data association: Matching the text recognized by OCR to the controls recognized by object detection; Control Relationship Analysis: Calculate the relative positional relationships between all controls; Calculate overlapping area: Determine if elements are nested; Calculate relative position: Determine if an element is left-aligned, right-aligned, top-aligned, horizontally centered, or vertically aligned; The tree structure algorithm for the hierarchical relationship uses heuristic rules to infer parent-child relationships, specifically: Nesting rule: If the bounding box of element A completely contains the bounding box of element B, then element B is a child element of element A. Grouping rule: If multiple elements are close in space and aligned, they are wrapped by an invisible container. The algorithm needs to infer and create this virtual parent container. Alignment rule: Adjacent elements with the same alignment are in the same layout manager and are sibling nodes; The structured output includes: metadata and a UI element tree; The metadata describes overall information about the identification task, including: Version number: Used to identify the version of the JSON schema, facilitating future format expansion and compatibility; Source image information: including the filename and original size of the source screenshot; Timestamp: Records the time when the recognition task was completed; Overall confidence level: A value between 0 and 1, representing the average confidence level of the entire recognition process; Target Qt version: Identifies the Qt framework version for which the generated code is intended; The UI element tree includes a root-level container and attribute descriptions for individual UI elements; The root-level container is an element that represents the window and the top-level dialog box, containing the size and position information of the entire interface, and all UI elements exist as child nodes or descendant nodes. The attribute descriptions of the UI elements: Each identified UI element is described in detail as a node in the tree, including the following information: Unique Identifier: A unique ID within a file, used for precise referencing within the tree; Element type: The type of QT control identified; Bounding box: Describes the precise location and size of an element in an image; Attribute set: including text content, visual style, state, and QT-specific attributes; Detection confidence: A value between 0 and 1, representing the accuracy of intelligent element recognition; Relationship pointer: includes parent node ID and list of child nodes; Layout tips: Provide key clues for what to do next, including suggested layout types, element alignment within the layout, element sizing strategies, and stretch factors.

[0024] The data transformation and layout reasoning includes data parsing and tree construction, layout strategy reasoning, layout parameter calculation, and intermediate representation generation; The data parsing and tree construction are specifically as follows: Parsing JSON: Use a standard JSON parsing library to read the file; Rebuild the control tree: Recursively create control nodes based on the children field in the JSON and establish parent-child pointer relationships; Geometric information enhancement: In addition to the absolute coordinates already present in the JSON, derived geometric information is calculated; Layout strategy reasoning includes analyzing the spatial relationships between sub-elements, applying rules for inference, and recursive processing. Analyze the spatial relationships of sub-elements, including: Alignment: Checks whether the left, right, top, or bottom center lines of child elements are aligned; Uniformity: Check if the spacing between child elements is equal; Grid mode: Checks whether child elements can be arranged into a non-incomplete grid; Inference based on applied rules includes: Horizontal layout: If most of the child elements are arranged horizontally and have similar heights and alignments, then choose QHBoxLayout; Vertical layout: If most of the child elements are arranged vertically and have similar widths and alignment, then choose QVBoxLayout; Grid layout: If the child elements can be clearly divided into rows and columns, choose QGridLayout. You need to calculate the row index and column index of each element. No layout or absolute positioning: If there is a lot of overlap between child elements, or the arrangement is irregular, or it is a complex control drawn by a custom artist, then it is determined that no layout manager is used and absolute coordinates are retained; Recursive processing: Once the layout type is inferred for the current parent container, the same layout reasoning process needs to be recursively performed on each child element under this parent container.

[0025] The layout parameters are calculated as follows: For BoxLayout: Stretch Factor: Calculates the stretch factor for each child element. If an element is significantly wider or taller than other elements in the same row or column, it is assigned a larger stretch value. The formula is: stretch_i = (element_size_i / average_size) * 10; Spacing: Calculates the average spacing between sibling elements and sets it as the spacing value for the layout; For GridLayout: Row or column span: If an element's bounding box covers multiple grid cells, then rowSpan and colSpan need to be calculated; For all layouts: Margin: Calculates the gap between the outermost child element and the boundary of the parent container, and sets it as the layout's margin value; The intermediate representation generation involves appending all calculated layout information to the control tree nodes in memory and finally serializing it into a format that is easy for the code generator to process, specifically: Additional information: Each node now contains not only its own attributes, but also layout attributes and a reference to the layout manager applied to its corresponding child nodes; Serialization: Converts the control tree object into a final intermediate representation, which is a more complex JSON, an XML, or a memory structure that maps directly to the Qt meta-object system.

[0026] The template selection and configuration are specifically as follows: Determine the output target: Read the configuration to decide whether to generate code based on C++ Widgets or declarative QML code; Load templates: The system has two built-in template libraries, which allow for the embedding of variables and logic control; Initialize context: Create a context dictionary or object to store and pass all variables that need to be populated into the template during code generation; The recursive code generation specifically involves: Node traversal: Start traversing from the root node of the IR tree (Intermediate Representation); Node type identification and processing: For each node, first determine whether it is a layout container node or a UI control node; The layout container node includes: Generate the code that creates the layout manager object; Generate code to set layout parameters, filling in the margin, spacing, and stretch values ​​calculated in stage three; Recursively process all child nodes: Recursive calls ensure the correct generation of nested layouts; Generate code that sets the layout to the parent control; The UI control nodes include: Generate code to create control objects; Instantiate the corresponding Qt class based on the type field in the IR; Generate code to set control properties, including geometric properties, text properties, and other properties; Generate style sheet code: If the IR contains the identified color and font style information, convert the style information into a QSS string and apply it through the setStyleSheet method; Establish parent-child relationship: Add the currently generated control or layout to the layout of the parent node.

[0027] The output and formatting specifically include: file generation, resource processing, code formatting, and output; The file generation includes: C++ mode: generating header files containing class declarations and control pointer member variables; generating source files containing the specific implementation of the setupUi function; QML pattern: Generate one or more .qml files to describe the entire UI using declarative syntax; Resource processing: If icon resources are extracted during the UI recognition phase, they will be saved to the project directory and a .qrc file will be generated to register the resource file with Qt's resource system; The code formatting involves calling Clang-Format or a custom formatter to format the generated raw code. Output: Finally, a complete Qt project directory is generated, containing all program files, with a clear structure and standardized format. Users can directly import it into QtCreator for compilation and further development.

[0028] The embodiments described above are merely illustrative of implementation methods of the present invention, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of the present invention. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of the present invention, and these modifications and improvements all fall within the scope of protection of the present invention. Therefore, the scope of protection of this patent should be defined by the appended claims.

Claims

1. A method for automatically generating offline user interface code based on image recognition, characterized in that, Includes the following steps: S1, UI element detection and recognition, including: image preprocessing stage, element detection and recognition stage, layout analysis and hierarchy inference stage, and structured output stage; S2, Data Transformation and Layout Reasoning, including: data parsing and tree construction, layout strategy reasoning, layout parameter calculation, and intermediate representation generation; S3 and Qt code generation, including: template selection and configuration, recursive code generation, and output and formatting.

2. The method for automatically generating offline user interface code based on image recognition according to claim 1, characterized in that, The image preprocessing stage includes: size normalization, contrast or brightness enhancement, and grayscale conversion; The element detection and recognition stage includes: target detection branch and OCR text recognition branch; The OCR text recognition branch includes the following steps: Input: Original image or preprocessed image; Text detection: Locate all regions containing text in an image; Text recognition: Identifies specific text content within the detection area; Tools: Use the offline version of the Tesseract OCR engine or the CRNN deep learning model; Output: A list of texts, each containing: {"content": "Username:", "bbox": [x, y, w,h]}.

3. The method for automatically generating offline user interface code based on image recognition according to claim 2, characterized in that, The layout analysis and hierarchy inference stage constructs a tree structure with hierarchical relationships, including: Data association: Matching the text recognized by OCR to the controls recognized by object detection; Control Relationship Analysis: Calculate the relative positional relationships between all controls; Calculate overlapping area: Determine if elements are nested; Calculate relative position: Determine if an element is left-aligned, right-aligned, top-aligned, horizontally centered, or vertically aligned; The tree structure algorithm for the hierarchical relationship uses heuristic rules to infer parent-child relationships, specifically: Nesting rule: If the bounding box of element A completely contains the bounding box of element B, then element B is a child element of element A. Grouping rule: If multiple elements are close in space and aligned, they are wrapped by an invisible container. The algorithm needs to infer and create this virtual parent container. Alignment rule: Adjacent elements with the same alignment are in the same layout manager and are sibling nodes; The structured output includes: metadata and a UI element tree; The metadata describes overall information about the identification task, including: Version number: Used to identify the version of the JSON schema, facilitating future format expansion and compatibility; Source image information: including the filename and original size of the source screenshot; Timestamp: Records the time when the recognition task was completed; Overall confidence level: A value between 0 and 1, representing the average confidence level of the entire recognition process; Target Qt version: Identifies the Qt framework version for which the generated code is intended; The UI element tree includes a root-level container and attribute descriptions for individual UI elements; The root-level container is an element that represents the window and the top-level dialog box, containing the size and position information of the entire interface, and all UI elements exist as child nodes or descendant nodes. The attribute descriptions of the UI elements: Each identified UI element is described in detail as a node in the tree, including the following information: Unique Identifier: A unique ID within a file, used for precise referencing within the tree; Element type: The type of QT control identified; Bounding box: Describes the precise location and size of an element in an image; Attribute set: including text content, visual style, state, and QT-specific attributes; Detection confidence: A value between 0 and 1, representing the accuracy of intelligent element recognition; Relationship pointer: includes parent node ID and list of child nodes; Layout tips: Provide key clues for what to do next, including suggested layout types, element alignment within the layout, element sizing strategies, and stretch factors.

4. The method for automatically generating offline user interface code based on image recognition according to claim 3, characterized in that, The data transformation and layout reasoning includes data parsing and tree construction, layout strategy reasoning, layout parameter calculation, and intermediate representation generation; The data parsing and tree construction are specifically as follows: Parsing JSON: Use a standard JSON parsing library to read the file; Rebuild the control tree: Recursively create control nodes based on the children field in the JSON and establish parent-child pointer relationships; Geometric information enhancement: In addition to the absolute coordinates already present in the JSON, derived geometric information is calculated; Layout strategy reasoning includes analyzing the spatial relationships between sub-elements, applying rules for inference, and recursive processing. Analyze the spatial relationships of sub-elements, including: Alignment: Checks whether the left, right, top, or bottom center lines of child elements are aligned; Uniformity: Check if the spacing between child elements is equal; Grid mode: Checks whether child elements can be arranged into a non-incomplete grid; Inference based on applied rules includes: Horizontal layout: If most of the child elements are arranged horizontally and have similar heights and alignments, then choose QHBoxLayout; Vertical layout: If most of the child elements are arranged vertically and have similar widths and alignment, then choose QVBoxLayout; Grid layout: If the child elements can be clearly divided into rows and columns, choose QGridLayout. You need to calculate the row index and column index of each element. No layout or absolute positioning: If there is a lot of overlap between child elements, or the arrangement is irregular, or it is a complex control drawn by a custom artist, then it is determined that no layout manager is used and absolute coordinates are retained; Recursive processing: Once the layout type is inferred for the current parent container, the same layout reasoning process needs to be recursively performed on each child element under this parent container.

5. The method for automatically generating offline user interface code based on image recognition according to claim 4, characterized in that, The layout parameters are calculated as follows: For BoxLayout: Stretch Factor: Calculates the stretch factor for each child element. If an element is significantly wider or taller than other elements in the same row or column, it is assigned a larger stretch value. The formula is: stretch_i = (element_size_i / average_size) * 10; Spacing: Calculates the average spacing between sibling elements and sets it as the spacing value for the layout; For GridLayout: Row or column span: If an element's bounding box covers multiple grid cells, then rowSpan and colSpan need to be calculated; For all layouts: Margin: Calculates the gap between the outermost child element and the boundary of the parent container, and sets it as the layout's margin value; The intermediate representation generation involves appending all calculated layout information to the control tree nodes in memory and finally serializing it into a format that is easy for the code generator to process, specifically: Additional information: Each node now contains not only its own attributes, but also layout attributes and a reference to the layout manager applied to its corresponding child nodes; Serialization: Converts the control tree object into a final intermediate representation, which is a more complex JSON, an XML, or a memory structure directly mapped to the Qt meta-object system.

6. The method for automatically generating offline user interface code based on image recognition according to claim 5, Its characteristics are: The template selection and configuration are specifically as follows: Determine the output target: Read the configuration to decide whether to generate code based on C++ Widgets or declarative QML code; Load templates: The system has two built-in template libraries, which allow for the embedding of variables and logic control; Initialize context: Create a context dictionary or object to store and pass all variables that need to be populated into the template during code generation; The recursive code generation specifically involves: Traversing nodes: Start traversing from the root node of the IR tree; Node type identification and processing: For each node, first determine whether it is a layout container node or a UI control node; The layout container node includes: Generate the code that creates the layout manager object; Generate code to set layout parameters, filling in the margin, spacing, and stretch values ​​calculated in stage three; Recursively process all child nodes: Recursive calls ensure the correct generation of nested layouts; Generate code that sets the layout to the parent control; The UI control nodes include: Generate code to create control objects; Instantiate the corresponding Qt class based on the type field in the IR; Generate code to set control properties, including geometric properties, text properties, and other properties; Generate style sheet code: If the IR contains the identified color and font style information, convert the style information into a QSS string and apply it through the setStyleSheet method; Establish parent-child relationship: Add the currently generated control or layout to the layout of the parent node.

7. The method for automatically generating offline user interface code based on image recognition according to claim 6, characterized in that, The output and formatting specifically include: file generation, resource processing, code formatting, and output; The file generation includes: C++ mode: generating header files containing class declarations and control pointer member variables; generating source files containing the specific implementation of the setupUi function; QML pattern: Generate one or more .qml files to describe the entire UI using declarative syntax; Resource processing: If icon resources are extracted during the UI recognition phase, they will be saved to the project directory and a .qrc file will be generated to register the resource file with Qt's resource system; The code formatting involves calling Clang-Format or a custom formatter to format the generated raw code. Output: Finally, a complete Qt project directory is generated, containing all program files, with a clear structure and standardized format. Users can directly import it into QtCreator for compilation and further development.

Images