Code generation method and device, equipment, storage medium and program product

Abstract

The invention provides a code generation method and device, equipment, a storage medium and a program product. The method comprises the steps of obtaining an interface design image of an application program, wherein the interface design image comprises design information of a view component in an application program interface of the application program; performing view component detection on the interface design image to obtain the type and the position of a view component in the interface design image; obtaining first view structure information of the interface design image; adjusting the first view structure information based on the type and position of a view component in the interface design image to obtain second view structure information; and on the basis of the second view structure information, an executable code of the interface design image is generated, and when the executable code is executed, an application program interface corresponding to the interface design image is generated. By means of the method and device, the code generation efficiency and accuracy can be improved.

Description

Technical Field

[0001] This application relates to the field of computer technology, and in particular to a code generation method, apparatus, device, storage medium, and program product. Background Technology

[0002] Interface design images (also known as visual drafts) are used in the development of application interfaces, which involves converting these images into executable code. Related technologies often require manual annotation and hierarchical division of the interface design images to ensure the generated executable code conforms to the developers' expected structure. However, the need for manual intervention with specialized knowledge and skills not only increases costs but also leads to inefficient code generation and inconsistent accuracy. Summary of the Invention

[0003] This application provides a code generation method, apparatus, electronic device, computer-readable storage medium, and computer program product, which can improve the efficiency and accuracy of code generation.

[0004] The technical solution of this application embodiment is implemented as follows:

[0005] This application provides a code generation method, including:

[0006] Obtain the interface design image of the application, wherein the interface design image includes the design information of the view components in the application interface of the application;

[0007] Perform view component detection on the interface design image to obtain the type and position of the view components in the interface design image;

[0008] Obtain the first view structure information of the interface design image;

[0009] Based on the type and position of the view components in the interface design image, the first view structure information is adjusted to obtain the second view structure information;

[0010] Based on the second view structure information, executable code for the interface design image is generated, wherein when the executable code is executed, the application interface corresponding to the interface design image is generated.

[0011] This application also provides a code generation apparatus, including:

[0012] The first acquisition module is used to acquire the interface design image of the application, the interface design image including the design information of the view components in the application interface of the application;

[0013] The detection module is used to detect view components in the interface design image to obtain the type and position of the view components in the interface design image.

[0014] The second acquisition module is used to acquire the first view structure information of the interface design image;

[0015] The adjustment module is used to adjust the first view structure information based on the type and position of the view components in the interface design image to obtain the second view structure information;

[0016] The generation module is used to generate executable code for the interface design image based on the second view structure information, wherein when the executable code is executed, the application interface corresponding to the interface design image is generated.

[0017] In the above scheme, the view components in the interface design image have multiple component types; the detection module is also used to obtain the target detection model corresponding to each component type; through each target detection model, the view component detection is performed on the interface design image of the application to obtain the type and position of the view component in the interface design image corresponding to each target detection model.

[0018] In the above scheme, the first view structure information is represented by a first view tree. The interface design image contains multiple view components, and the first view tree includes a first node corresponding to each view component. The adjustment module is further configured to generate a second view tree including multiple node levels based on the position of the view components; determine the target node level to which each first node belongs in the second view tree; determine the node constraint relationship of the first node in the target node level; and adjust the first view tree based on the target node level, the node constraint relationship, and the type of the view component to obtain a third view tree, which is used to represent the second view structure information.

[0019] In the above scheme, the adjustment module is further configured to create a second node corresponding to each view component; for each view component, determine the node level to which the second node corresponding to the view component belongs based on the position of the view component; and construct the second view tree based on multiple second nodes and the node level to which each second node belongs.

[0020] In the above scheme, the position includes sub-positions obtained from multiple view component detections, and the sub-positions indicate the region box where the view component is located; the adjustment module is further configured to perform the following processing for each first node: determine a first region box including the view component corresponding to the first node from the region boxes indicated by the multiple sub-positions; when there are multiple first region boxes, determine the selection parameter value of each first region box, and select a second region box whose selection parameter value meets the parameter value condition from the multiple first region boxes; determine the target view component included in the second region box; and take the node level to which the second node corresponding to the target view component belongs as the target node level to which the first node belongs in the second view tree.

[0021] In the above scheme, the adjustment module is further configured to perform the following processing for each first region box: obtain the region area of ​​the first region box, the component area of ​​the view component corresponding to the first node, and the orthogonal area of ​​the view component corresponding to the first node and the first region box; obtain the detection confidence of the sub-position corresponding to the first region box and the component parameter value of the view component corresponding to the first node; and determine the selection parameter value of the first region box based on the region area, the component area, the orthogonal area, the detection confidence, and the component parameter value.

[0022] In the above scheme, the adjustment module is further configured to adjust the first view tree based on the target node hierarchy and the node constraint relationship to obtain an intermediate view tree; and add the type of the view component to the first node corresponding to the view component in the intermediate view tree to obtain the third view tree.

[0023] In the above scheme, the adjustment module is further used to delete the node view structure information of the first node in the first view tree to obtain an intermediate view tree; based on the target node level, the node constraint relationship and the type of the view component, to add new node view structure information to the first node in the intermediate view tree to obtain the third view tree.

[0024] In the above scheme, the adjustment module is further configured to, before adjusting the first view tree based on the target node level, the node constraint relationship, and the view component type to obtain the third view tree, determine a target first node in the first view tree that does not belong to the target node level; the adjustment module is further configured to, based on the target node level, the node constraint relationship, and the view component type, adjust the first view tree to obtain an intermediate view tree; and add the target first node to the intermediate view tree to obtain the third view tree.

[0025] In the above scheme, the adjustment module is further configured to determine the first node constraint relationship between the first nodes in the target node hierarchy based on the position of the view component corresponding to the first node; obtain the second node constraint relationship set for the first node in the interface design image; and construct the node constraint relationship of the first node in the target node hierarchy based on the first node constraint relationship and the second node constraint relationship.

[0026] In the above scheme, the generation module is further configured to generate component code for view components in the interface design image; generate a code framework based on the second view structure information; and add the component code to the code framework to obtain the executable code of the interface design image.

[0027] This application also provides an electronic device, including:

[0028] Memory is used to store executable instructions for a computer;

[0029] The processor, when executing computer-executable instructions stored in the memory, implements the code generation method provided in the embodiments of this application.

[0030] This application also provides a computer-readable storage medium storing computer-executable instructions or computer programs, which, when executed by a processor, implement the code generation method provided in this application.

[0031] This application also provides a computer program product, including computer-executable instructions or a computer program, which, when executed by a processor, implements the code generation method provided in this application.

[0032] The embodiments of this application have the following beneficial effects:

[0033] Applying the above embodiments of this application, firstly, an interface design image of the application is obtained. Then, view component detection is performed on the interface design image to obtain the type and position of the view components in the interface design image. Next, first view structure information of the interface design image is obtained. Based on the type and position of the view components in the interface design image, the first view structure information is adjusted to obtain second view structure information. Finally, based on the second view structure information, executable code for the interface design image is generated. When the executable code is executed, the application interface corresponding to the interface design image is generated. In this way, based on the type and position of the view components in the interface design image, the view structure information of the view components in the interface design image is automatically adjusted, thereby improving the efficiency and accuracy of code generation. Attached Figure Description

[0034] Figure 1 This is a schematic diagram of the architecture of the code generation system provided in the embodiments of this application;

[0035] Figure 2 This is a schematic diagram of the structure of the electronic device provided in the embodiments of this application;

[0036] Figure 3A This is a first flowchart illustrating the code generation method provided in an embodiment of this application;

[0037] Figure 3B This is a second flowchart illustrating the code generation method provided in the embodiments of this application;

[0038] Figure 3C This is a schematic diagram of the third process of the code generation method provided in the embodiments of this application;

[0039] Figure 4 This is a schematic diagram of the interface design images provided in the embodiments of this application;

[0040] Figure 5 This is a schematic diagram of the first view structure information provided in the embodiments of this application;

[0041] Figure 6 This is a schematic diagram of the executable code and application interface provided in the embodiments of this application;

[0042] Figure 7 This is a schematic diagram of the visual draft processing interface provided in an embodiment of this application;

[0043] Figure 8 This is a schematic diagram showing the correspondence between the visual draft and the code provided in the embodiments of this application;

[0044] Figure 9 This is a schematic diagram of the fourth process of the code generation method provided in the embodiments of this application;

[0045] Figure 10 This is a schematic diagram of the fifth process of the code generation method provided in the embodiments of this application;

[0046] Figure 11 This is a schematic diagram illustrating the detection effect of the target detection model provided in the embodiments of this application;

[0047] Figure 12 This is a schematic diagram of the sixth process of the code generation method provided in the embodiments of this application;

[0048] Figure 13 This is a schematic diagram of determining the second region box to which the view component belongs, provided in an embodiment of this application;

[0049] Figure 14 This is a schematic diagram of the seventh process of the code generation method provided in the embodiments of this application;

[0050] Figure 15 This is a schematic diagram of the view structure information provided in the embodiments of this application.

[0051] It should be noted that the terms "first" and "second" mentioned above are only used to distinguish between different options and do not represent the degree of superiority or inferiority of the options or their priority in the implementation process. Detailed Implementation

[0052] To make the objectives, technical solutions, and advantages of this application clearer, the application will be further described in detail below with reference to the accompanying drawings. The described embodiments should not be regarded as limitations on this application. All other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0053] In the following description, references are made to “some embodiments,” which describe a subset of all possible embodiments. However, it is understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0054] In the following description, the terms "first, second, third" are used merely to distinguish similar objects and do not represent a specific ordering of objects. It is understood that "first, second, third" may be interchanged in a specific order or sequence where permitted, so that the embodiments of this application described herein can be implemented in an order other than that illustrated or described herein.

[0055] In this application, the terms "module" or "unit" refer to a computer program or part of a computer program that has a predetermined function and works with other related parts to achieve a predetermined goal, and can be implemented wholly or partially using software, hardware (such as processing circuitry or memory), or a combination thereof. Similarly, a processor (or multiple processors or memory) can be used to implement one or more modules or units. Furthermore, each module or unit can be part of a larger module or unit that includes the functionality of the module or unit.

[0056] Unless otherwise defined, all technical and scientific terms used in the embodiments of this application have the same meaning as commonly understood by one of ordinary skill in the art. The terminology used in the embodiments of this application is for the purpose of describing the embodiments of this application only and is not intended to limit this application.

[0057] In the implementation of this application, the collection and processing of relevant data should strictly comply with the requirements of relevant laws and regulations, obtain the informed consent or separate consent of the personal information subject, and carry out subsequent data use and processing within the scope of laws and regulations and the authorization of the personal information subject.

[0058] Before providing a further detailed description of the embodiments of this application, the nouns and terms involved in the embodiments of this application will be explained, and the nouns and terms involved in the embodiments of this application shall be interpreted as follows.

[0059] 1) Client: An application running on an electronic device that provides various services, such as a client that supports code generation for interface design images (i.e., visual drafts).

[0060] 2) Responding to: used to indicate the conditions or states on which the operation is performed. When the conditions or states on which the operation is performed are met, one or more operations may be performed in real time or with a set delay. Unless otherwise specified, there is no restriction on the order in which the multiple operations are performed.

[0061] This application provides a code generation method, apparatus, electronic device, computer-readable storage medium, and computer program product, which can improve the efficiency and accuracy of code generation. The following is a detailed description of the embodiments of this application based on the above explanation of the terms and concepts used.

[0062] The code generation system provided in the embodiments of this application is described below. See also Figure 1 , Figure 1 This is a schematic diagram of the architecture of the code generation system provided in this application embodiment. To support an exemplary application, the code generation system 100 includes: a server 200, a network 300, and a terminal 400. The terminal 400 is connected to the server 200 via the network 300, which can be a wide area network (WAN), a local area network (LAN), or a combination of both, using wireless or wired links for data transmission.

[0063] Here, terminal 400 (e.g., a client running code generation that supports interface design images) responds to a code generation instruction for an application's interface design image by acquiring the application's interface design image, which includes design information of view components in the application's interface; performing view component detection on the interface design image to obtain the type and position of the view components in the interface design image; acquiring first view structure information of the interface design image; adjusting the first view structure information based on the type and position of the view components in the interface design image to obtain second view structure information; sending the second view structure information of the interface design image to server 200; server 200 receiving the second view structure information sent by terminal 400; generating executable code for the interface design image based on the second view structure information; returning the executable code for the interface design image to terminal 400; terminal 400 receiving the executable code for the interface design image returned by server 200; and upon receiving an execution instruction for the executable code, responding to the execution instruction by executing the executable code and generating the application interface corresponding to the interface design image.

[0064] The code generation method provided in this application is implemented by an electronic device. For example, it can be implemented by a terminal alone, by a server alone, or by a terminal and a server working together. The electronic device implementing the code generation method provided in this application can be various types of terminals or servers. The server (e.g., server 200) can be an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms. The terminal (e.g., terminal 400) can be a laptop, tablet, desktop computer, smartphone, smart voice interaction device (e.g., smart speaker), smart home appliance (e.g., smart TV), smartwatch, in-vehicle terminal, wearable device, virtual reality (VR) device, aircraft, etc., but is not limited to these. The terminal and server can be connected directly or indirectly through wired or wireless communication, and this application does not impose any restrictions on this.

[0065] In some embodiments, the terminal or server can implement the code generation method provided in this application by running various computer-executable instructions or computer programs. For example, computer-executable instructions can be microprogram-level commands, machine instructions, or software instructions. Computer programs can be native programs or software modules in an operating system; they can be native applications (APPs), i.e., programs that need to be installed in the operating system to run, such as a client that supports code generation for interface design images; or they can be applets that can be embedded in any APP, i.e., programs that only need to be downloaded to a browser environment to run. In summary, the aforementioned computer-executable instructions can be any form of instruction, and the aforementioned computer programs can be any form of application, module, or plugin.

[0066] The following describes an electronic device implementing the code generation method provided in an embodiment of this application. See also... Figure 2 , Figure 2 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. The electronic device 500 provided in this embodiment can be a terminal or a server. Figure 2 As shown, electronic device 500 includes at least one processor 510, memory 550, at least one network interface 520, and user interface 530. The various components in electronic device 500 are coupled together via a bus system 540. It is understood that the bus system 540 is used to implement communication between these components. In addition to a data bus, the bus system 540 also includes a power bus, a control bus, and a status signal bus. However, for clarity, in… Figure 2 The general labeled all buses as Bus System 540.

[0067] The processor 510 can be an integrated circuit chip with signal processing capabilities, such as a general-purpose processor, a digital signal processor (DSP), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or any conventional processor, etc.

[0068] User interface 530 includes one or more output devices 531 that enable the presentation of media content, including one or more speakers and / or one or more visual displays. User interface 530 also includes one or more input devices 532, including user interface components that facilitate user input, such as a keyboard, mouse, microphone, touch screen display, camera, other input buttons and controls.

[0069] Memory 550 may be removable, non-removable, or a combination thereof. Memory 550 may include one or more storage devices physically located away from processor 510. Memory 550 may include volatile memory or non-volatile memory, or both. Non-volatile memory may be read-only memory (ROM), and volatile memory may be random access memory (RAM). The memory 550 described in this application embodiment is intended to include any suitable type of memory.

[0070] In some embodiments, memory 550 is capable of storing data to support various operations, examples of which include programs, modules, and data structures or subsets or supersets thereof, as illustrated below.

[0071] Operating system 551 includes system programs for handling various basic system services and performing hardware-related tasks, such as the framework layer, core library layer, driver layer, etc., for implementing various basic business functions and handling hardware-based tasks;

[0072] The network communication module 552 is used to reach other electronic devices via one or more (wired or wireless) network interfaces 520, exemplary network interfaces 520 including: Bluetooth, WiFi, and Universal Serial Bus (USB), etc.

[0073] Presentation module 553 is configured to enable the presentation of information (e.g., a user interface for operating peripheral devices and displaying content and information) via one or more output devices 531 (e.g., a display screen, a speaker, etc.) associated with user interface 530;

[0074] The input processing module 554 is used to detect and translate one or more user inputs or interactions from one or more input devices 532.

[0075] In some embodiments, the code generation apparatus provided in this application can be implemented in software. Figure 2 A code generation device 555 stored in memory 550 is shown. It can be software in the form of programs and plug-ins, including the following software modules: a first acquisition module 5551, a detection module 5552, a second acquisition module 5553, an adjustment module 5554, and a generation module 5555. These modules are logically related and can therefore be arbitrarily combined or further divided according to the functions they implement. The functions of each module will be described below.

[0076] The code generation method provided in the embodiments of this application is described below. As mentioned above, the code generation method provided in the embodiments of this application is implemented by an electronic device, such as a server or terminal alone, or a server and terminal working together. Therefore, the executing entity of each step will not be described again below. See Figure 3A , Figure 3A This is a flowchart illustrating the code generation method provided in this application embodiment. The code generation method provided in this application embodiment includes:

[0077] Step 101: Obtain the interface design images of the application.

[0078] The interface design images include design information for the view components in the application's interface.

[0079] It should be noted that the code generation method provided in this application embodiment is used to: generate executable code corresponding to an application's interface design image based on the application's interface design image; and when the executable code is executed, generate the application interface corresponding to the interface design image. Therefore, in step 101, an interface design image to be generated is obtained, which corresponds to an application; this application can be any application, such as a mobile application (e.g., a smartphone application), a web application, a computer application, etc. The interface design image includes the design information of the view components in the application's interface.

[0080] Interface design images (also known as visual mockups) are graphical representations used to showcase and communicate design intent during software development. They typically serve as a bridge between the design and development teams, enabling developers without a design background to understand and implement the designer's ideas. Here are some characteristics and components of interface design images: 1) Characteristics: a) Graphical: Visual mockups are presented in image form, typically including visual elements such as layout, colors, fonts, and icons; b) Interactivity: Contains interactive elements such as buttons and links to demonstrate possible user interactions with the interface; c) Static or Dynamic: Can be static or dynamic images, the latter showcasing transitions and animations between interfaces. 2) Components: a) Layout: Shows the arrangement and partitioning of elements in the design, including the page header, footer, sidebar, and content areas. b) Color: Define the color scheme for the design, including primary color, secondary color, background color, etc.; c) Font and Typography: Specify the font type, size, line spacing, and paragraph spacing used in the design; d) Icons and Graphics: Include all icons, images, and graphic elements used in the design; e) View Components: User-interactive elements such as buttons, input boxes, and drop-down menus; f) States and Variations: Show the appearance of elements in different states, such as normal, hover, disabled, etc.; etc. See also: For an example, see Figure 4 , Figure 4 This is a schematic diagram of the interface design image provided in the embodiments of this application, which includes multiple view components.

[0081] View components are the basic building blocks of a user interface (UI), such as buttons, input boxes, selectors, labels, and images. They not only define the visual style of an element but may also include how it interacts with the user, such as clicking, swiping, and dragging. View components are designed to be reusable, allowing them to be reused across multiple interfaces or different projects, maintaining consistency and reducing development time. For example, the design information for a view component may include: 1) Style: including visual attributes such as color, font, size, margins, and padding; 2) Layout: defining the position and arrangement of the view component on the page; 3) Interaction: event detectors and handlers that determine how the view component responds to user input; 4) State: view components may contain internal state, which affects their appearance and behavior; 5) Content: including text, images, videos, and so on; and so on.

[0082] View components can be categorized in various ways, resulting in different component types. For example, based on their function, component types can include: 1) Basic components: such as text boxes, buttons, and switches; 2) Container components: used for grouping and laying out other components, such as panels, tables, and lists; 3) Navigation components: such as menus, tabs, and breadcrumbs, used for navigation and page switching; 4) Media components: such as images, videos, and audio, used to display multimedia content; and so on. For example, based on their structural characteristics, component types can include: 1) Modal components: such as virtual keyboards, dialog boxes, and panels; 2) Layout components: such as blocks, cards, status bars, action bars, and bottom navigation; 3) Element components: such as image views, text views, buttons, switches, checkboxes, and radio buttons.

[0083] In some embodiments, a code generation application that supports generating code for interface design images can be provided. When the code generation application is run, a code generation interface is displayed; based on the code generation interface, code generation instructions for the interface design images of the application are received. For example, the interface design image to be processed can be selected based on the code generation controls on the code generation interface, thereby triggering code generation instructions for the interface design image.

[0084] Step 102: Perform view component detection on the interface design image to obtain the type and position of the view components in the interface design image.

[0085] Here, after obtaining the interface design image, view component detection is performed on the interface design image to obtain the type (i.e., component type) and position of the view components in the interface design image. In some embodiments, view component detection can be implemented using an object detection model (such as YOLO series models, R-CNN series models, SSD model, RetinaNet model, EfficientDet model, etc., which are not limited here). When the object detection model is called to perform view component detection on the interface design image, object detection results can be obtained, which can include the type, position, and confidence score of the view component. Here, the position is used to indicate the position of the region box where the view component is located. For example, the coordinate information of this position can be (x, y, width, height), where (x, y) are the horizontal and vertical coordinates of a point in the region box (such as the center point of the region box, the top left corner point of the region box), width is the ratio of the width of the region box to the width of the interface design image, and height is the ratio of the height of the region box to the height of the interface design image. It should be noted that, in the embodiments of this application, the type of the view component is the detection type obtained by performing view component detection on the interface design image (such as by detecting through an object detection model), and the position is the detection position obtained by performing view component detection through an object detection model.

[0086] In some embodiments, there are multiple component types of view components in the interface design image; based on this, the view component detection of the interface design image can be performed by executing the following steps to obtain the type and position of the view components in the interface design image: obtain the target detection model corresponding to each component type; through each target detection model, perform view component detection on the interface design image of the application to obtain the type and position of the view components in the interface design image corresponding to each target detection model.

[0087] Here, we first obtain the object detection model corresponding to each component type. Different object detection models for different component types have varying detection accuracies for view components of different component types. That is, the object detection model for the first component type has higher accuracy for view components of the first component type than it does for view components of the second component type (different from the first component type). Each object detection model for each component type can be pre-trained. Specifically, we can perform the following processing for each component type's object detection model: construct training samples for that component type's object detection model. For example, the training samples can include general samples (applicable to training object detection models for each component type) and specific training samples for that component type; then, we train the object detection model for that component type based on the training samples, thus obtaining the object detection model for that component type. Based on this, after obtaining the object detection model for each component type, we use each object detection model to perform view component detection on the application's interface design image, obtaining the type and position of the view component in the interface design image corresponding to each object detection model. This improves the accuracy of view component detection, which in turn improves the accuracy of the subsequently generated second view structure information, resulting in more reasonable and accurate executable code generated based on the second view structure information.

[0088] Step 103: Obtain the first view structure information of the interface design image.

[0089] Here, we obtain the first view structure information of the interface design image. This first view structure information can be the initial view structure information of the interface design image. For example, interface design images are often implemented using interface design tools (tools that support vector graphics editors and prototyping). The view components represented by the first view structure information of the interface design image obtained by the interface design tool are all tiled and cannot describe the actual view structure of the view components in the interface design image (such as the layout of view components, the hierarchical relationship between view components, the component type of view components, etc.). For example, see [link to example]. Figure 5 , Figure 5 This is a schematic diagram of the first view structure information provided in an embodiment of this application. The interface design image is a grid view, but in the first view structure information, all view components are laid out flat and do not have a grid structure. If code is generated based on the first view structure information, the generated executable code will be too fragmented and lack good structure. Therefore, in this embodiment of the application, the first view structure information of the interface design image is first obtained to adjust the first view structure information.

[0090] Step 104: Based on the type and position of the view components in the interface design image, adjust the first view structure information to obtain the second view structure information.

[0091] Here, the first view structure information is adjusted. Specifically, based on the type and position of the view components in the interface design image, the first view structure information is adjusted to obtain the second view structure information. The second view structure information can include: the layout structure of each view component in the interface design (including the absolute position of the view components, the relative position between view components, the component size of the view components, the constraint relationships between view components (position constraints and size constraints), linear layout, relative layout, frame layout, grid layout, constraint layout, etc.), the hierarchical structure between view components (also called the hierarchical (containment) relationship between view components, such as top-level view components, middle-level view components, and bottom-level view components), the component type of the view components, etc. In practical applications, the second view structure information can also include auxiliary elements of the view components, such as resource files of the view components: such as images, fonts, colors, etc.; styles and themes: defining the appearance and style of UI elements, which can be applied to the entire application or specific views; animations: adding dynamic effects to the views; etc.

[0092] In some embodiments, the first view structure information is represented by a first view tree. The interface design image contains multiple view components, and the first view tree includes a first node corresponding to each view component; based on this, see [link to relevant documentation]. Figure 3B Step 104, "Adjusting the first view structure information based on the type and position of the view components in the interface design image to obtain the second view structure information," can be achieved by executing the following steps 1041-1044: Step 1041, generating a second view tree including multiple node levels based on the position of the view components; Step 1042, determining the target node level to which each first node belongs in the second view tree; Step 1043, determining the node constraint relationship of the first nodes in the target node level; Step 1044, adjusting the first view tree based on the target node level, node constraint relationship, and view component type to obtain a third view tree, which is used to represent the second view structure information.

[0093] The first view structure information is represented by a first view tree. The first view tree includes a first node corresponding to each view component. The first view tree can include at least one first node level, and each first node level includes at least one first node. For step 1041, since the first view structure information is represented by a first view tree, the detection results of the view components (including the position and type of the view components) can also be converted into a tree-like form. That is, based on the position of the view components, a second view tree including multiple node levels is generated. The second view tree includes multiple node levels, and the node level where the root node is located can be used as the first node level. Each node level includes at least one second node, and each second node also corresponds to a view component. Each second node also carries the type and position of the view component it corresponds to. Since the second view tree is constructed based on the detection results obtained from the detection of view components (i.e., the detection position or position of the view components), for a given view component, the node position (including the node level of the view tree where the node is located and the specific position of the node at the node level) of the node corresponding to that view component in the first view tree may be different from the node position (i.e., the second node) of the node corresponding to that view component in the second view tree. That is, the second view tree and the first view tree may be different.

[0094] For step 1042, after obtaining the second view tree, the following process is performed for each first node in the first view tree: from multiple node levels in the second view tree, the target node level to which the first node belongs in the second view tree is determined. Thus, the target node level to which each first node belongs in the second view tree is obtained. It can be understood that the number of target node levels is at least one.

[0095] For step 1043, the following processing is performed for each target node level determined in step 1042: First, determine the first node included in the target node level, which can also be denoted as the target first node; then, determine the node constraint relationships between the target first nodes. The node constraint relationships are the constraint relationships between the view components corresponding to the target first nodes. The constraint relationships between view components refer to the positional and dimensional constraints of the view components defined in the layout structure. These constraints are used to determine how the view components are positioned and resized relative to other view components or relative to the edges of the layout container.

[0096] For step 1044, based on the determined at least one target node level, the node constraint relationships between the target first nodes in each target node level, and the type of the view component corresponding to the first node, the first view tree is adjusted to obtain a third view tree. The third view tree is used to represent the second view structure information. Here, the third view tree includes multiple second node levels, each second node level includes at least one third node, and each third node corresponds to a view component. For example, when node 1 in the third view tree is the parent node of node 2 and node 3, it means that the view component corresponding to node 1 includes the view component corresponding to node 2 and the view component corresponding to node 3, and it also means that the view component corresponding to node 2 and the view component corresponding to node 3 are at the same level, and both the view component corresponding to node 2 and the view component corresponding to node 3 are at the next level below the view component corresponding to node 1.

[0097] Based on steps 1041-1044, the first view tree was adjusted according to the type and position of the view components, resulting in a third view tree that better represents the view structure information of the view components in the interface design image. This improved the accuracy of the view structure information of the interface design image, thereby improving the rationality and accuracy of the code generated based on the second view structure information.

[0098] In some embodiments, step 1041, "generating a second view tree including multiple node levels based on the position of the view component," can be achieved by performing the following steps: creating a second node corresponding to each view component; for each view component, determining the node level to which the second node corresponding to the view component belongs based on the position of the view component; and constructing a second view tree based on multiple second nodes and the node level to which each second node belongs.

[0099] Here, we first create a second node corresponding to each view component. Each second node also carries the type and position of the view component it corresponds to. Then, for each view component, based on its position, we determine the node hierarchy to which the second node corresponding to that view component belongs. Specifically, as mentioned above, the position of a view component indicates the position of the bounding box of the area where the view component is located. Therefore, we can determine the containment relationship between view components based on their positions. For example, view component 1 contains view components 2 and 3, view component 2 contains view component 4, view component 3 contains view components 5 and 6, and so on. Based on this, we can determine the node hierarchy to which the second node corresponding to each view component belongs based on the containment relationship between the view components. It can be understood that the hierarchy to which a view component belongs is also the node hierarchy to which the second node corresponding to that view component belongs. Finally, based on multiple second nodes and the node hierarchy to which each second node belongs, we construct a second view tree. For example, based on the node hierarchy to which each second node belongs, we construct a tree structure for multiple second nodes to obtain the second view tree. Based on this, the detection results obtained from the view component detection are converted into a tree structure, which facilitates the adjustment of the first view tree based on the second view tree and improves the accuracy of the adjustment.

[0100] In some embodiments, the location includes sub-locations obtained from multiple view component detections, where each sub-location indicates the bounding box of the view component; based on this, see [link to relevant documentation]. Figure 3C Step 1042, "Determine the target node level of each first node in the second view tree," can be achieved by performing steps 201-204 for each first node: Step 201, determine the first region box including the view component corresponding to the first node from multiple sub-location indicated region boxes; Step 202, when there are multiple first region boxes, determine the selection parameter value of each first region box, and select a second region box whose selection parameter value meets the parameter value condition from multiple first region boxes; Step 203, determine the target view component included in the second region box; Step 204, take the node level to which the second node corresponding to the target view component belongs as the target node level to which the first node belongs in the second view tree.

[0101] Here, as mentioned above, the location is obtained by multiple object detection models. Therefore, the location includes sub-locations obtained from multiple view component detections. Each sub-location indicates the bounding box of the view component. The explanation of the sub-locations can be found in the explanation of the location above, and will not be repeated here. Based on this, for step 201, a first bounding box including the view component corresponding to the first node is determined from the bounding boxes indicated by the multiple sub-locations. That is, the first bounding box includes the view component corresponding to the first node.

[0102] For step 202, when there are multiple first region boxes, the selection parameter value for each first region box is determined separately. In some embodiments, the selection parameter value for each first region box can be determined by performing the following steps: For each first region box, perform the following processing: obtain the region area of ​​the first region box, the component area of ​​the view component corresponding to the first node, and the orthogonal area of ​​the view component corresponding to the first node and the first region box; obtain the detection confidence of the sub-position corresponding to the first region box and the component parameter value of the view component corresponding to the first node; determine the selection parameter value of the first region box based on the region area, component area, orthogonal area, detection confidence, and component parameter value.

[0103] It should be noted that the detection confidence score is obtained by the target detection model when detecting sub-positions, with each sub-position corresponding to a detection confidence score. The component parameter value can be pre-set based on the component type of the view component; that is, for each component type, corresponding component parameter values ​​can be set. When obtaining the component parameter value of the view component corresponding to the first node, the target component type of the view component corresponding to the first node is first determined, and then the component parameter value corresponding to the target component type is used as the component parameter value of the view component corresponding to the first node. After obtaining the region area, component area, orthogonal area, detection confidence score, and component parameter value, the mapping relationship between the region area, component area, orthogonal area, detection confidence score, and component parameter value can also be obtained. Then, based on the region area, component area, orthogonal area, detection confidence score, and component parameter value, combined with this mapping relationship, the selection parameter value of the first region box is determined. For example, this mapping relationship can be as follows:

[0104]

[0105] Where score is the selected parameter value, crossArea is the orthogonal area, nodeArea is the component area, yoloArea is the region area, probability is the detection confidence, and extraFactor is the component parameter value.

[0106] In some embodiments, the area of ​​the region is determined based on the coordinate information (x, y, width, height) of the first region box. Therefore, when determining the area of ​​the region, scaling parameters of the coordinate information of the first region box can be obtained. These scaling parameters can be pre-set general scaling parameters or type scaling parameters set according to the component type of the view component corresponding to the first node (i.e., the first node whose selection parameter value is to be determined). No limitation is made here. For example, the general scaling parameter for width is 1.2, the general scaling parameter for x is 1.2, the general scaling parameter for height is 1.2, and the general scaling parameter for y is 1.2; the text type scaling parameter for width is 1.2, the text type scaling parameter for x is 2, the text type scaling parameter for height is 1.3, and the text type scaling parameter for y is 2; the image type scaling parameter for width is 1.1, the image type scaling parameter for x is 10, the image type scaling parameter for height is 1.1, and the image type scaling parameter for y is 5. After obtaining the scaling parameters of the coordinate information of the first region box, the coordinate information of the first region box is scaled using the scaling parameters to obtain the scaled coordinate information of the first region box. Then, based on the scaled coordinate information of the first region box, the area of ​​the first region box is determined.

[0107] Continuing with step 202, based on the selection parameter value of each first region box, a second region box whose selection parameter value meets the parameter value condition is selected from multiple first region boxes. This parameter value condition can be preset, for example, selecting the first region box with the largest parameter value as the second region box.

[0108] For step 203, determine the target view component included in the second region box. For step 204, take the node level to which the second node corresponding to the target view component belongs as the target node level to which the first node belongs in the second view tree.

[0109] Based on steps 201-204, by determining the corresponding selection parameter values ​​for each first region box, the second region box that best matches the first node is determined, thereby realizing the determination of the target node level, improving the accuracy of the target node level determination, and thus improving the accuracy of the second view structure information.

[0110] In some embodiments, based on the target node hierarchy, node constraint relationships, and view component types, the first view tree can be adjusted by performing the following steps to obtain a third view tree: the first view tree is adjusted based on the target node hierarchy and node constraint relationships to obtain an intermediate view tree; the view component types are added to the first node corresponding to the view component in the intermediate view tree to obtain the third view tree.

[0111] Here, the first view tree can be adjusted based on the target node hierarchy and node constraints to obtain an intermediate view tree. Specifically, the following processing can be performed for each target node hierarchy: create a node group corresponding to the target node hierarchy; for the first node in the target node hierarchy, place each first node in the node group sequentially according to the node constraints of the first node in the target node hierarchy, thus constructing each target node hierarchy in the intermediate view tree. Each constructed target node hierarchy is then combined according to the hierarchical relationship between the target node hierarchy levels to obtain the intermediate view tree. Finally, the type of the view component is added to the first node corresponding to the view component in the intermediate view tree to obtain the third view tree. In this way, the obtained third view tree can clearly represent the layout structure, hierarchical structure, and type of view components in the interface design image, thereby improving the accuracy of the second view structure information in describing the view components (specifically, the layout structure, hierarchical structure, and type of view components) in the interface design image, and thus improving the rationality and accuracy of the code generated based on the second view structure information.

[0112] In some embodiments, based on the target node hierarchy, node constraint relationships, and view component types, the first view tree can be adjusted by performing the following steps to obtain a third view tree: deleting the node view structure information of the first node in the first view tree to obtain an intermediate view tree; and adding new node view structure information to the first node in the intermediate view tree based on the target node hierarchy, node constraint relationships, and view component types to obtain a third view tree.

[0113] Here, when adjusting the first view tree, we can first delete the node view structure information of the first node in the first view tree to obtain the intermediate view tree. Then, based on the target node hierarchy, node constraint relationships, and view component types, we add new node view structure information to the first node in the intermediate view tree to obtain the third view tree. Specifically, we can perform the following processing for each target node hierarchy: create a node group corresponding to the target node hierarchy; for the first node in the target node hierarchy, according to the node constraint relationships of the first node in the target node hierarchy, place each first node in the node group in sequence, thus constructing each target node hierarchy. Combine each constructed target node hierarchy according to the hierarchical relationship between the target node hierarchy to obtain the target view tree. Finally, add the view component type to the first node corresponding to the view component in the target view tree to obtain the third view tree. In this way, unnecessary error information in the first view structure information can be deleted, improving the accuracy of the second view structure information in describing the view components (specifically the layout structure, hierarchy structure, and view component types) in the interface design image, thereby improving the rationality and accuracy of the code generated based on the second view structure information.

[0114] In some embodiments, before adjusting the first view tree based on the target node hierarchy, node constraint relationships, and view component types to obtain the third view tree, the following steps may be performed: determining the target first node in the first view tree that does not belong to the target node hierarchy; based on this, and based on the target node hierarchy and node constraint relationships, the first view tree may be adjusted by performing the following steps to obtain the third view tree: adjusting the first view tree based on the target node hierarchy, node constraint relationships, and view component types to obtain an intermediate view tree; adding the target first node to the intermediate view tree to obtain the third view tree.

[0115] Here, the first view tree may contain target first nodes that do not belong to the target node hierarchy. For target first nodes, after adjusting the first view tree based on the target node hierarchy, node constraints, and view component types to obtain an intermediate view tree, the target first node is added to the intermediate view tree to obtain the third view tree. For example, the target first node can be added to the intermediate view tree to represent a node or data structure of the background layer. In this way, the omission of first nodes in the first view tree can be avoided, further improving the accuracy of the second view structure information in describing the view components (specifically the layout structure, hierarchy, and type of view components) in the interface design image, thereby improving the rationality and accuracy of the code generated based on the second view structure information.

[0116] In some embodiments, the node constraint relationship of the first node in the target node hierarchy can be determined by performing the following steps: constructing the first node constraint relationship between the first nodes in the target node hierarchy; obtaining the second node constraint relationship set for the first node in the interface design image; and constructing the node constraint relationship of the first node in the target node hierarchy based on the first node constraint relationship and the second node constraint relationship.

[0117] Here, firstly, the first-node constraint relationships between the first nodes in the target node hierarchy are constructed. These first-node constraint relationships can include the constraints on the detection position and detection size of the components corresponding to the first nodes. Then, the second-node constraint relationships set for the first nodes in the interface design image are obtained. These second-node constraint relationships can include the pre-set constraints on the preset position and preset size of the components corresponding to the first nodes. Finally, based on the first-node constraint relationships and the second-node constraint relationships, the node constraint relationships of the first nodes in the target node hierarchy are constructed. Here, if the first-node constraint relationships and the second-node constraint relationships of the first nodes are different, then the first-node constraint relationship is retained as the node constraint relationship. If the first-node constraint relationships and the second-node constraint relationships of the first nodes are the same, then either the first-node constraint relationship or the second-node constraint relationship can be retained as the node constraint relationship.

[0118] Step 105: Based on the second view structure information, generate executable code for the interface design image.

[0119] When the executable code is executed, the application interface corresponding to the interface design image is generated.

[0120] Here, after obtaining the second view structure information, executable code for the interface design image is generated based on the second view structure information. When this executable code is executed, the application interface corresponding to the interface design image can be generated. In some embodiments, a preview control for the application interface can be provided. By triggering this preview control, the execution of the executable code is triggered, thereby displaying the application interface in the preview area. As an example, see [link to example]. Figure 6 , Figure 6 This is a schematic diagram of the executable code and application interface provided in an embodiment of this application. Here, the left side shows the executable code corresponding to the interface design image, which in turn represents the application interface corresponding to the interface design image generated in the preview area.

[0121] In some embodiments, based on the second view structure information, executable code for the interface design image can be generated by performing the following steps: generating component code for view components in the interface design image; generating a code framework based on the second view structure information; adding the component code to the code framework to obtain the executable code for the interface design image.

[0122] Here, firstly, component code is generated for each view component in the interface design image; then, a code framework is generated based on the second view structure information; finally, the component code is added to the code framework to obtain the executable code for the interface design image. In some embodiments, the executable code can be further optimized to obtain optimized executable code, thereby improving the quality of the executable code corresponding to the interface design image.

[0123] In some embodiments, the present application can be applied to code generation scenarios for interface design images of various applications, such as instant messaging clients, social clients, game clients, etc. For example, an interface design image of a social client is obtained, then view component detection is performed on the interface design image to obtain the type and position of the view components in the interface design image, and then the first view structure information of the interface design image is obtained. Based on the type and position of the view components in the interface design image, the first view structure information is adjusted to obtain second view structure information. Finally, based on the second view structure information, executable code for the interface design image is generated. When the executable code is executed, the application interface corresponding to the interface design image is generated. In this way, based on the type and position of the view components in the interface design image, the view structure information of the view components in the interface design image is automatically adjusted, thereby improving the efficiency and accuracy of code generation.

[0124] Applying the above embodiments of this application, firstly, an interface design image of the application is obtained. Then, view component detection is performed on the interface design image to obtain the type and position of the view components in the interface design image. Next, first view structure information of the interface design image is obtained. Based on the type and position of the view components in the interface design image, the first view structure information is adjusted to obtain second view structure information. Finally, based on the second view structure information, executable code for the interface design image is generated. When the executable code is executed, the application interface corresponding to the interface design image is generated. In this way, based on the type and position of the view components in the interface design image, the view structure information of the view components in the interface design image is automatically adjusted, thereby improving the efficiency and accuracy of code generation.

[0125] The following describes an exemplary application of the embodiments of this application in a real-world application scenario.

[0126] A visual draft (i.e., the interface design image mentioned above) is a file that annotates the layout and type of page elements. When using D2C for code generation, if the initial view structure of the interface designed by the interface design tool (i.e., the first view structure information mentioned above) is inconsistent with the expected view structure, it will lead to an unreasonable code structure generated by D2C. Figure 5 For example: Figure 5The selected area is a grid view, but in the initial view structure, all elements are laid out flat, lacking a grid structure. This results in the D2C-generated code being too fragmented and lacking good structure. Related technologies require manual intervention to repair and hierarchically divide the visual design, ensuring the generated code conforms to the expected development structure. However, this manual intervention increases labor costs; moreover, it requires direct manipulation of the visual design, demanding specific professional knowledge and skills from the operator, creating a high barrier to entry.

[0127] Based on this, this application provides a code generation method that automatically collects screenshots of application interfaces (such as mobile application interfaces) and combines manual cleaning and automatic annotation techniques to construct a training dataset specific to the application interface. The training dataset is then used to further train and optimize an object detection model (such as the YOLO model), enabling the object detection model to recognize and return the layout and element information of the application interface. An innovative scoring mechanism is used to integrate and process this information, achieving a more reasonable structural division and element type annotation of the visual design, thereby improving the automation and accuracy of interface design.

[0128] Through the embodiments of this application, the following can be achieved: 1) High accuracy: By specifically training and optimizing the object detection model, the accuracy of hierarchical division is improved, reducing the need for subsequent corrections and increasing speed. 2) Fast processing: The object detection model has a fast recognition speed, which greatly improves processing efficiency and shortens the conversion time from visual draft to code. 3) Full automation: The fully automated visual draft repair process reduces manual intervention and improves the quality and consistency of the generated code.

[0129] In some embodiments, the present application can be applied to the following scenarios: In the development process of an application (also called an application), after receiving a visual draft provided by a designer (such as the original visual draft obtained through an interface design tool (corresponding to the original view structure)), the developer can automatically optimize and repair the visual draft using the methods provided in the present application to obtain a repaired view draft (corresponding to the repaired view structure). The specific process may include key steps such as clearing invalid nodes, intelligent image merging, simplification and re-division of hierarchies, implementing adaptive layout of elements (i.e., the aforementioned view components), and performing semantic naming and component identification. This improves the structure and usability of the visual draft, ensuring that the code generated based on the repaired visual draft is more efficient and standardized. After the visual draft is repaired, the user can trigger an operation on the code generation control to quickly generate code suitable for various platforms, greatly simplifying the conversion process from visual draft to code and improving development efficiency and code quality. The specific interaction is as follows:

[0130] (1) Visual draft repair (corresponding to steps 101-104 above, i.e., the process of obtaining the second view structure information by adjusting the first view structure information). After the user selects the visual draft to be processed, they can trigger a fully automatic repair process for the visual draft through the provided visual draft repair controls. See here. Figure 7 The visual artwork repair controls include two different repair levels, such as "High-Quality Repair" and "Extreme-Speed ​​Repair." Different repair levels correspond to different visual artwork repair qualities and speeds. For example, "High-Quality Repair" provides higher quality visual artwork repair than "Extreme-Speed ​​Repair," and "Extreme-Speed ​​Repair" is faster than "High-Quality Repair," and so on.

[0131] See also Figure 7 ( Figure 7 This is a schematic diagram of the processing interface for a visual draft provided in this application embodiment. The repair process for the visual draft includes: preprocessing, intelligent grouping, intelligent layout, and post-processing. Preprocessing includes: sequentially performing invalid node cleaning, intelligent image merging, cropping optimization, and hierarchical simplification on the visual draft to obtain a preprocessed result; intelligent grouping includes: sequentially performing block recognition, fine content recognition, and grouping on the preprocessed result to obtain an intelligent grouping result; intelligent layout includes: sequentially performing adaptive layout processing, text width and height expansion, and fixed-size optimization on the intelligent grouping result to obtain an intelligent layout processing result; post-processing includes: sequentially performing semantic naming, component recognition, module recognition, and clickability recognition on the intelligent layout processing result to obtain the repaired visual draft.

[0132] (2) Visual draft repair successful. After successful visual draft repair, please refer to... Figure 4 It can display the repaired visual draft while retaining the original visual draft, and at the same time, it can also display the view structure information in the repaired visual draft.

[0133] See Figure 8 ( Figure 8 (This is a schematic diagram showing the correspondence between the visual draft and the code provided in the embodiments of this application). It can be determined that the node naming, structure, and type labeling of the repaired visual draft meet the expectations of the developers.

[0134] (3) Code Generation. After the visual design is repaired, the user can trigger the code generation control to quickly generate visual design code applicable to various platforms (such as Windows, macOS, Android, iOS, and other operating system platforms, as well as mobile, desktop, and embedded device platforms); see [link to relevant documentation]. Figure 6It can also display the application interface generated based on the visual design code for users to preview.

[0135] The embodiments of this application will now be described in detail. See also Figure 9 The embodiments of this application mainly involve the D2C plugin side and the code generation server side, wherein,

[0136] The D2C plugin reads the input visual draft and uses a pre-trained object detection model to identify the layout and element types. It also uses the layout information and element type labels output by the object detection model to regroup the original hierarchical structure of the visual draft using a scoring mechanism, and labels the elements that match the type labels output by the object detection model with the corresponding element type labels.

[0137] The code generation server is used to build the backend view tree based on the view tree reconstructed by the D2C plugin, and to perform original type marking on each node by combining the element type marking of the front-end object detection model; and to generate corresponding code for each node. After generation, AI is used to optimize the structure and readability of the code.

[0138] The training and optimization of the object detection model will be explained next. See [link / reference] Figure 10 The specific steps include: S11: Evaluate the required annotation type; S12: Collect the dataset, clean and label the dataset to obtain the training dataset; S13: Train and optimize the object detection model using the training dataset.

[0139] S11: Evaluate the required annotation types. For example, starting from the requirements, to construct a structured page, it's necessary to identify the specific type of the page structure and the specific type of each element. In some examples, based on structural characteristics, the following three types of annotations can be derived: (i) Modal structure annotations: keyboard, dialog, panel; (ii) Conventional layout structure annotations: block, card, status bar, action bar, bottom navigation, list; (iii) Element type annotations: imageView, textView, button, switch, checkbox, radioButton.

[0140] S12: Collect the dataset, clean and label it to obtain the training dataset. Here, the construction of three types of labeled datasets is completed through automatic and manual dataset collection.

[0141] S13: Model training and tuning.

[0142] S131: Methods for obtaining positive and negative samples.

[0143] For example, the training dataset constructed above and the open-source RICO dataset can be used as the final training data. The open-source RICO dataset contains screenshots of approximately 72,000 Android applications and their annotations. The training dataset constructed above, through automatic and manual collection, can cover component types in specific application scenarios. Each image in the training dataset can be manually annotated using the Label Studio tool to generate bounding boxes and type labels.

[0144] S132: The specific composition of positive and negative samples.

[0145] Positive samples consist of images containing the target object, and each positive sample comprises an image, bounding box coordinates, and a type label. For example, the annotation format for samples containing the target object in the training dataset can be as follows:

[0146]

[0147]

[0148] Negative samples include image regions that do not contain the target object; they typically contain only the image itself and no bounding boxes or type labels.

[0149] S133: Training process.

[0150] S1331: Data Preparation. Divide the above training data into training set, validation set, and test set. For example, 80% of the training data can be used as the training set, 10% as the validation set, and 10% as the test set.

[0151] S1332: Data preprocessing. Normalize and resize the images in the training set, convert the bounding box coordinates to proportional values ​​relative to the image size, and perform soft encoding of type labels.

[0152] S1333: Model Training. Construct an object detection model (e.g., using a YOLOv8x model) and initialize it with pre-trained weights. For example, use YOLO-specific loss functions, including localization loss, classification loss, and confidence loss. To reduce false background detections, improve the cross-entropy classification loss by replacing it with the Quality Focal Loss (QFL) function.

[0153] QFL(σ)=-|y-σ| β((1-y)log(1-y)+y log(σ));

[0154] Where σ represents the prediction result, y is the label multiplied by the corresponding IoU score, and β is a hyperparameter, typically set to 2. QFL() is a classification loss function that directly integrates the object localization quality (such as the overlap measure between the bounding box and the ground truth object, e.g., IoU score) into the classification loss, forming a joint representation. By assigning weights adjusted based on localization quality to each type of score, QFL() allows the object detection model to focus more on samples that are difficult to locate or classify during training, thereby reducing false detections and addressing the inconsistency between classification and localization tasks in object detection.

[0155] For the collected dataset, random variation strategies can be employed, such as color adjustment, translation, cropping, scaling, merging, and mosaic enhancement, to obtain more training data for model training and improve the model's generalization ability. During training, the Adam optimizer can be selected, with an initial learning rate of 0.001, decaying every 10 epochs. Input images, labels, and bounding boxes are used for forward propagation to calculate predictions. Then, the loss is calculated based on the predictions and type labels, and backpropagation is performed to update the model parameters. For example, at the end of every 3 epochs, a validation set can be used to evaluate model performance, and the training strategy can be adjusted based on the validation results.

[0156] S1334: Model Tuning. Model hyperparameters, including learning rate, batch size, and regularization parameters, were tuned using Bayesian optimization methods. Model weights pre-trained on the RICO dataset were used, and fine-tuned on a self-collected dataset.

[0157] S1335: Model Evaluation. Evaluate model performance using metrics such as mean AP, precision, and recall. Plot a confusion matrix to analyze the model's predictive performance across different data types.

[0158] Background false positive correlation analysis: such as Figure 11 ( Figure 11 As shown in the schematic diagram of the analysis of the detection effect of the target detection model provided in the embodiment of this application, after adding the QFL loss function, the accuracy detection rate is improved, and the positional bias, type false detection, and background false detection are all reduced.

[0159] After obtaining the trained object detection model, it is used to perform object detection on the visual draft, obtaining the detection results. Based on these results, the third-view tree of the visual draft (i.e., the view tree after visual draft repair) is reconstructed. The reconstruction of the third-view tree will be explained below. See [link to documentation]. Figure 12 The steps include:

[0160] S21: Construct the YOLO view tree (i.e., the second view tree mentioned above).

[0161] Here, the detection results from multiple object detection models are sequentially transformed into a matrix and then into a tree structure, resulting in a detection result in a tree structure, i.e., the Yolo view tree. This Yolo view tree is used to match the first view tree of the visual draft (i.e., the view tree corresponding to the initial view structure mentioned above). The root node of this second view tree is the structure of YoloSchema. For example, key fields of YoloSchema may include: 1) type: Represents the type of the region selected by the detection box output by the object detection model, such as layout type List, element type Button, etc. `type` is used to infer the type of nodes in the second view tree, ensuring accurate code generation. 2) rect: Contains the coordinate information (x, y, width, height) of the detection box (i.e., the region box mentioned above), used for matching with nodes in the first view tree. 3) probability: Represents the confidence level of the detection box output by the object detection model, which will participate in the matching score during node matching (i.e., determining the value of the selected parameter mentioned above). 4) children: Child nodes of the root node in the second view tree. An example of YoloSchema is as follows:

[0162]

[0163] S22: Construct a new node hierarchy structure for the nodes in the first view tree (i.e., the first node mentioned above).

[0164] S221: Establish a scoring mechanism for different component types.

[0165] Here, before matching nodes in the first view tree with each YOLO Schema, different matching scaling parameters are first set for the first node of different component types. For example, the matching scaling parameters can be: 1.2 for width, 1.2 for x, 1.2 for height, and 1.2 for y.

[0166] For example, a) General matching scaling parameters const COMMON_TRANSFORM:Transform = {widthScale:1.2; widthOffset (i.e., x):2; heightScale:1.2; heightOffset (i.e., y):2}.

[0167] b) Text matching scaling parameters const TEXT_TRANSFORM:Transform={widthScale:1.2;widthOffset(i.e. x):2;heightScale:1.3;heightOffset(i.e. y):2}.

[0168] c) Image matching scaling parameters const IMAGE_TRANSFORM:Transform={widthScale:1.1;widthOffset(i.e. x):10;heightScale:1.1;heightOffset(i.e. y):5}.

[0169] S222: Find multiple YOLOSchemas that contain the first node.

[0170] S223: Score each YOLOSchema and select the YOLOSchema that best matches the first node.

[0171] Specifically, 1) Traverse the first view tree, and for the current first node, perform a Rect Transform on the matching YOLO Schema to find all YOLO Schemas that can accommodate the current first node. For example... Figure 13 ( Figure 13 (This is a schematic diagram of the second region box to which the view component belongs, provided in an embodiment of this application.) Figure 13 Each detection box represents an independent YOLOSchema. Both YOLOSchema1 and YOLOSchema2 contain TextNodes, so the schemas that can currently accommodate TextNodes are [YOLOSchema1, YOLOSchema2].

[0172] 2) Score all YoloSchemas that match the current first node, and select the YoloSchema with the highest score as the best matching YoloSchema for the current first node.

[0173] The scoring calculation rules are as follows:

[0174]

[0175] Wherein, score is the selected parameter value (i.e., the score), crossArea is the orthogonal area (i.e., the orthogonal area between the current first node and the yoloSchema), nodeArea is the component area (i.e., the area of ​​the component corresponding to the first node), yoloArea is the region area (i.e., the area of ​​the yoloSchema), prossibility is the detection confidence, and extraFactor is the component parameter value (i.e., the area score specific to a particular component type, such as a text control, which requires an additional area score).

[0176] After the score is calculated, the level corresponding to the highest-scoring YoloSchema is selected as the target node level of the current first node.

[0177] See also Figure 13 The first node is currently TextNode. Assuming that the probability and extraFactor are the same, the orthogonal areas of YOLOSchema1 and YOLOSchema2 are the same, but YOLOArea YOLOScheme1 will be larger. Therefore, the most suitable node for TextNode is YOLOSchema1.

[0178] Each first node can only match one YOLOSchema, and each YOLOSchema can match multiple first nodes. A YOLOSchema represents a new node level. After matching is complete, the construction of a new view tree (i.e., the third view tree) begins.

[0179] S23: Regroup the same node hierarchy.

[0180] S231: Create a new group and incorporate it into the first view tree.

[0181] S232: Establish layout constraint information for nodes within the new group.

[0182] S24: New view tree element type marker.

[0183] S25: Missing elements are merged into the background layer.

[0184] The implementation process of S23-S25 above includes: traversing the YoloSchema view tree and finding the first node array that matches the current Yolo node; creating a new group for the first node array and adding constraint relationships between nodes in the new group; processing the remaining unmatched first nodes, merging them into the background layer, and completing the creation of the new view tree. For details on the specific implementation process of S23-S25 above, please refer to [link to documentation]. Figure 14 ,include:

[0185] 1. Traverse the YOLO view tree; 2. Find the first matching node in the current YOLO schema; 3. If a matching first node exists, proceed to step 5; otherwise, proceed to step 4; 4. Discard the current YOLO result; 5. Remove the view structure information of the first node in the first view tree; 6. Create a new group for the first node in the same YOLO schema; 7. Establish node constraints under the new group; 8. Incorporate the new group into the first view tree; 9. If an unmatched first node exists, proceed to step 10; otherwise, proceed to step 11; 10. Merge the unmatched first node into the background layer; 11. Complete the reconstruction of the new view tree.

[0186] S26: Complete the reconstruction of the new view tree (i.e., the third view tree mentioned above).

[0187] like Figure 15 ( Figure 15 As shown in the schematic diagram of the view structure information provided in the embodiment of this application, for the view draft, the original view structure is scattered, the text hierarchy is incorrect, and the type mark is missing; after the repair, each node in the view structure has a type mark, and the hierarchy is reasonable.

[0188] By applying the embodiments described above in this application, and through specialized training and optimization of the YOLO model, the accuracy of visual manuscript hierarchical division is significantly improved, and an automatic node type annotation function is added. Specifically, 1) Improved accuracy: Compared with the traditional manual hierarchical division method, this application significantly improves the division accuracy. In traditional methods, manual layer-by-layer review is prone to omissions and errors, while the automated method of this application can quickly identify and correct all problem levels, reducing human error. 2) Accelerated processing speed: This application also demonstrates a significant advantage in processing speed. Traditionally, manually processing a visual manuscript takes several minutes, while this application, utilizing the automatic identification and repair function of the YOLO model, can complete the same task in seconds, greatly improving efficiency. 3) Enhanced code generation accuracy: In addition to hierarchical division and repair, this application also performs automatic type annotation on visual manuscript node elements. These type tags provide assistance in the code generation stage, further improving the accuracy and quality of generated code.

[0189] The following description continues to illustrate the exemplary structure of the code generation apparatus 555 provided in the embodiments of this application as a software module. In some embodiments, such as... Figure 2As shown, the software modules stored in the code generation device 555 of the memory 550 may include: a first acquisition module 5551, used to acquire an interface design image of an application, the interface design image including design information of view components in the application interface of the application; a detection module 5552, used to perform view component detection on the interface design image to obtain the type and position of the view components in the interface design image; a second acquisition module 5553, used to acquire first view structure information of the interface design image; an adjustment module 5554, used to adjust the first view structure information based on the type and position of the view components in the interface design image to obtain second view structure information; and a generation module 5555, used to generate executable code of the interface design image based on the second view structure information, wherein, when the executable code is executed, an application interface corresponding to the interface design image is generated.

[0190] In some embodiments, the view components in the interface design image have multiple component types; the detection module 5552 is further configured to obtain a target detection model corresponding to each component type; and to perform view component detection on the interface design image of the application through each target detection model to obtain the type and position of the view components in the interface design image corresponding to each target detection model.

[0191] In some embodiments, the first view structure information is represented by a first view tree, the interface design image contains multiple view components, and the first view tree includes a first node corresponding to each view component; the adjustment module 5554 is further configured to generate a second view tree including multiple node levels based on the position of the view components; determine the target node level to which each first node belongs in the second view tree; determine the node constraint relationship of the first node in the target node level; and adjust the first view tree based on the target node level, the node constraint relationship, and the type of the view component to obtain a third view tree, the third view tree being used to represent the second view structure information.

[0192] In some embodiments, the adjustment module 5554 is further configured to create a second node corresponding to each of the view components; for each view component, determine the node level to which the second node corresponding to the view component belongs based on the position of the view component; and construct the second view tree based on multiple second nodes and the node level to which each second node belongs.

[0193] In some embodiments, the location includes sub-locations obtained from multiple view component detections, the sub-locations indicating the region box where the view component is located; the adjustment module 5554 is further configured to perform the following processing for each first node: determine a first region box including the view component corresponding to the first node from the region boxes indicated by the multiple sub-locations; when there are multiple first region boxes, determine the selection parameter value of each first region box, and select a second region box whose selection parameter value meets the parameter value condition from the multiple first region boxes; determine the target view component included in the second region box; and take the node level to which the second node corresponding to the target view component belongs as the target node level to which the first node belongs in the second view tree.

[0194] In some embodiments, the adjustment module 5554 is further configured to perform the following processing for each of the first region boxes: obtain the region area of ​​the first region box, the component area of ​​the view component corresponding to the first node, and the orthogonal area of ​​the view component corresponding to the first node and the first region box; obtain the detection confidence of the sub-position corresponding to the first region box and the component parameter value of the view component corresponding to the first node; and determine the selection parameter value of the first region box based on the region area, the component area, the orthogonal area, the detection confidence, and the component parameter value.

[0195] In some embodiments, the adjustment module 5554 is further configured to adjust the first view tree based on the target node hierarchy and the node constraint relationship to obtain an intermediate view tree; and add the type of the view component to the first node corresponding to the view component in the intermediate view tree to obtain the third view tree.

[0196] In some embodiments, the adjustment module 5554 is further configured to delete the node view structure information of the first node in the first view tree to obtain an intermediate view tree; and to add new node view structure information to the first node in the intermediate view tree based on the target node level, the node constraint relationship and the type of the view component to obtain the third view tree.

[0197] In some embodiments, the adjustment module 5554 is further configured to, before adjusting the first view tree based on the target node hierarchy, the node constraint relationship, and the type of the view component to obtain the third view tree, determine a target first node in the first view tree that does not belong to the target node hierarchy; the adjustment module 5554 is further configured to, based on the target node hierarchy, the node constraint relationship, and the type of the view component, adjust the first view tree to obtain an intermediate view tree; and add the target first node to the intermediate view tree to obtain the third view tree.

[0198] In some embodiments, the adjustment module 5554 is further configured to determine the first node constraint relationship between the first nodes in the target node hierarchy based on the position of the view component corresponding to the first node; obtain the second node constraint relationship set for the first node in the interface design image; and construct the node constraint relationship of the first node in the target node hierarchy based on the first node constraint relationship and the second node constraint relationship.

[0199] In some embodiments, the generation module 5555 is further configured to generate component code for view components in the interface design image; generate a code framework based on the second view structure information; and add the component code to the code framework to obtain executable code for the interface design image.

[0200] It should be noted that the description of the apparatus embodiments in this application is similar to the description of the method embodiments described above, and has similar beneficial effects as the method embodiments, so it will not be repeated here. Any technical details not covered in the code generation apparatus provided in the embodiments of this application can be understood based on the description of the technical details in the above method embodiments.

[0201] This application also provides a computer program product, which includes computer-executable instructions or a computer program stored in a computer-readable storage medium. The processor of an electronic device reads the computer-executable instructions or computer program from the computer-readable storage medium and executes the computer-executable instructions or computer program, causing the electronic device to perform the code generation method provided in this application.

[0202] This application also provides a computer-readable storage medium storing computer-executable instructions or computer programs. When the computer-executable instructions or computer programs are executed by a processor, the processor will execute the code generation method provided in this application.

[0203] In some embodiments, the computer-readable storage medium may be a memory such as RAM, ROM, flash memory, magnetic surface memory, optical disk, or CD-ROM; or it may be a variety of devices including one or any combination of the above-mentioned memories.

[0204] In some embodiments, computer-executable instructions may take the form of programs, software, software modules, scripts, or code, written in any form of programming language (including compiled or interpreted languages, or declarative or procedural languages), and may be deployed in any form, including as stand-alone programs or as modules, components, subroutines, or other units suitable for use in a computing environment.

[0205] As an example, computer-executable instructions may, but do not necessarily, correspond to files in a file system. They may be stored as part of a file that holds other programs or data, for example, in one or more scripts in a Hyper Text Markup Language (HTML) document, in a single file dedicated to the program in question, or in multiple co-located files (e.g., files that store one or more modules, subroutines, or code sections).

[0206] As an example, computer-executable instructions can be deployed to execute on a single electronic device, or on multiple electronic devices located at one location, or on multiple electronic devices distributed across multiple locations and interconnected via a communication network.

[0207] The above description is merely an embodiment of this application and is not intended to limit the scope of protection of this application. Any modifications, equivalent substitutions, and improvements made within the spirit and scope of this application are included within the scope of protection of this application.

Claims

1. A code generation method, characterized in that, The method includes: Obtain the interface design image of the application, wherein the interface design image includes the design information of the view components in the application interface of the application; Perform view component detection on the interface design image to obtain the type and position of the view components in the interface design image; Obtain the first view structure information of the interface design image; Based on the type and position of the view components in the interface design image, the first view structure information is adjusted to obtain the second view structure information; Based on the second view structure information, executable code for the interface design image is generated, wherein when the executable code is executed, the application interface corresponding to the interface design image is generated.

2. The method as described in claim 1, characterized in that, The interface design image contains various types of view components; the process of detecting view components in the interface design image to obtain the type and position of the view components includes: Obtain the target detection model corresponding to each component type; By using each of the target detection models, view component detection is performed on the interface design image of the application to obtain the type and position of the view component in the interface design image corresponding to each target detection model.

3. The method as described in claim 1, characterized in that, The first view structure information is represented by a first view tree. The interface design image contains multiple view components, and the first view tree includes a first node corresponding to each view component. The adjustment of the first view structure information based on the type and position of the view components in the interface design image to obtain the second view structure information includes: Based on the position of the view component, a second view tree including multiple node levels is generated; Determine the target node level to which each of the first nodes belongs in the second view tree; Determine the node constraint relationships of the first node in the target node hierarchy; Based on the target node hierarchy, the node constraint relationship, and the type of the view component, the first view tree is adjusted to obtain a third view tree, which is used to represent the second view structure information.

4. The method as described in claim 3, characterized in that, The generation of a second view tree, comprising multiple node levels, based on the position of the view component includes: Create a second node corresponding to each of the aforementioned view components; For each view component, the node level to which the second node corresponding to the view component belongs is determined based on the position of the view component; The second view tree is constructed based on multiple second nodes and the node hierarchy to which each second node belongs.

5. The method as described in claim 4, characterized in that, The location includes sub-locations obtained from multiple view component detections, and the sub-locations indicate the bounding box of the view component. Determining the target node level to which each of the first nodes belongs in the second view tree includes: For each of the first nodes, perform the following processing: From the region boxes indicating the multiple sub-locations, a first region box including the view component corresponding to the first node is determined; When there are multiple first region boxes, determine the selection parameter value for each first region box, and select a second region box whose selection parameter value meets the parameter value condition from the multiple first region boxes; Determine the target view components included in the second region box; The node level to which the second node corresponding to the target view component belongs is taken as the target node level to which the first node belongs in the second view tree.

6. The method as described in claim 5, characterized in that, Determining the selection parameter value for each of the first region boxes includes: For each of the first region boxes, perform the following processing: Obtain the area of ​​the first region box, the area of ​​the view component corresponding to the first node, and the orthogonal area of ​​the view component corresponding to the first node and the first region box; Obtain the detection confidence of the sub-position corresponding to the first region box, and the component parameter value of the view component corresponding to the first node; The selection parameter value of the first region box is determined based on the region area, the component area, the orthogonal area, the detection confidence level, and the component parameter value.

7. The method as described in claim 3, characterized in that, The step of adjusting the first view tree based on the target node hierarchy, the node constraint relationships, and the view component type to obtain the third view tree includes: Based on the target node hierarchy and the node constraint relationships, the first view tree is adjusted to obtain an intermediate view tree; The type of the view component is added to the first node corresponding to the view component in the intermediate view tree to obtain the third view tree.

8. The method as described in claim 3, characterized in that, The step of adjusting the first view tree based on the target node hierarchy, the node constraint relationships, and the view component type to obtain the third view tree includes: Delete the node view structure information of the first node in the first view tree to obtain the intermediate view tree; Based on the target node hierarchy, the node constraint relationship, and the type of the view component, new node view structure information is added to the first node in the intermediate view tree to obtain the third view tree.

9. The method as described in claim 3, characterized in that, Before adjusting the first view tree based on the target node hierarchy, the node constraint relationships, and the view component type to obtain the third view tree, the method further includes: Determine the first target node in the first view tree that does not belong to the target node level; The step of adjusting the first view tree based on the target node hierarchy and the node constraint relationships to obtain the third view tree includes: Based on the target node hierarchy, the node constraint relationships, and the type of the view component, the first view tree is adjusted to obtain an intermediate view tree; The target first node is added to the intermediate view tree to obtain the third view tree.

10. The method as described in claim 3, characterized in that, Determining the node constraint relationship of the first node in the target node hierarchy includes: Construct the first-node constraint relationships between the first nodes in the target node hierarchy; Obtain the second node constraint relationship set for the first node in the interface design image; Based on the first node constraint relationship and the second node constraint relationship, the node constraint relationship of the first node in the target node hierarchy is constructed.

11. The method as described in claim 1, characterized in that, The executable code for generating the interface design image based on the second view structure information includes: Generate the component code for the view components in the interface design image; Based on the second view structure information, a code framework is generated; Add the component code to the code framework to obtain the executable code for the interface design image.

12. A code generation device, characterized in that, The device includes: The first acquisition module is used to acquire the interface design image of the application, the interface design image including the design information of the view components in the application interface of the application; The detection module is used to detect view components in the interface design image to obtain the type and position of the view components in the interface design image. The second acquisition module is used to acquire the first view structure information of the interface design image; The adjustment module is used to adjust the first view structure information based on the type and position of the view components in the interface design image to obtain the second view structure information; The generation module is used to generate executable code for the interface design image based on the second view structure information, wherein when the executable code is executed, the application interface corresponding to the interface design image is generated.

13. An electronic device, characterized in that, The electronic device includes: Memory is used to store executable instructions for a computer; A processor, when executing computer-executable instructions stored in the memory, implements the code generation method according to any one of claims 1 to 11.

14. A computer-readable storage medium storing computer-executable instructions or a computer program, characterized in that, When the computer-executable instructions or computer program are executed by a processor, the code generation method according to any one of claims 1 to 11 is implemented.

15. A computer program product comprising computer-executable instructions or a computer program, characterized in that, When the computer-executable instructions or computer program are executed by a processor, the code generation method according to any one of claims 1 to 11 is implemented.

Images