Code generation and model training method, computing device, storage medium and product

Abstract

Embodiments of the present application provide a code generation and model training method, a computing device, a storage medium and a product. The code generation process includes generating an initial interface code based on an interface visual file, iteratively performing at least one round of code update operation on the initial interface code until a first update requirement is met to obtain a target interface code, the code update operation including rendering the current round of interface code to be processed to generate a candidate rendering image, and the first round of interface code to be processed being the initial interface code. The difference information between the reference rendering image corresponding to the interface visual file and the candidate rendering image is determined, and based on the difference information, the current round of interface code to be processed is updated using a code processing model to generate the next round of interface code to be processed. The technical scheme provided by the embodiments of the present application can improve the generation efficiency and accuracy of the interface code.

Description

Technical Field

[0001] This application relates to the field of computer technology, and in particular to a code generation and model training method, computing device, storage medium and product. Background Technology

[0002] In application development, the user interface, as the direct medium for user interaction with the application system, plays a crucial role in the application's market competitiveness and user satisfaction. Therefore, it is necessary to carefully generate corresponding interface code for the user interface.

[0003] Currently, the UI code generation process is typically a collaborative effort between UI designers and software engineers. For example, UI designers first use design tools to create high-fidelity UI visual files (such as UI drafts) for the user interface. Then, software engineers manually convert these visual files into UI code that can run on the front end. However, this method requires software engineers to manually convert visual files to UI code, resulting in low UI code development efficiency. Furthermore, due to potential misunderstandings of the visual files by software engineers and the technical difficulties involved, the accuracy of the manually converted UI code is hard to guarantee. Summary of the Invention

[0004] This application provides a code generation and model training method, computing device, storage medium, and product to solve the problems of low efficiency and difficulty in guaranteeing accuracy of interface code generation in the prior art.

[0005] Firstly, this application provides a code generation method, including: Generate initial interface code based on the interface visual file; Iteratively execute at least one round of code update operations until the first update requirement is met to obtain the target interface code; the code update operations include: The current round of pending interface code is rendered to generate candidate rendering images; the first round of pending interface code is the initial interface code. Determine the differences between the reference rendering image corresponding to the interface visual file and the candidate rendering image; Based on the difference information, the interface code is updated using a code processing model to generate the next round of interface code to be processed.

[0006] Secondly, embodiments of this application provide a model training method, including: Acquire multiple first training data; the first training data includes first sample interface code and the first sample reference rendering image corresponding to the first sample interface code; For any given first training data, perform at least one code perturbation operation on the first sample interface code to obtain perturbed interface code, and generate recovery guidance information based on the at least one code perturbation operation; The disturbed interface code is rendered to generate a disturbed rendering image; Determine the first sample difference information between the first sample reference rendering image and the perturbation rendering image; Using the perturbation interface code, the first sample difference information, the recovery guidance information, and the first sample interface code, a code processing model is trained. The code processing model is used to iteratively execute at least one round of code update operations on the initial interface code corresponding to the interface visual file until the target interface code is obtained. The code update operation is used to update the interface code to be processed in the current round based on the difference information between the reference rendering image and the candidate rendering image corresponding to the interface visual file, so as to generate the interface code to be processed in the next round. The candidate rendering image is obtained by rendering the interface code to be processed in the current round.

[0007] Thirdly, embodiments of this application provide a model training method, including: Acquire multiple second training data sets; the second training data sets include second sample reference rendering images and second sample intermediate representation data. For any given set of second training data, based on the second sample parameter rendering map and / or the second sample intermediate representation data, initial candidate interface code is generated, and at least one round of sample code update operations is iteratively executed until the second update requirement is met. The sample code update operations include: The candidate interface code to be processed in the current round is rendered to generate a candidate sample rendering image; the candidate interface code to be processed in the first round is the initial candidate interface code; Determine the second sample difference information between the second sample reference rendering image and the candidate sample rendering image; Based on the second sample difference information, the candidate interface code is updated using a code processing model to generate the next round of candidate interface code to be processed. Based on the second sample difference information corresponding to the sample code update operation satisfying the second update requirement, reward information is determined, and the model parameters of the code processing model are adjusted according to the reward information. The code processing model is used to iteratively execute at least one round of code update operations on the initial interface code corresponding to the interface visual file until the target interface code is obtained. The code update operation is used to update the interface code to be processed in the current round based on the difference information between the reference rendering image and the candidate rendering image corresponding to the interface visual file, so as to generate the interface code to be processed in the next round. The candidate rendering image is obtained by rendering the interface code to be processed in the current round.

[0008] Fourthly, this application provides a computing device, including a processing component and a storage component; the storage component stores a computing program; the computer program is used to be called and executed by the processing component to implement the code generation method of the first aspect above, or to implement the model training method of the second or third aspect above.

[0009] Fifthly, this application provides a computer storage medium storing a computer program thereon. When the computer program is executed by a processing component, it implements the code generation method of the first aspect described above, or implements the model training method of the second or third aspect described above.

[0010] Sixthly, this application provides a computer program product, including a computer program or instructions, which, when executed by a processing component, implement the code generation method of the first aspect described above, or implement the model training method of the second or third aspect described above.

[0011] This application's embodiments generate initial interface code based on an interface visual file. At least one round of code update operations is performed on this initial interface code until a first update requirement is met to obtain the target interface code. The code update operation includes rendering the interface code to be processed in the current round to generate a candidate rendering image, where the interface code to be processed in the first round is the initial interface code; determining the difference information between the reference rendering image corresponding to the interface visual file and the candidate rendering image; and updating the interface code to be processed in the current round using a code processing model based on the difference information to generate the interface code to be processed in the next round. This application's embodiments propose a multi-round iterative interface code generation method based on a code processing model, which greatly improves code generation efficiency compared to manual generation. Furthermore, this embodiment no longer pursues generating accurate interface code in a single step. Instead, it simulates a debugging approach, first generating initial interface code, then comparing the candidate rendering images of the generated interface code with the corresponding reference rendering images of the interface visual file. After identifying differences, the generated interface code is updated in a targeted manner. Through multiple rounds of iterative updates, the candidate rendering images of the generated interface code gradually approach the reference rendering images of the interface visual file, significantly improving the accuracy of interface code generation. This process does not require excessive reliance on the standardization of visual files and is suitable for real-world development environments with complex interface visual files, providing a new solution for interface code generation that balances accuracy and efficiency.

[0012] These or other aspects of this application will become more apparent in the following description of the embodiments. Attached Figure Description

[0013] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings: Figure 1 A flowchart of one embodiment of the code generation method provided in this application is shown; Figure 2 This application provides a schematic diagram of the code processing model. Figure 3 A flowchart of an embodiment of the model training method provided in this application is shown; Figure 4 A flowchart of another embodiment of the model training method provided in this application is shown; Figure 5 A schematic diagram illustrating the code generation process in a real-world application scenario provided in this application is shown; Figure 6 A schematic diagram illustrating the model training process in a real-world application scenario provided in this application is shown. Figure 7 A schematic diagram of the structure of one embodiment of the code generation apparatus provided in this application is shown; Figure 8 A schematic diagram of the structure of one embodiment of the model training apparatus provided in this application is shown; Figure 9 A schematic diagram of another embodiment of the model training apparatus provided in this application is shown; Figure 10 A schematic diagram of the structure of the computing device provided in this application is shown. Detailed Implementation

[0014] To make the objectives, technical solutions, and advantages of this application clearer, the technical solutions of this application will be clearly and completely described below in conjunction with specific embodiments and corresponding drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of them. Based on the embodiments in this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0015] It should be noted that, in the cases involving user information in the embodiments of this application, the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in the embodiments of this application are all information and data authorized by the user or fully authorized by all parties. Furthermore, the collection, use, and processing of related data must comply with the relevant laws, regulations, and standards of the relevant countries and regions, and corresponding operation entry points are provided for users to choose to authorize or refuse. In addition, the various models involved in this application (including but not limited to language models or large models) comply with relevant laws and standards.

[0016] As described in the background section, the current interface code generation process, which is collaboratively completed by interface designers and software engineers, suffers from low efficiency and difficulty in guaranteeing accuracy due to manual operation. To address this, the inventors conceived of an automatic conversion of the interface visual files designed by interface designers into interface code, thereby reducing the impact of manual operation on efficiency and accuracy.

[0017] For example, one approach is to first extract structured intermediate expression (IR) data from the interface visual file, and then use a rule engine or a general large model to convert it into interface code. This approach heavily relies on the completeness and logical accuracy of the intermediate expression data. For complex interface visual files (such as those containing complex nesting, custom components, or non-standard layouts), the intermediate expression data often cannot accurately reproduce the interface visual file, thus affecting the accuracy of the interface code converted from it.

[0018] Another approach is to directly input the interface visual file into an AI model, using the model's image recognition technology to convert the visual file into interface code. However, this method suffers from inaccurate recognition and poor generalization ability when dealing with complex interface visual files in real-world applications, resulting in inaccurate converted interface code.

[0019] To address the aforementioned problems, the inventors, after a series of studies, proposed the technical solution of this application. The basic idea is as follows: generate initial interface code based on the interface visual file, iteratively perform at least one round of code update operations on the initial interface code until the first update requirement is met to obtain the target interface code. The code update operation includes rendering the interface code to be processed in the current round to generate a candidate rendering image, where the interface code to be processed in the first round is the initial interface code; determine the difference information between the reference rendering image corresponding to the interface visual file and the candidate rendering image; and based on the difference information, update the interface code to be processed in the current round using a code processing model to generate the interface code to be processed in the next round.

[0020] This embodiment proposes a multi-round iterative method for generating interface code based on a code processing model, which significantly improves code generation efficiency compared to manual generation. Furthermore, this embodiment does not aim for accurate interface code generation in a single iteration. Instead, it simulates a debugging approach, first generating initial interface code, then comparing the candidate rendering images of the generated interface code with the corresponding reference rendering images in the interface visual file. After identifying differences, the generated interface code is updated accordingly. Through multiple rounds of iterative updates, the candidate rendering images of the generated interface code gradually approach the reference rendering images in the interface visual file, significantly improving the accuracy of interface code generation. This process does not rely excessively on the standardization of visual files and is suitable for real-world development environments with complex interface visual files, providing a new solution for interface code generation that balances accuracy and efficiency.

[0021] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0022] Figure 1 The flowchart illustrates an embodiment of a code generation method provided in this application. The technical solution of this embodiment can be executed by a processing end, which can be a server in an online system, or it can be another node independent of the server in the online system.

[0023] In practical applications, online systems typically consist of a user terminal and a server terminal, with the user terminal and server terminal connected via a network. The network provides the medium for communication between the user terminal and the server terminal. Networks can include various connection types, such as wired and wireless communication links or fiber optic cables, etc.

[0024] The client-side interface can be geared towards application developers, such as software engineers or UI designers, who upload visual files that require code generation for the interface. The client can interact with the server over the network to receive and send messages. For example, the client can detect file uploads and code retrieval operations by application developers and send these to the server, allowing the server to perform code generation or delivery operations.

[0025] The user end can be a browser, an app (application), a web application such as an H5 (HyperText Markup Language 5) application, a mini-program (also known as a lightweight application), or a cloud application. The user end can be deployed on electronic devices and depends on the device to run or on certain apps within the device. Electronic devices can have displays and support information browsing, such as personal mobile terminals like smartphones, tablets, personal computers, desktop computers, smart speakers, smartwatches, etc.

[0026] The aforementioned processing end or server may include servers that provide various services, such as a server for background training that supports the code processing model of this embodiment, or a server that performs code conversion on the interface visual files sent by the user end.

[0027] It should be noted that the processing or server end can be implemented as a distributed server cluster consisting of multiple servers, or as a single server. The server can also be a server in a distributed system, or a server combined with blockchain. The server can also be a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDN), and big data and artificial intelligence platforms, or an intelligent cloud computing server or intelligent cloud host with artificial intelligence technology.

[0028] Figure 1 The code generation method shown may include the following steps: S101, Generate initial interface code based on interface visual files.

[0029] The interface visual file is a high-fidelity design file created using professional design tools (such as vector design tools like Figma or Sketch). It fully defines the layout, appearance, component structure, and interaction details of the user interface and serves as the core basis for the implementation of user interface development. It should be noted that one interface visual file corresponds to one user interface of the application, and the target interface code generated based on it is the interface code corresponding to that user interface.

[0030] In practical applications, interface visual files can be designed manually by interface designers or automatically created by intelligent assisted design tools. In this embodiment, after the interface visual file is created, the application development personnel (such as interface designers or software engineers) can upload the interface visual file that needs to generate interface code to the processing end through their client; alternatively, it can be automatically transmitted to the processing end after being created by intelligent assisted design tools.

[0031] Optionally, there are many ways to generate initial interface code based on interface visual files in this embodiment, and no limitation is imposed on this method.

[0032] One implementation approach involves using pre-defined rule algorithms to generate interface visual files. For example, parsing tools can be used to perform structured parsing of the interface visual files, transforming the graphical, visual-oriented files into semantic, code-oriented abstract descriptions, i.e., intermediate expression data (IR). Then, each interface element in the intermediate expression data is automatically converted into initial interface code using predefined code templates and matching rules. The intermediate expression data is a structured data representation that records the type, style (e.g., color, font, spacing), parent-child nesting relationships, and layout rules of each component (e.g., button, text box, image) in the corresponding interface of the visual file. While the intermediate data expression provides the logical structure of the design, inconsistencies between the layer order of the design tool and the actual development order can lead to element misalignment, potentially resulting in errors in the initial interface code generated from the intermediate expression data.

[0033] Another approach is to utilize an artificial intelligence model (such as the code processing model in this embodiment) to generate initial interface code based on the interface visual file. For example, the interface visual file can be directly input into the artificial intelligence model, which will then parse and output the converted initial interface code. Alternatively, based on the interface visual file, corresponding intermediate representation data and / or reference rendering images can be generated first, and then an intelligent generation model can be used to generate the initial interface code based on the intermediate representation data and / or reference rendering images.

[0034] The intelligent generation models involved in the technical solutions provided in this application, such as the code processing models mentioned below, can employ deep learning models with relatively large model parameter scales. However, the term "large model" is merely an example; this application does not limit the number of model parameters supported by the deep learning model used, aiming to meet actual needs. The deep learning models involved in this application can be artificial intelligence-based visual language models (VLM) or multimodal models (MM).

[0035] In practical applications, this embodiment can generate intermediate representation data and reference rendering images based on interface visual files; construct first context data based on intermediate representation data and reference rendering images; and input the first context data into the code processing model to generate initial interface code.

[0036] Specifically, the vector layers, styles, and layout data can be converted into visual image formats, such as bitmaps or vector graphics, using the rendering engine built into the design tool. The converted image is then called the reference rendering image. This reference rendering image is a high-resolution image obtained from rendering the interface visual file and can serve as the visual standard that the final generated user interface should meet. The process of generating intermediate representation data based on the interface visual file has been described in the above embodiments and will not be repeated here. Then, based on the intermediate representation data and the reference rendering image, the first context data is constructed. It should be noted that in practical applications, the first context data is composed of multiple source data, aiming to provide sufficient, relevant, and structured guiding information for the code processing model to generate interface code. Therefore, the first context data constructed in this embodiment not only includes input data consisting of intermediate representation data and the reference rendering image, but also includes first prompts and processing tools (such as file generation tools) for guiding model reasoning, as well as domain knowledge and historical interaction data to assist model reasoning. The content items included in the first context data can be set according to the actual situation, and this application does not limit them. Finally, the constructed first context data is input into the code processing model, which then generates the initial interface code based on this first context data. This embodiment utilizes the code processing model, combined with the first context data containing intermediate representation data and reference rendering images, to infer the initial interface code, thus ensuring the accuracy of the initial interface code as much as possible.

[0037] In one implementation, such as Figure 2As shown, the code processing model in this embodiment may include, for example, an input layer 10, an encoder 20, a decoder 30, and an output layer 40. It may also include a self-attention layer and a feed-forward neural network, etc., and this application does not impose any limitations on this. The input layer 10 is used to receive input data (such as first context data, and second and third context data as described later). The encoder 20 is mainly used to convert the input data (usually in sequence form) into a vector representation. This process can incorporate the semantic features of the input data. The decoder 30 is responsible for converting the intermediate representation generated by the encoder 20 into output data (usually in sequence form). The output layer 40 is used to output data (e.g., initial interface code, and updated interface code as described later). A self-attention layer is a mechanism that allows a model to focus on other positions in a sequence to better encode information about the current position. A feedforward neural network can perform nonlinear transformations on the output of the self-attention layer to enhance the model's expressive power. The various parts work together to enable the model built on them to perform well in a variety of complex processing tasks, such as natural language processing, computer vision, and code generation.

[0038] The code processing model in this embodiment can work with the prompt in the first context data, combined with input data, domain knowledge, historical interaction data, etc., to generate the initial interface code. The prompt may include processing instructions, role information, processing requirements, thought chain information, and / or example data, etc. The content items included in the prompt can be set according to the actual situation, etc., and this application does not limit them.

[0039] The processing instructions explicitly tell the model what to do, such as "combining the input data, converting the intermediate representation data and the reference rendering into the corresponding interface code, and outputting it." Role information can instruct the model to play a specific role, such as changing its professional nature, for example, "You are a professional software development expert." Processing requirements specify constraints, such as requiring the visual layout of the generated code after rendering (e.g., element position, spacing, alignment) to be highly consistent with the reference rendering. Thought chain information can guide the model to reason step by step. Sample data can provide learning samples to help the model understand the operations performed. This embodiment utilizes the powerful semantic understanding and intelligent reasoning capabilities of the code processing model to improve the accuracy of the initial interface code.

[0040] In practical applications, due to factors such as the standardization of interface visual files and the generalization of models, the initial interface code generated in S101 may inevitably have semantic deviations, structural errors, or inconsistencies with the design intent compared to the interface visual file. Therefore, to ensure the accuracy of interface code generation, this example iteratively executes at least one round of code update operations (i.e., S102-S104 below) after generating the initial interface code until the first update requirement is met to obtain the target interface code. The target interface code is the final interface code converted from the interface visual file, used to render the target interface corresponding to the generated interface view file. Specifically, the target interface code is the interface code to be processed in the current round when the code update operation meets the first update requirement. The first update requirement can be that the number of iterations of the code update operation reaches a preset number; or it can be that the difference information determined by the code update operation meets the difference requirement. The specific implementation methods for determining the target interface code for these two first update requirements will be described in detail in subsequent embodiments.

[0041] Optionally, the above code update operation includes the following steps: S102, render the interface code to be processed in the current round to generate candidate rendering images.

[0042] The interface code to be processed in the first round is the initial interface code generated by S101 based on the interface visual file. The interface code to be processed in other rounds is the interface code output by the code processing model in the previous round of code update operation (i.e., the interface code to be processed in the next round generated by step S104 in the previous round of code update operation).

[0043] Specifically, the current round of UI code to be processed is input into a rendering engine or tool to parse the layout, style, and component information within the code. Based on this parsed information, a visual rendering of the user interface design effect—the candidate rendering—is then created. For example, the current round of UI code to be processed can be loaded and executed in a headless browser. Its complete rendering engine parses the code, and after the interface layout and drawing are complete, an image of the visible area is automatically captured as a candidate rendering. The headless browser can be a browser without a graphical user interface, and the rendering operation can be performed via command line or as an interface controller.

[0044] S103, determine the difference information between the reference rendering image and the candidate rendering image corresponding to the interface visual file.

[0045] The differences between the reference rendered image and the candidate rendered image may include, but are not limited to, differences in at least one dimension such as pixel visual differences, layout structure offsets, and semantic content inconsistencies. Optionally, the difference information in this embodiment may be represented by at least one form such as similarity scoring, heatmaps, and text descriptions, and there is no limitation on this.

[0046] In this embodiment, the reference rendering image can be obtained by first converting the interface visual file into a reference rendering image as described in the above embodiments, and then the difference information between the reference rendering image and the candidate rendering image can be determined.

[0047] When determining the difference information, one approach is to scale the reference rendering image and the candidate rendering image to the same size, and then compare the differences in pixel values ​​at the same pixel positions in the two rendering images. For example, the mean square error of the pixel values ​​at each pixel position can be calculated as the difference information between the reference rendering image and the candidate rendering image.

[0048] Another approach is to extract interface elements (such as buttons, text, icons, etc.) from the reference rendering and the candidate rendering respectively, and compare the differences in the type, position, size, and content of the interface elements in the two renderings as the difference information between the reference rendering and the candidate rendering.

[0049] Another approach is to use a pre-trained visual model to extract the feature vectors corresponding to the reference rendering image and the candidate rendering image respectively, and then use the similarity of the feature vectors (such as cosine similarity) to measure the overall semantic similarity, thereby determining the difference information between the reference rendering image and the candidate rendering image.

[0050] S104, based on the difference information, uses the code processing model to update the interface code to generate the next round of interface code to be processed.

[0051] Optionally, the code processing model can be used to analyze potential errors in the current round of UI code based on discrepancy information. Then, for any errors found, the UI code can be updated (e.g., correcting existing errors), and the updated UI code can be used as the UI code for the next round of processing. In practical applications, this example can also utilize the code processing model, along with discrepancy information and reference renderings and / or intermediate representation data from the UI visual file, to update the UI code. In this case, the reference renderings and / or intermediate representation data from the UI visual file can serve as a reference standard during the update, assisting the code processing model in updating the code, further improving the accuracy of the update results, and reducing the number of iterative update operations.

[0052] It should be noted that the structure of the code processing model for performing the interface code update operation in this step can be found in [reference needed]. Figure 2As shown, it will not be elaborated upon here.

[0053] In some embodiments, this embodiment may construct context data (i.e., third context data) based on the interface code to be processed in the current round, the reference rendering image, and the difference information; and input the third context data into the code processing model to update the interface code.

[0054] The third context data is the data input to the code processing model during the code update operation. Similar to the first context data, the third context data is also composed of multi-source data, but it aims to provide sufficient, relevant, and structured guidance information for the code processing model to perform code update operations. The third context data constructed in this embodiment includes not only input data consisting of the interface code to be processed in the current round, the reference rendering image, and the aforementioned difference information, but also prompts and processing tools (such as file reading tools for reading interface code, file writing tools for modifying interface code, etc.) to guide model reasoning, as well as domain knowledge (such as common code error types and their correction methods) and historical interaction data to assist model reasoning. The content items included in the third context data can be set according to actual conditions; for example, intermediate representation data of the interface visual file can be further added to the input data. This application does not limit this. Finally, the constructed third context data is input into the code processing model so that the code processing model can update the interface code to be processed.

[0055] The code processing model in this embodiment can work with prompts in the second context data, combining input data, domain knowledge, historical interaction data, etc., to realize the code update process. These prompts can include update instructions, role information, update requirements, thought chain information, and / or example data, etc. The content items included in the second prompts can be set according to actual circumstances, and this application does not limit this. Specifically, the update instructions clearly tell the model what needs to be done, such as "combining the difference information, update the interface code so that the rendered image of the interface code approaches the visualization effect of the reference rendered image," etc.; role information can instruct the model to play a specific role, changing its professionalism, such as "you are a professional software development expert," etc.; update requirements are used to specify constraints, such as the priority of correcting different error types, the number of error types corrected each time, etc.; thought chain information can be used to guide the model to reason step by step, etc.; example data can be used to provide learning samples to help the model understand the operations performed, etc. This embodiment utilizes the powerful semantic understanding and intelligent reasoning capabilities of the code processing model to improve the accuracy of interface code updates.

[0056] In practical applications, code update operations are executed in multiple iterations. As the number of iterations increases, the amount of historical interaction data in the historical context data also increases, leading to excessively long context data and affecting the performance of the code processing model. Furthermore, since errors corresponding to historical interaction data have already been corrected, retaining historical interaction data may interfere with the current round of code update operations. Therefore, to avoid the above problems, this embodiment, when updating the interface code based on difference information using the code processing model, can delete historical interaction data from the historical context data of the code processing model to obtain updated historical context data. The deleted historical interaction data in this embodiment can include historical interface code to be processed, as well as historical input data. Based on the updated historical context data, the current round of interface code to be processed, the reference rendering image, and the difference information, second context data is constructed. The second context data is then input into the code processing model to update the interface code. It should be noted that the process of constructing the second context data and inputting the second context data into the code processing model to update the interface code is similar to the process of constructing the third context data and inputting the third context data into the code processing model to update the interface code described in the above embodiments. The difference is that the historical interaction data is deleted from the historical context data of the second context data.

[0057] In this embodiment, historical interaction data is deleted when constructing the second context data. This introduces a context interaction data truncation mechanism, which retains the interface code to be processed in the current round, difference information, and reference rendering images. This prevents the code processing model from experiencing performance degradation and exceeding the context window due to excessively long context. It also allows the code processing model to focus on the error types of the interface code to be processed in the current round, without being disturbed by previous errors, thus improving the accuracy of code updates.

[0058] Regarding the code update operations described in S102-S104 above, if the first update requirement is that the number of iterations of the code update operation reaches a preset number, before executing S102 to render the current round of pending interface code into a candidate rendering image, it is determined whether the number of iterations has reached the preset number. If the number of iterations has reached the preset number, the current round of pending interface code is used as the target interface code. If the number of iterations has not reached the preset number, the current round of code update operations (i.e., the current round of S102-S104 operations) continues. In other words, before the current round of iterative update operations, it is determined whether the cumulative number of iterations has reached a preset number. If so, the iteration ends, the current round of pending interface code is not updated, and it is directly used as the target interface code; otherwise, the current round of iterative update operations is executed. This implementation method can effectively control computational resources and time overhead, avoiding infinite loops or inefficiency caused by repeated updates of interface code.

[0059] If the difference information determined by the code update operation in the first update requirement meets the difference requirement, after executing S103 to determine the difference information between the reference rendering image corresponding to the interface visual file and the candidate rendering image, the method further includes determining whether the difference information meets the difference requirement; if the difference information meets the difference requirement, the interface code to be processed in the current round is taken as the target interface code; if the difference information does not meet the difference requirement, the interface code is updated based on the difference information using the code processing model to generate the interface code to be processed in the next round.

[0060] Specifically, the difference requirements in this embodiment can be pre-set. Meeting the difference requirements indicates that the difference between the reference rendering image and the candidate rendering image is small. For example, small pixel visual differences, small layout architecture deviations, or small semantic content deviations. If the difference information is a score that measures the difference (such as a content similarity score), then a content similarity score higher than a first similarity threshold can be considered as meeting the difference requirements. In this embodiment, for each round of code update operations, after performing operation S103, it is determined whether the difference information determined in the current round meets the difference requirements. If so, the interface code to be processed in the current round is directly used as the target interface code, and operation S104 is not required. Otherwise, operation S104 is performed to update the interface code to be processed in the current round before continuing to perform the code update operation in the next round. This implementation method is code quality oriented. The code update operation stops only when the updated interface code is relatively accurate, that is, when the difference between the candidate rendering image and the reference rendering image of the interface code is small, thereby ensuring the reliability and accuracy of the output target interface code.

[0061] In practical application scenarios, this embodiment can also combine the above two situations, that is, satisfy either the current round of code update operation's iteration execution count reaches the preset number, or the difference information meets the difference requirements, so that the interface code to be processed in the current round can be used as the target interface code, thereby achieving both efficiency and accuracy in determining the target interface code.

[0062] In practical applications, to further improve the comprehensiveness and accuracy of the difference information representation, the difference information in this embodiment may include content similarity and a difference heatmap. Correspondingly, another implementation of step S103 may be to calculate the content similarity between the reference rendering image and the candidate rendering image corresponding to the interface visual file; and generate a difference heatmap based on the target pixel positions that differ between the reference rendering image and the candidate rendering image.

[0063] Specifically, when calculating content similarity, the similarity between the reference and candidate rendered images can be measured by comparing the pixel values. Alternatively, an intelligent recognition model can be used to identify the content features of the two rendered images, and the similarity of these features can be compared to determine their content similarity. It should be noted that in this embodiment, when representing difference information through content similarity, a higher content similarity may correspond to a smaller difference. When determining the difference heatmap, the reference and candidate rendered images can be compared to find target pixel locations with large pixel value differences (e.g., pixel value differences greater than a preset difference value). A heatmap representing the differences in the rendered images is then constructed based on these target pixel locations. This difference heatmap can highlight the pixel locations with color. This embodiment combines content similarity with the difference heatmap to represent the difference information between the reference and candidate rendered images, which helps the code processing model better perceive errors in the interface code and thus update it accurately.

[0064] In practical applications, this embodiment can calculate content similarity more comprehensively and accurately through the following sub-steps: Sub-step 1: Determine the grayscale distribution matching score based on the difference in grayscale distribution between the reference rendering image and the candidate rendering image. Specifically, grayscale distribution images of the reference rendering image and the candidate rendering image can be generated separately, and then the difference in the dispersion of the grayscale distribution between the two can be compared. The difference in dispersion is then quantified into a grayscale distribution matching score, where the smaller the difference in dispersion, the higher the corresponding grayscale distribution matching score.

[0065] Sub-step 2: Slice the reference rendering image and the candidate rendering image according to their respective slice sizes, and determine the slice number matching score based on the difference in the number of slices between the reference rendering image and the candidate rendering image.

[0066] In this embodiment, the slice size corresponding to the reference and candidate rendered images depends on the size of their respective rendered images. For example, the slice height can be determined based on the width of the reference and candidate rendered images. For instance, the product of the image width and a preset ratio (e.g., 0.05) can be used as the slice height. Then, the rendered image corresponding to that slice size is sliced ​​according to the slice size to obtain the number of slices for that rendered image. The number of slices is equal to the ratio of the image height to the slice height (rounded up). This embodiment can compare the difference in the number of slices between the reference and candidate rendered images after slicing, and then quantify this difference into a slice number matching score. The smaller the difference in the number of slices between the reference and candidate rendered images, the higher the slice number matching score. For example, this embodiment can deduct points from the total slice number matching score based on the ratio of the slice number difference to the larger slice number in the two rendered images, thus obtaining the slice number matching score.

[0067] Sub-step 3: Determine the region matching score based on the region similarity of at least one set of sub-regions that match between the reference rendering image and the candidate rendering image. Specifically, local regions that match between the reference rendering image and the candidate rendering image can be identified as a set of sub-regions. After obtaining at least one sub-region, the region similarity between two local regions in each set of sub-regions is calculated, and the region similarity measure of at least one sub-region is quantified into a region matching score. For example, the region similarity measure of at least one set of sub-regions can be averaged to obtain the region matching score.

[0068] In practical applications, another possible way to determine the region matching score in this embodiment may include: segmenting the reference rendering image and the candidate rendering image according to the segmentation requirements to obtain multiple first sub-regions corresponding to the reference rendering image and multiple second sub-regions corresponding to the candidate rendering image; for any first sub-region, searching the candidate rendering image for the second sub-region with the highest content similarity to the first sub-region to form a group of sub-regions; calculating the region similarity between the first sub-region and the second sub-region in the same group of sub-regions; and determining the content matching score based on the region similarity corresponding to at least one group of sub-regions.

[0069] The segmentation requirements can be pre-set fixed and uniform size requirements, or they can be determined based on the sizes of the reference rendering image and the candidate rendering image; there is no limitation on this. In this embodiment, the reference rendering image and the candidate rendering image are segmented according to the segmentation requirements. The sub-regions after segmentation of the reference rendering image are called the first sub-regions, and the sub-regions after segmentation of the candidate reference image are called the second sub-regions. For any first sub-region, the second sub-region with the highest similarity in the candidate rendering image is searched, forming a group of sub-regions with the first sub-region, thus obtaining at least one group of sub-regions. For each group of sub-regions, the regional similarity between the first and second sub-regions contained within it is calculated as the regional similarity corresponding to that group of sub-regions. The regional similarities corresponding to each group of sub-regions are summed or averaged to obtain the content matching score between the reference rendering image and the candidate rendering image. In this embodiment, the content similarity between the first and second regions can be calculated by identifying the region content through model recognition or computer vision algorithms. Optionally, this embodiment can also adjust the reference rendering image and the candidate rendering image to the same size before segmenting them. For example, the images can be uniformly adjusted to a fixed size, or the size of one rendered image can be adjusted based on the other to make them the same size. Then, segmentation is performed according to the segmentation requirements. Since the reference and candidate rendered images have been adjusted to the same size, when searching for the second sub-region with the highest content similarity to any first sub-region, the search range of the first sub-region in the candidate rendered image can be determined first based on its position in the reference rendered image. For example, based on the position of the first sub-region in the reference rendered image, a preset range around that position in the candidate rendered image can be used as the search range. Then, within this search range, the second sub-region with the highest content similarity to the first sub-region is searched, forming a group of sub-regions. This method greatly improves the efficiency and accuracy of determining each group of sub-regions, thereby improving the accuracy of the similarity score determination.

[0070] Sub-step 4: Determine the content similarity between the reference rendering image and the candidate rendering image based on the grayscale distribution matching score, the slice quantity matching score, and the region matching score. Specifically, this embodiment can sum, average, or weight the grayscale distribution matching score, slice quantity matching score, and region matching score (such as weighted summation, weighted average, etc.), and use the processing result as the content similarity between the reference rendering image and the candidate rendering image.

[0071] This embodiment measures the content similarity between the reference rendering image and the candidate rendering image from multiple dimensions, including overall grayscale distribution, number of slices, and regional similarity, thereby improving the accuracy of content similarity determination.

[0072] In practical applications, this embodiment can also calculate the difference heatmap more comprehensively and accurately through the following sub-steps: Sub-step A: Adjust the reference rendering image and the candidate rendering image to the same image size. The specific implementation method has been described in the above embodiments and will not be repeated here.

[0073] Sub-step B: Determine the pixel difference at the same pixel position in the adjusted reference rendering image and candidate rendering image. Specifically, after the image sizes of the reference rendering image and candidate rendering image are adjusted to be consistent, for each pixel position in the reference rendering image, determine the difference (i.e., pixel difference) between the first pixel value in the reference rendering image and the second pixel value in the corresponding candidate rendering image for that pixel position.

[0074] Sub-step C: Determine the target pixel position where the pixel difference is greater than a preset value. Specifically, the pixel position where the pixel difference determined in sub-step B is greater than a preset value (e.g., 28) is taken as the target pixel position.

[0075] Sub-step D: Determine the mask image corresponding to the reference rendering image, and set the target pixel value for the target pixel position in the mask image. Specifically, first initialize a mask image with the same size as the adjusted reference rendering image, and set the target pixel value (e.g., 255) for the target pixel position in the mask image.

[0076] Sub-step E: Map the mask image after setting the target pixel value to an initial heatmap, and overlay the initial heatmap with the adjusted reference rendering image to obtain a difference heatmap. Specifically, in this embodiment, the mask image after setting the target pixel value can be mapped to a heatmap, i.e., the initial heatmap. For example, the mask image can be converted into a heatmap mapped with pseudo colors (such as Jet colors) (e.g., red represents the difference area, and blue represents the no difference). Finally, the reference rendering image and the initial heatmap are overlaid, for example, according to a certain weight ratio (e.g., 0.5:0.5), to generate a semi-transparent blended difference visualization result, i.e., a difference heatmap. This difference heatmap can use different color regions to characterize which regions have misalignment, missing parts, or style errors.

[0077] Figure 3 This is a flowchart illustrating an embodiment of a model training method provided in this application. The technical solution of this embodiment can be executed by a processing terminal. Figure 3 The model training method shown is used for training. Figure 1 The code processing model in the code generation method shown. For example... Figure 3 As shown, the model training method includes the following steps: S301, acquire multiple first training data.

[0078] The first training data includes first sample interface code and a first sample reference rendering image corresponding to the first sample interface code. The first sample interface code can be high-quality interface code covering multiple fields such as e-commerce, social media, and multimedia. This first sample interface code can be obtained in at least one of the following ways: Method 1: Obtain high-quality interface code written manually for the user interface.

[0079] Method 2: Utilize a code generation model to generate the first sample interface code based on preset prompts. This code generation model can be a pre-trained intelligent model capable of writing high-quality code based on prompts. The preset prompts may include generation instructions, role information, code generation requirements, thought chain information, and / or example data. The content of the preset prompts can be set according to actual circumstances, and this application does not limit this. Specifically, the generation instructions clearly tell the model what to do, such as "Please generate a high-quality interface code according to the requirements," etc.; role information can instruct the model to play a specific role, changing its professional nature, such as "You are a professional software development expert," etc.; code generation requirements specify constraints such as structure and layout requirements, technology stack limitations, code quality and standards, etc.; thought chain information can guide the model to reason step by step, etc.; example data can provide learning samples to help the model understand the operations being performed, etc. This embodiment utilizes the powerful semantic understanding and intelligent reasoning capabilities of the code generation model to improve the accuracy of interface code generation.

[0080] Method 3: From the target interface code obtained by iteratively executing at least one round of code update operations using the above code processing model, select the first sample interface code; the difference information between the rendered image of the first sample interface code and the reference rendered image of the corresponding visual file meets the sample code requirements. This sample code requirement can be pre-set and used to judge high-quality code. For example, it can be that the content similarity in the difference information is higher than a second similarity threshold. It should be noted that the second similarity threshold at this time can be the same as the first similarity threshold that meets the difference requirement in the above embodiments, or the second similarity threshold can be higher than the first similarity threshold.

[0081] In this embodiment, after obtaining the first sample interface code, the rendering process is performed according to the rendering method described in the above embodiment, and the resulting rendering image is the first sample reference rendering image.

[0082] S302, for any first training data, perform at least one code perturbation operation on the first sample interface code to obtain the perturbation interface code, and generate recovery guidance information based on the at least one code perturbation operation.

[0083] In this embodiment, for the first sample interface code in each training data, common error types during code writing, such as syntax errors and structural errors, are simulated. At least one perturbation operation is applied to the first sample interface code. For example, it can be to modify the width of a component, delete a necessary style attribute, or mistakenly write a button as a regular component label. The interface code after performing at least one code perturbation operation is called perturbed interface code, which is a degenerate code with incomplete functionality or visual errors.

[0084] This implementation can record every code perturbation operation performed on the first sample interface code, and after the perturbation ends, it will reverse at least one code perturbation operation to obtain recovery guidance information to restore the degraded perturbation interface code to the high-quality first sample interface code. This recovery guidance information can be used to guide a set of reverse operation instructions to gradually correct the perturbation interface code and restore it to the first sample interface code. Its core function is to provide an executable repair path to trace back from the degraded state to the high-quality state.

[0085] This embodiment can construct perturbation interface codes with different recovery difficulties and their recovery guidance information for the same first sample interface code by controlling the number of executions of code perturbation operations. For example, perturbation counts in the first range (e.g., 1-3 times) represent perturbation interface codes with lower recovery difficulty, while perturbation counts in the second range (e.g., 4-7 times) represent perturbation interface codes with higher recovery difficulty. The values ​​corresponding to the second range are higher than those corresponding to the first range. The perturbation interface codes and their recovery guidance information can be selected in order of increasing recovery difficulty to perform the following model training operation: as the number of training iterations increases, the number of code perturbation operations applied to the perturbation interface codes also gradually increases. This approach allows the model to first master simple code update tasks before learning complex code update tasks such as deep nesting and multi-component linkage, helping the code processing model establish solid code update capabilities and improving model convergence stability and final performance.

[0086] S303 renders the perturbation interface code to generate a perturbation rendering image.

[0087] S304, determine the first sample difference information between the sample reference rendering image and the perturbation rendering image.

[0088] Specifically, the implementation of S303-S304 is similar to the implementation of generating candidate rendering images from the interface code to be processed in the current round in the above embodiment, and determining the difference information between the reference rendering image and the candidate rendering image corresponding to the interface visual file, and will not be described in detail here.

[0089] S305 uses the perturbation interface code, the first sample difference information, the recovery guidance information, and the first sample interface code to train the code processing model.

[0090] In this process, the code processing model iteratively executes at least one round of code update operations. Based on the difference information between the reference rendering image and the candidate rendering image corresponding to the interface visual file, it updates the interface code to be processed in the current round to generate the interface code to be processed in the next round. The candidate rendering image is obtained by rendering the interface code to be processed in the current round. The iterative execution of at least one round of code update operations is used to update the initial interface code generated based on the interface visual file until the first update requirement is met to obtain the target interface code. The specific implementation method has been described in the above embodiments and will not be repeated here.

[0091] This embodiment can use perturbation interface code, first sample difference information, and recovery guidance information as input data, and the first sample interface code as training labels to perform supervised training on the code processing model. For example, the perturbation interface code, first sample difference information, and recovery guidance information can be input into the code processing model, which predicts the updated interface code based on the input data. Then, using the first sample interface code as a standard, the loss value between the updated interface code and the first sample interface code is calculated, and the model parameters of the code processing model are adjusted based on the loss value.

[0092] This embodiment proposes a method for generating training data with inverse perturbation. Specifically, after obtaining high-quality first sample interface code and its corresponding first sample reference rendering image, code perturbation is added to the first sample interface code to obtain degraded perturbed interface code and its corresponding recovery guidance information. This information is then used to train the code processing model for the code update task. This solves the fundamental bottleneck problem of missing real training data. Furthermore, the accurate recovery guidance information can assist the model in learning, enabling the construction of a large amount of training data to complete the training process for the code update task even when real training data is missing.

[0093] The training of the code processing model in this step is essentially training it for the code update task. In practical applications, this code processing model may also be used to generate initial interface code based on intermediate representation data and reference rendering images from the interface visual file. Therefore, this embodiment can also train the code processing model for the code generation task. The first training data at this time also includes the first sample intermediate representation data corresponding to the first sample interface code. Specifically, the first sample interface code can be decompiled to obtain its corresponding intermediate representation data (i.e., the first sample intermediate representation data). Correspondingly, before performing the training operation of the above-mentioned S305 code update task, it may also include performing at least one representation perturbation operation on the first sample intermediate representation data to obtain perturbed intermediate representation data; using the first sample reference rendering image, the perturbed intermediate representation data, and the first sample interface code, training the code processing model; the code processing model is also used to generate initial interface code based on the intermediate representation data generated from the interface visual file and the reference rendering image. Specifically, common errors that occur when design tools convert interface visual files into intermediate representation data can be simulated, such as incorrect layer order. At least one representation perturbation operation is performed on the first sample intermediate representation data, for example, at least one sibling node reordering perturbation, to obtain perturbed intermediate representation data. It should be noted that when performing perturbation operations on the first sample intermediate representation data, all components must be preserved; no additional components are added or deleted. Then, using the sample reference rendering and the perturbed intermediate representation data as input data, and the first sample interface code as the training label, supervised training is performed on the code processing model. For example, the sample reference rendering and the perturbed intermediate representation data can be input into the code processing model, which then performs a code generation task based on the input data, outputting initial interface code. The first sample interface code is then used as a standard to calculate the loss value between the initial interface code and the first sample code, and the model parameters are adjusted based on this loss value. This embodiment, for training the code generation task of the code processing model, constructs perturbed intermediate representation data by applying representation perturbation operations when real training data is lacking, ensuring the accuracy of the training results.

[0094] Furthermore, for training code generation tasks, the number of times the expression perturbation operation is executed can be controlled. Different numbers of perturbation operations can be performed on the same initial sample intermediate expression data to obtain perturbed intermediate expression data corresponding to different perturbation counts. Then, in order of increasing perturbation count, the corresponding perturbed intermediate expression data can be selected to perform the aforementioned code generation task training operations. This allows the model to master simple code generation tasks before learning code generation tasks for deep and complex scenarios, thereby improving the model's convergence stability and final performance.

[0095] In practical applications, this embodiment can further train the model using the following method based on the above training process to improve the final performance of the model. Specifically, after training the code processing model using the perturbation interface code, the first sample difference information, the recovery guidance information, and the first sample interface code in the above embodiment, the following sub-steps are also included: Sub-step 1: Obtain multiple second training data sets; the second training data sets include the second sample reference rendering image and the second sample intermediate representation data.

[0096] This embodiment can acquire a sample interface visual file and generate a second sample reference rendering image and second sample intermediate representation data corresponding to the sample interface visual file, following the method described in the above embodiment. Alternatively, a similar acquisition method as the first sample interface code can be used to acquire the second sample interface code, or the first sample interface code can be directly used as the second sample interface code. Then, the second sample interface code is rendered to obtain the second sample reference rendering image, and the second sample interface code is decompiled to obtain the second sample intermediate representation data.

[0097] It should be noted that the second sample intermediate expression data in this embodiment can be intermediate expression data directly generated based on the sample interface visual file or the second sample interface code; or it can be intermediate expression data with added perturbation obtained by generating intermediate expression data based on the sample interface visual file or the second sample interface code and then performing at least one expression perturbation operation on the generated intermediate expression data.

[0098] Sub-step two: For any second training data, based on the second sample parameter rendering map and / or the second sample intermediate expression data, generate initial candidate interface code, and iteratively execute at least one round of sample code update operation until the second update requirement is met. The sample code update operation includes: rendering the candidate interface code to be processed in the current round to generate a candidate sample rendering map; the candidate interface code to be processed in the first round is the initial candidate interface code; determining the second sample difference information between the second sample reference rendering map and the candidate sample rendering map; and updating the candidate interface code using a code processing model based on the second sample difference information to generate the candidate interface code to be processed in the next round.

[0099] It should be noted that the implementation method of this sub-step is similar to that of S101-S104 in the above embodiments, and will not be described in detail here.

[0100] In practical applications, when updating the candidate interface code using the code processing model based on the second sample difference information, this sub-step can be performed as follows: Historical interaction data is deleted from the sample historical context data of the code processing model to obtain updated sample historical context data; the historical interaction data includes historical candidate interface code; based on the updated sample historical context data, the candidate interface code to be processed in the current round, the second sample reference rendering image, and the second sample difference information, current sample context data is constructed; the current sample context data is input into the code processing model to update the candidate interface code. The specific implementation method is similar to the above embodiment of deleting historical interaction data, constructing second context data, and inputting it into the code processing model to update the interface code, and will not be elaborated here.

[0101] Sub-step 3: Based on the sample code update operation meeting the second update requirement, determine the reward information corresponding to the second sample difference information, and adjust the model parameters of the code processing model according to the reward information.

[0102] The second update requirement is similar to the first update requirement in the above embodiment. Therefore, the method used in this embodiment to determine whether the sample iterative update operation meets the second update requirement is also similar to the method used in the above embodiment to determine whether the iterative update operation meets the first update requirement, and will not be elaborated here. The difference is that in this embodiment, the second sample difference information generated in sub-step two when the sample code update operation meets the second update requirement is obtained, and the reward information is determined based on the second sample difference information. For example, the reward information is determined based on the content similarity in the second sample difference information; the higher the content similarity, the higher the corresponding reward score. With maximizing the reward information as the training objective, the model parameters of the code processing model are continuously adjusted using an intensity learning algorithm (such as the GRPO algorithm) based on the currently determined reward information.

[0103] Optionally, in this embodiment, the code processing model can be trained iteratively in multiple rounds as described above until the training requirements are met, in order to obtain the trained code processing model. The training requirements at this point could be that the accuracy of the trained code processing model reaches a preset accuracy, or that the number of training iterations reaches a preset prediction level, etc.

[0104] This embodiment adds reinforcement learning training based on reward information on the basis of supervised training, giving the code processing model the ability to update autonomously based on environmental feedback, and further improving the performance of the trained code processing model.

[0105] Figure 4 This is a flowchart illustrating an embodiment of a model training method provided in this application. The technical solution of this embodiment can be executed by a processing terminal. Figure 4The model training method shown is used for training. Figure 1 The code processing model in the code generation method shown. For example... Figure 4 As shown, the model training method includes the following steps: S401, acquire multiple second training data; the second training data includes a second sample reference rendering image and second sample intermediate expression data.

[0106] S402, for any second training data, generate initial candidate interface code based on the second sample parameter rendering map and / or the second sample intermediate expression data.

[0107] Iteratively execute at least one round of sample code update operations until the second update requirement is met. These sample code update operations include: S403, render the candidate interface code to be processed in the current round to generate a candidate sample rendering image; the candidate interface code to be processed in the first round is the initial candidate interface code.

[0108] S404, determine the second sample difference information between the second sample reference rendering image and the candidate sample rendering image.

[0109] S405, based on the second sample difference information, the candidate interface code is updated using the code processing model to generate the candidate interface code to be processed in the next round.

[0110] S406, based on the second sample difference information corresponding to the sample code update operation meeting the second update requirement, determine the reward information, and adjust the model parameters of the code processing model according to the reward information.

[0111] The code processing model is used to iteratively execute at least one round of code update operations on the initial interface code corresponding to the interface visual file until the target interface code is obtained. The code update operation is used to update the interface code to be processed in the current round based on the difference information between the reference rendering image and the candidate rendering image corresponding to the interface visual file, so as to generate the interface code to be processed in the next round. The candidate rendering image is obtained by rendering the interface code to be processed in the current round.

[0112] The detailed implementation methods and beneficial effects of each step in this embodiment have been described in detail in the foregoing embodiments, and will not be elaborated here.

[0113] In a practical application scenario, an intelligent agent with code generation and iterative update capabilities can be deployed on the processing end. This intelligent agent is based on the code processing model described in the above embodiment and may also include functional modules such as image rendering and difference comparison. Next, combined with... Figure 5 The code generation method of the embodiments of this application will be described.

[0114] In this embodiment, after receiving the interface visual file (such as the original high-fidelity visual draft) provided by the interface designer, the agent deployed on the processing end can first generate corresponding intermediate expression data and reference rendering based on the interface visual file. Then, using the code processing model, it integrates the intermediate expression data and reference image (i.e., multimodal information) to generate initial interface code. The syntax of this initial interface code is usually reasonable, but its rendering image still has certain differences compared with the reference rendering image. For example, an icon may be offset by 5 pixels, or the color of a certain text may not match. Therefore, the agent needs to perform at least one round of code update operations on the generated initial interface code. Specifically, the agent will use the initial interface code as the interface code to be processed in the first round, start the rendering and difference comparison function, execute the interface code to be processed in the headless browser and capture the rendering result as a candidate rendering image, calculate the content similarity between the reference rendering image and the candidate rendering image, determine the target pixel positions where there are differences between the reference rendering image and the candidate rendering image, and generate a difference heatmap. If the content similarity does not reach the first similarity threshold, the intelligent system constructs the current round of context data based on the interface code to be processed, the difference heatmap, the content similarity, the reference rendering image, intermediate expression data, and the historical context data after deleting historical interaction data. This data is then input into the code processing model. The code processing model will autonomously decide to call the corresponding tool to update the interface code to be processed. For example, it calls a file reading tool to read the interface code from the working directory; it calls a file writing tool to correct the read interface code; and it calls an end tool to end the current round of generation and modification operations to obtain the interface code to be processed in the next round. After that, the rendering and difference comparison functions are restarted. This iterative loop of generation, rendering, comparison, and updating will be executed a maximum of a preset number of times, or until the content similarity in the comparison results meets the first similarity threshold. At this point, the interface code to be processed is the final target interface code generated for the interface visual file.

[0115] Although this embodiment also relies on intermediate representation data in generating the target interface code, this data serves only as supplementary clues, not the sole reference. Even if the intermediate representation data is misinterpreted, the agent can still employ a visual feedback-driven, multi-round iterative code generation paradigm. This transforms the interface code generation from a static, single-output process into a verifiable and updatable iterative task. Specifically, it introduces an iterative mechanism of rendering, comparison, and updating. After each generation of interface code, it is automatically rendered in a headless browser. Difference analysis quantifies the gap between the generated interface code and the interface visual file, generating a feedback signal containing content similarity and a difference heatmap. This feedback signal acts as a new context input code processing module, driving targeted code repair. This enables the agent to debug code, ensuring the accuracy of the interface code. This method does not limit the component library of the interface visual file; even complex elements such as custom icons and dynamic styles can generate high-quality interface code through the visual comparison and iterative update method of this embodiment.

[0116] To support the iterative generation and updating capabilities of the intelligent agent, this embodiment also requires training the code processing model within the agent specifically for this capability. However, in real-world training scenarios, training data is scarce. Therefore, this embodiment introduces a low-cost, large-scale data synthesis strategy and employs a two-stage joint training strategy of supervised training and reinforcement learning to train the code processing model, enabling it to learn the tasks of generating and updating interface code. Figure 6 As shown, the specific training process includes: Using a pre-trained code generation model, complete and high-quality interface code (such as React code) covering multiple domains is generated based on specially designed prompts as sample interface code. Additionally, high-quality code with high content similarity to the reference rendering of the interface visual file can be selected from target interface code generated by the agent based on real interface visual files as sample interface code. For example, target interface code with content similarity exceeding a second similarity threshold can be selected as sample interface code (S601). The sample interface code is then decompiled into intermediate sample representation data. Next, at least one controllable code perturbation operation (such as adding style or structural perturbations) is performed on the sample interface code to obtain perturbed interface code (S602). This embodiment also records each code perturbation operation and processes it in reverse order to generate recovery guidance information from perturbed interface code to sample interface code. At least one controllable representation perturbation operation (such as reordering sibling nodes) is performed on the intermediate sample representation data to obtain perturbed intermediate representation data (S603). Based on the perturbed intermediate representation data, the perturbed interface code, the rendering of the sample interface code (i.e., the sample reference rendering), and the recovery guidance information, a synthetic sample is generated (S604). This embodiment allows for the construction of synthetic samples with varying training difficulty by controlling the number of perturbation additions. This enables the model to first master basic generation and update capabilities, and then learn to generate and update deep nested and multi-component collaborative scenes. The sample synthesis process requires no manual annotation, is infinitely scalable, and provides a large-scale training dataset for the model, fundamentally solving the problem of scarce real training data.

[0117] like Figure 6 As shown, this embodiment introduces a two-stage training approach during the model training phase: supervised training (S605) and reinforcement learning (S606). Supervised training is further divided into training for the code generation task and training for the code update task. The code generation task training process can use the sample reference rendering image and perturbed intermediate representation data from the synthesized samples as input, and the sample interface code as the training label to perform supervised training on the code processing model for the code generation task. The code update task training process can use the perturbed interface code from the synthesized samples, restored guidance information, perturbed intermediate representation data, and sample difference information between the rendering image of the perturbed interface code and the sample reference rendering image as input, and the sample interface code as the training label to perform supervised training on the code processing model for the code update task. This stage of training provides the code processing model with a reliable initial strategy, teaching it how to call the correct tools for code updates. For example, if a button is off-center to the right, its left margin value is reduced, enabling it to respond appropriately to common errors.

[0118] The second stage of reinforcement learning can involve having the agent follow [the instructions / methods] in a simulated environment. Figure 5 The illustrated process automatically completes the initial code generation and multiple rounds of iterative updates. Then, based on the difference between the final output rendering image of the interface code (i.e., the candidate rendering image) and the sample parameter rendering image after the task is completed, reward information is determined, and the model parameters of the code processing model are updated using a reinforcement learning algorithm (such as the GRPO algorithm). To ensure the stability of long-sequence decision-making, this embodiment can also introduce a context truncation mechanism in the reinforcement learning phase. That is, after each round of code update operations, historical interaction data in the historical context data is cleared, forcing the model to focus on the current state. It learns to explore and correct errors based on feedback differences in uncertain environments, proactively updating the code.

[0119] In this embodiment, after performing two stages of training on the code processing model—supervised training in S605 and reinforcement learning in S606—the trained code processing model can be deployed in the execution agent (S607) to execute... Figure 5 The code generation method shown.

[0120] The detailed implementation methods and beneficial effects of each step in this embodiment have been described in detail in the foregoing embodiments, and will not be elaborated here.

[0121] It should be noted that some processes described in the above embodiments and accompanying drawings include multiple operations that appear in a specific order. However, it should be clearly understood that these operations may not be executed in the order they appear in this document, or they may be executed in parallel. The sequence numbers of the operations are merely used to distinguish different operations and do not represent any execution order. Furthermore, these processes may include more or fewer operations, and these operations may be executed sequentially or in parallel. It should also be noted that the descriptions such as "first" and "second" in this document are used to distinguish different messages, devices, modules, etc., and do not represent a sequential order, nor do they limit "first" and "second" to different types.

[0122] Figure 7 A schematic diagram of a code generation apparatus provided for an exemplary embodiment of this application is shown. The apparatus includes: The first code generation module 701 is used to generate initial interface code based on the interface visual file; The code update operation is iteratively executed at least once until the first update requirement is met to obtain the target interface code; the code update operation is implemented through the following modules: The first rendering module 702 is used to render the interface code to be processed in the current round to generate a candidate rendering image; the interface code to be processed in the first round is the initial interface code. The first difference determination module 703 is used to determine the difference information between the reference rendering image corresponding to the interface visual file and the candidate rendering image; The first code update module 704 is used to update the interface code based on the difference information using a code processing model to generate the next round of interface code to be processed.

[0123] In some embodiments, the first update requirement includes: the number of iterations of the code update operation reaches a preset number; the device further includes: a first update judgment module, configured to determine whether the number of iterations has reached a preset number before generating a candidate rendering image from the interface code to be processed in the current round; if the number of iterations has reached a preset number, the interface code to be processed in the current round is used as the target interface code; if the number of iterations has not reached a preset number, the code update operation in the current round continues to be executed.

[0124] In some embodiments, the first update requirement includes: the difference information determined by the code update operation satisfies the difference requirement; the apparatus further includes: a second update judgment module, configured to determine whether the difference information satisfies the difference requirement after determining the difference information between the reference rendering image corresponding to the interface visual file and the candidate rendering image; if the difference information satisfies the difference requirement, the interface code to be processed in the current round is taken as the target interface code; the first code update module 704 is configured to update the interface code based on the difference information and using a code processing model to generate the interface code to be processed in the next round if the difference information does not satisfy the difference requirement.

[0125] In some embodiments, the first code generation module 701 is specifically used to generate intermediate expression data and the reference rendering image based on the interface visual file; construct first context data based on the intermediate expression data and the reference rendering image; and input the first context data into the code processing model to generate initial interface code.

[0126] In some embodiments, the first code update module 704 is specifically used to delete historical interaction data from the historical context data of the code processing model to obtain updated historical context data; the historical interaction data includes historical interface code to be processed; based on the updated historical context data, the interface code to be processed in the current round, the reference rendering image, and the difference information, a second context data is constructed; the second context data is input into the code processing model to update the interface code.

[0127] In some embodiments, the first difference determination module 703 includes a similarity determination unit and a heatmap generation unit; the similarity determination unit is used to calculate the content similarity between the reference rendering image and the candidate rendering image corresponding to the interface visual file; the heatmap generation unit is used to generate a difference heatmap based on the target pixel positions that differ between the reference rendering image and the candidate rendering image; the difference information includes the content similarity and the difference heatmap.

[0128] In some embodiments, the similarity determination unit is specifically used to: determine a grayscale distribution matching score based on the difference in grayscale distribution between the reference rendering image and the candidate rendering image; slice the reference rendering image and the candidate rendering image according to their respective corresponding slice sizes; determine a slice number matching score based on the difference in the number of slices between the reference rendering image and the candidate rendering image; determine a region matching score based on the region similarity of at least one set of matching sub-regions between the reference rendering image and the candidate rendering image; and determine the content similarity between the reference rendering image and the candidate rendering image based on the grayscale distribution matching score, the slice number matching score, and the region matching score.

[0129] In some embodiments, the similarity determination unit is further configured to perform segmentation processing on the reference rendering image and the candidate rendering image according to segmentation requirements, to obtain multiple first sub-regions corresponding to the reference rendering image and multiple second sub-regions corresponding to the candidate rendering image. For any one first sub-region, the unit searches the candidate rendering image for the second sub-region with the highest content similarity to the first sub-region, forming a group of sub-regions; calculates the region similarity between the first sub-region and the second sub-region in the same group of sub-regions; and determines the content matching score based on the region similarity corresponding to at least one group of sub-regions.

[0130] In some embodiments, the heatmap generation unit is specifically used to adjust the reference rendering image and the candidate rendering image to the same image size; determine the pixel difference at the same pixel position in the adjusted reference rendering image and the candidate rendering image; determine the target pixel position where the pixel difference is greater than a preset value; determine the mask image corresponding to the reference rendering image, and set a target pixel value for the target pixel position in the mask image; map the mask image after setting the target pixel value to an initial heatmap, and superimpose the initial heatmap and the reference rendering image to obtain a difference heatmap.

[0131] Figure 7 The code generation device can execute Figure 1The implementation principle and technical effects of the code generation method described in the illustrated embodiments will not be repeated here. The specific methods by which each module and unit of the code generation device in the above embodiments performs its operations have been described in detail in the embodiments related to this method, and will not be elaborated upon here.

[0132] Figure 8 A schematic diagram of a model training apparatus provided for an exemplary embodiment of this application is shown. The apparatus includes: The first sample acquisition module 801 is used to acquire multiple first training data; the first training data includes first sample interface code and first sample reference rendering image corresponding to the first sample interface code. The perturbation operation module 802 is used to perform at least one code perturbation operation on the first sample interface code for any first training data, obtain perturbation interface code, and generate recovery guidance information based on the at least one code perturbation operation; The second rendering module 803 is used to render the disturbed interface code to generate a disturbed rendering image. The second difference determination module 804 is used to determine the first sample difference information between the first sample reference rendering image and the perturbation rendering image. The first training module 805 is used to train a code processing model using the perturbed interface code, the first sample difference information, the recovery guidance information, and the first sample interface code. The code processing model is used to iteratively execute at least one round of code update operations on the initial interface code corresponding to the interface visual file until the target interface code is obtained. The code update operation is used to update the interface code to be processed in the current round based on the difference information between the reference rendering image and the candidate rendering image corresponding to the interface visual file, so as to generate the interface code to be processed in the next round. The candidate rendering image is obtained by rendering the interface code to be processed in the current round.

[0133] In some embodiments, the first training data further includes first sample intermediate expression data corresponding to the first sample interface code; the perturbation operation module 802 is further configured to perform at least one expression perturbation operation on the first sample intermediate expression data to obtain perturbed intermediate expression data; the first training module 805 is further configured to train the code processing model using the first sample reference rendering image, the perturbed intermediate expression data, and the first sample interface code; the code processing model is further configured to generate the initial interface code based on the intermediate expression data generated by the interface visual file and the reference rendering image.

[0134] In some embodiments, the apparatus further includes a second training module for acquiring multiple second training data; the second training data includes a second sample reference rendering image and second sample intermediate representation data; for any one of the second training data, based on the second sample parameter rendering image and / or the second sample intermediate representation data, an initial candidate interface code is generated, and at least one round of sample code update operation is iteratively executed until a second update requirement is met, the sample code update operation including: rendering the candidate interface code to be processed in the current round to generate a candidate sample rendering image; the candidate interface code to be processed in the first round is the initial candidate interface code; determining second sample difference information between the second sample reference rendering image and the candidate sample rendering image; updating the candidate interface code using a code processing model based on the second sample difference information to generate the candidate interface code to be processed in the next round; determining reward information according to the second sample difference information corresponding to the sample code update operation meeting the second update requirement, and adjusting the model parameters of the code processing model according to the reward information.

[0135] In some embodiments, the second training module is specifically used to delete historical interaction data from the sample historical context data of the code processing model to obtain updated sample historical context data; the historical interaction data includes historical candidate interface code; based on the updated sample historical context data, the candidate interface code to be processed in the current round, the second sample reference rendering image, and the second sample difference information, the current sample context data is constructed; the current sample context data is input into the code processing model to update the candidate interface code.

[0136] In some embodiments, the first sample acquisition module 801 is specifically used to generate a first sample interface code based on a preset prompt instruction using a code generation model; and / or to filter the first sample interface code from the target interface code obtained by iteratively executing at least one round of code update operations from the code processing model; the difference information between the rendered image of the first sample interface code and the reference rendered image of the corresponding visual file meets the sample code requirements.

[0137] Figure 8 The model training device described above can perform Figure 3 The implementation principle and technical effects of the model training method described in the illustrated embodiments will not be repeated here. The specific methods by which each module and unit of the model training device in the above embodiments performs its operations have been described in detail in the embodiments related to this method, and will not be elaborated upon here.

[0138] Figure 9 A schematic diagram of a model training apparatus provided for an exemplary embodiment of this application is shown. The apparatus includes: The second sample acquisition module 901 is used to acquire multiple second training data; the second training data includes a second sample reference rendering image and second sample intermediate expression data. The second code generation module 902 is used to generate initial candidate interface code for any second training data based on the second sample parameter rendering graph and / or the second sample intermediate expression data.

[0139] The sample code update operation is iteratively executed at least once until the second update requirement is met. The sample code update operation is implemented through the following module: The third rendering module 903 is used to render the candidate interface code to be processed in the current round to generate a candidate sample rendering image; the candidate interface code to be processed in the first round is the initial candidate interface code. The third difference determination module 904 is used to determine the second sample difference information between the second sample reference rendering image and the candidate sample rendering image; The second code update module 905 is used to update the candidate interface code based on the second sample difference information using a code processing model, so as to generate the candidate interface code to be processed in the next round. The third training module 906 is used to determine reward information based on the second sample difference information corresponding to the sample code update operation meeting the second update requirement, and to adjust the model parameters of the code processing model based on the reward information. The code processing model is used to iteratively execute at least one round of code update operations on the initial interface code corresponding to the interface visual file until the target interface code is obtained. The code update operation is used to update the interface code to be processed in the current round based on the difference information between the reference rendering image and the candidate rendering image corresponding to the interface visual file, so as to generate the interface code to be processed in the next round. The candidate rendering image is obtained by rendering the interface code to be processed in the current round.

[0140] Figure 9 The model training device described above can perform Figure 4 The implementation principle and technical effects of the model training method described in the illustrated embodiments will not be repeated here. The specific methods by which each module and unit of the model training device in the above embodiments performs its operations have been described in detail in the embodiments related to this method, and will not be elaborated upon here.

[0141] Figure 10 This is a schematic diagram of the structure of one embodiment of a computing device provided in this application. Figure 10 As shown, in practice, the computing device may include a storage component 1001 and a processing component 1002.

[0142] Storage component 1001 is used to store computer programs and can be configured to store various other data to support operation on a computing device. Examples of this data include instructions for any application or method used to operate on the computing device, data structures, contact data, phone book data, messages, pictures, videos, etc.

[0143] Processing component 1002, coupled to storage component 1001, is used to execute computer programs in storage component 1001 for implementing, etc. Figure 1 The code generation method shown, or its implementation as follows Figure 3 Or the model training method shown in Figure 4.

[0144] Furthermore, such as Figure 10 As shown, the computing device may also include other components such as a communication component 1003, a display component 1004, a power supply component 1005, and an audio component 1006. Figure 10 The diagram only shows some components and does not mean that the device includes only these components. Figure 10 The components shown. Additionally... Figure 10 The components within the dashed box are optional, not mandatory, and their specific requirements depend on the product form of the computing device. The computing device in this embodiment can be a terminal device such as a desktop computer, laptop computer, smartphone, or IoT (Internet of Things) device, or a server-side device such as a conventional server, cloud server, or server array. If the computing device in this embodiment is implemented as a terminal device such as a desktop computer, laptop computer, or smartphone, it may include... Figure 10 The components within the dashed box; if the computing device in this embodiment is implemented as a conventional server, cloud server, or server array, etc., then it may not include... Figure 10 The component within the dashed box.

[0145] The processing component described above includes one or more processors to execute computer instructions to complete all or part of the steps in the method described above. Alternatively, the processing component may be implemented as one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field-programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors, or other electronic components to perform the method described above.

[0146] The aforementioned storage components can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as Static Random-Access Memory (SRAM), Electrically Erasable Programmable Read Only Memory (EEPROM), Erasable Programmable Read Only Memory (EPROM), Programmable Read-Only Memory (PROM), Read-Only Memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk.

[0147] The aforementioned communication component is configured to facilitate wired or wireless communication between the device housing the communication component and other devices. The device housing the communication component can access wireless networks based on communication standards, such as mobile communication networks, or combinations thereof. In one exemplary embodiment, the communication component receives broadcast signals or broadcast-related information from an external broadcast management system via a broadcast channel.

[0148] The aforementioned display components may include a screen, which may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen can be implemented as a touchscreen to receive input signals from the user. The touch panel includes one or more touch sensors to sense touches, swipes, and gestures on the touch panel. The touch sensors can sense not only the boundaries of touch or swipe actions but also the duration and pressure associated with the touch or swipe operation.

[0149] The aforementioned power supply components provide power to various components within the device in which they reside. These power supply components may include a power management system, one or more power sources, and other components associated with generating, managing, and distributing power to the device in which they reside.

[0150] The aforementioned audio component can be configured to output and / or input audio signals. For example, the audio component includes a microphone (MIC) configured to receive external audio signals when the device containing the audio component is in an operating mode, such as call mode, recording mode, or voice recognition mode. The received audio signals can be further stored in memory or transmitted via a communication component. In some embodiments, the audio component also includes a speaker for outputting audio signals.

[0151] Accordingly, embodiments of this application also provide a computer-readable storage medium storing a computer program, which, when executed by a processor, enables the processor to implement the steps in the above-described method embodiments. The computer-readable storage medium includes volatile or non-volatile components, or a combination thereof, and can be removable or non-removable. Examples of computer-readable storage media include, but are not limited to, phase-change random access memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random-access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), flash memory or other memory technologies, CD-ROM, Digital Video Disc (DVD) or other optical storage, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other non-transmission medium. Accordingly, this application also provides a computer program product, which includes a computer program or instructions that, when executed by a processor, cause the processor to implement the steps in the above method embodiments. It should be understood that each step or combination of steps in the above method flow can be implemented by the computer program or instructions. Furthermore, these computer programs or instructions can be applied to the processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device, enabling the processor of the general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing device to function as an apparatus for implementing the corresponding functions in the above method embodiments.

[0152] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0153] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0154] Finally, it should be noted that the above are merely embodiments of this application and are not intended to limit the scope of this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of the claims of this application.

Claims

1. A code generation method, characterized in that, include: Generate initial interface code based on the interface visual file; Iterate through at least one round of code update operations until the first update requirement is met to obtain the target interface code; The code update operation includes: The current round of pending interface code is rendered to generate candidate rendering images; the first round of pending interface code is the initial interface code. Determine the differences between the reference rendering image corresponding to the interface visual file and the candidate rendering image; Based on the difference information, the interface code is updated using a code processing model to generate the next round of interface code to be processed.

2. The method according to claim 1, characterized in that, The first update requirement includes: the code update operation is executed iteratively a preset number of times; Before generating candidate rendering images from the current round of pending interface code, the method further includes: Determine whether the number of iterations has reached the preset number; If the number of iterations reaches a preset number, the interface code to be processed in the current round will be used as the target interface code; If the preset number of iterations has not been reached, the current round of code update operations will continue.

3. The method according to claim 1, characterized in that, The first update requirement includes: the difference information determined by the code update operation meets the difference requirements; After determining the difference information between the reference rendering image corresponding to the interface visual file and the candidate rendering image, the method further includes: Determine whether the difference information meets the difference requirements; If the difference information meets the difference requirements, the interface code to be processed in the current round will be used as the target interface code. The step of updating the interface code using a code processing model based on the difference information to generate the next round of interface code to be processed includes: If the difference information does not meet the difference requirements, the interface code is updated based on the difference information using a code processing model to generate the next round of interface code to be processed.

4. The method according to claim 1, characterized in that, The process of generating initial interface code based on the interface visual file includes: Based on the interface visual file, intermediate representation data and the reference rendering image are generated; Based on the intermediate representation data and the reference rendering graph, first context data is constructed; The first context data is input into the code processing model to generate the initial interface code.

5. The method according to claim 1, characterized in that, The step of updating the interface code using a code processing model based on the difference information includes: Delete historical interaction data from the historical context data of the code processing model to obtain updated historical context data; the historical interaction data includes historical interface code to be processed. Based on the updated historical context data, the current round of pending interface code, the reference rendering image, and the difference information, a second context data is constructed. The second context data is input into the code processing model to update the interface code.

6. The method according to claim 1, characterized in that, The step of determining the difference information between the reference rendering image corresponding to the interface visual file and the candidate rendering image includes: Calculate the content similarity between the reference rendering image and the candidate rendering image corresponding to the interface visual file; A difference heatmap is generated based on the target pixel positions that differ between the reference rendering image and the candidate rendering image. The difference information includes the content similarity and the difference heatmap.

7. The method according to claim 6, characterized in that, The calculation of the content similarity between the reference rendering image and the candidate rendering image corresponding to the interface visual file includes: Based on the difference in grayscale distribution between the reference rendering image and the candidate rendering image, a grayscale distribution matching score is determined; The reference rendering image and the candidate rendering image are sliced ​​according to their respective slice sizes, and a slice number matching score is determined based on the difference in the number of slices between the reference rendering image and the candidate rendering image. A region matching score is determined based on the region similarity of at least one set of sub-regions that match the reference rendering image and the candidate rendering image. The content similarity between the reference rendering image and the candidate rendering image is determined based on the grayscale distribution matching score, the slice quantity matching score, and the region matching score.

8. The method according to claim 7, characterized in that, The step of determining the region matching score based on the region similarity of at least one set of sub-regions that match the reference rendering image and the candidate rendering image includes: According to the segmentation requirements, the reference rendering image and the candidate rendering image are segmented respectively to obtain multiple first sub-regions corresponding to the reference rendering image and multiple second sub-regions corresponding to the candidate rendering image. For any first sub-region, search the candidate rendering image for the second sub-region that has the greatest similarity to the content of the first sub-region, and form a group of sub-regions; Calculate the region similarity between the first and second sub-regions within the same group of sub-regions; The content matching score is determined based on the region similarity corresponding to at least one set of sub-regions.

9. The method according to claim 6, characterized in that, The step of generating a difference heatmap based on the target pixel locations that differ between the reference rendering image and the candidate rendering image includes: Adjust the reference rendering image and the candidate rendering image to have the same image size; Determine the pixel difference at the same pixel position in the adjusted reference rendering image and candidate rendering image; Determine the location of target pixels whose pixel difference is greater than a preset value; Determine the mask image corresponding to the reference rendering image, and set the target pixel value for the target pixel position in the mask image; The mask image after setting the target pixel value is mapped to the initial heatmap, and the initial heatmap and the reference rendering image are superimposed to obtain the difference heatmap.

10. A model training method, characterized in that, include: Obtain multiple initial training data sets; The first training data includes the first sample interface code and the first sample reference rendering image corresponding to the first sample interface code; For any given first training data, perform at least one code perturbation operation on the first sample interface code to obtain perturbed interface code, and generate recovery guidance information based on the at least one code perturbation operation; The disturbed interface code is rendered to generate a disturbed rendering image; Determine the first sample difference information between the first sample reference rendering image and the perturbation rendering image; The code processing model is trained using the perturbation interface code, the first sample difference information, the recovery guidance information, and the first sample interface code. The code processing model is used to iteratively execute at least one round of code update operations on the initial interface code corresponding to the interface visual file until the target interface code is obtained. The code update operation is used to update the interface code to be processed in the current round based on the difference information between the reference rendering image and the candidate rendering image corresponding to the interface visual file, so as to generate the interface code to be processed in the next round. The candidate rendering image is obtained by rendering the interface code to be processed in the current round.

11. The method according to claim 10, characterized in that, The first training data also includes the intermediate representation data of the first sample corresponding to the interface code of the first sample; The method further includes: Perform at least one expression perturbation operation on the intermediate expression data of the first sample to obtain perturbed intermediate expression data; The code processing model is trained using the first sample reference rendering image, the perturbation intermediate representation data, and the first sample interface code; the code processing model is also used to generate the initial interface code based on the intermediate representation data generated from the interface visual file and the reference rendering image.

12. The method according to claim 10, characterized in that, After training the code processing model using the perturbation interface code, the first sample difference information, the recovery guidance information, and the first sample interface code, the method further includes: Acquire multiple second training data sets; the second training data sets include second sample reference rendering images and second sample intermediate representation data. For any given set of second training data, based on the second sample parameter rendering map and / or the second sample intermediate representation data, initial candidate interface code is generated, and at least one round of sample code update operations is iteratively executed until the second update requirement is met. The sample code update operations include: The candidate interface code to be processed in the current round is rendered to generate a candidate sample rendering image; the candidate interface code to be processed in the first round is the initial candidate interface code; Determine the second sample difference information between the second sample reference rendering image and the candidate sample rendering image; Based on the second sample difference information, the candidate interface code is updated using a code processing model to generate the next round of candidate interface code to be processed. Based on the second sample difference information corresponding to the sample code update operation meeting the second update requirement, the reward information is determined, and the model parameters of the code processing model are adjusted according to the reward information.

13. The method according to claim 12, characterized in that, The step of updating the candidate interface code using a code processing model based on the second sample difference information includes: Delete historical interaction data from the sample historical context data of the code processing model to obtain updated sample historical context data; the historical interaction data includes historical candidate interface code; Based on the updated historical context data of the sample, the candidate interface code to be processed in the current round, the reference rendering image of the second sample, and the difference information of the second sample, the current context data of the sample is constructed. The current context data of the sample is input into the code processing model to update the candidate interface code.

14. The method according to claim 10, characterized in that, The code for the first sample interface was obtained as follows: Using a code generation model, the first sample interface code is generated based on preset prompts; And / or, From the target interface code obtained by iteratively executing at least one round of code update operations in the code processing model, select the first sample interface code; The difference information between the rendered image of the first sample interface code and the reference rendered image of the corresponding visual file meets the requirements of the sample code.

15. A model training method, characterized in that, include: Obtain multiple second training data sets; The second training data includes a second sample reference rendering image and second sample intermediate representation data; For any given set of second training data, based on the second sample parameter rendering map and / or the second sample intermediate representation data, initial candidate interface code is generated, and at least one round of sample code update operations is iteratively executed until the second update requirement is met. The sample code update operations include: The candidate interface code to be processed in the current round is rendered to generate a candidate sample rendering image; the candidate interface code to be processed in the first round is the initial candidate interface code; Determine the second sample difference information between the second sample reference rendering image and the candidate sample rendering image; Based on the second sample difference information, the candidate interface code is updated using a code processing model to generate the next round of candidate interface code to be processed. Based on the second sample difference information corresponding to the sample code update operation satisfying the second update requirement, reward information is determined, and the model parameters of the code processing model are adjusted according to the reward information. The code processing model is used to iteratively execute at least one round of code update operations on the initial interface code corresponding to the interface visual file until the target interface code is obtained. The code update operation is used to update the interface code to be processed in the current round based on the difference information between the reference rendering image and the candidate rendering image corresponding to the interface visual file, so as to generate the interface code to be processed in the next round. The candidate rendering image is obtained by rendering the interface code to be processed in the current round.

16. A computing device, characterized in that, This includes processing components and storage components; The storage component stores a computer program; the computer program is invoked and executed by the processing component to implement the code generation method as described in any one of claims 1-9, or to implement the model training method as described in any one of claims 10-15.

17. A computer-readable storage medium, characterized in that, It stores a computer program, which, when executed by a processing component, implements the code generation method as described in any one of claims 1-9, or the model training method as described in any one of claims 10-15.

18. A computer program product, characterized in that, It includes a computer program or instructions that, when executed by a processing component, implement the code generation method as described in any one of claims 1-9, or the model training method as described in any one of claims 10-15.

Images