Training data generation method, electronic device and computer program product

Abstract

The invention provides a training data generation method, electronic equipment and a computer program product. The training data generation method comprises the steps that target codes are rendered into target elements, graphic data of the target elements are obtained, the target codes record styles and functions of the target elements in a code mode, and the graphic data present the styles of the target elements in a picture mode; analyzing the target code and the graphic data to obtain text data which describes the style and the function of the target element in a text form; and obtaining training data including the target code, the graphic data and the text data.

Description

Technical Field

[0001] The present disclosure relates to technical fields such as data processing, and more particularly to a training data generating method, an electronic device, and a computer program product. Background Art

[0002] Large Language Models (LLMs) can generate high-quality code based on some simple instructions, but natural language text cannot accurately express all development requirements. Therefore, it is necessary to evolve the large language model into a model that can handle multimodal instructions, so that it can not only understand text instructions, but also parse and use graphic instructions to generate more accurate code. Undoubtedly, the core of training the model lies in obtaining high-quality multimodal training data. At present, although there is a certain amount of front-end code for the model to learn, there is very little graphic data to match it. Constructing multimodal training data containing "graphics-text-code" is one of the main challenges in obtaining a multimodal model. Summary of the invention

[0003] The present disclosure provides a training data generation method, an electronic device, and a computer program product.

[0004] According to one aspect of the present disclosure, a training data generation method is provided, comprising: rendering a target code as a target element, obtaining graphic data of the target element, wherein the target code records the style and function of the target element in code form, and the graphic data presents the style of the target element in picture form; analyzing the target code and the graphic data to obtain text data, wherein the text data describes the style and function of the target element in text form; and obtaining training data including the target code, the graphic data, and the text data.

[0005] In some embodiments, rendering the target code as a target element and obtaining graphic data of the target element include: identifying a dependency identifier in the target code, the dependency identifier being used to mark a dependency library, the dependency library recording at least part of the style information and / or at least part of the functional information of the target element; generating an initial framework, the initial framework providing a blank development environment for the target element; downloading the dependency library to the initial framework according to the dependency identifier; rendering the target element in the initial framework according to the dependency library and the target code; and obtaining the graphic data containing the target element, the graphic data including a photo or screenshot of the target element.

[0006] In some embodiments, downloading the dependent library to the initial framework includes: monitoring a first feedback result regarding the dependent library, the first feedback result recording a problem that occurs when the dependent library is downloaded to the initial framework; analyzing the first feedback result and the dependent library to determine a processing solution for resolving the first feedback result; and processing the dependent library according to the processing solution so that the dependent library can be downloaded to the initial framework.

[0007] In some embodiments, determining a processing solution for resolving the first feedback result includes: when the first feedback result is a version conflict problem, determining a compatible version that the existing library can support for the dependent library, the existing library is a dependent library that has been downloaded to the initial framework; and when the compatible version exists, generating a processing solution for downloading the compatible version of the dependent library; or, when the compatible version does not exist, abandoning the target code.

[0008] In some embodiments, determining a processing solution for resolving the first feedback result includes: when the first feedback result is that there is a derivative dependency problem, determining a derivative library related to the dependent library, wherein the derivative library is a database referenced by the dependent library; and generating a processing solution for downloading the derivative library.

[0009] In some embodiments, rendering the target element in the initial frame includes: monitoring second feedback information about the target element, the second feedback information being a problem that occurs when rendering the target element; correcting the target code and / or the dependent library according to the second feedback information; rendering the corrected target code based on the corrected dependent library; and rendering the target element in the initial frame; or, abandoning the target code when the number of renderings is greater than a threshold number.

[0010] In some embodiments, obtaining text data includes: generating prompt information including the style and function of the target element, the prompt information is a processing requirement issued to a first model, and the first model has the ability to generate text data according to the prompt information; synchronizing the target code, the graphic data and the prompt information to the first model, controlling the first model to output a first text; and verifying the first text, using the verified qualified first text as the text data.

[0011] In some embodiments, verifying the first text includes: calling the first model to analyze the similarity between the first text and the target code, the graphic data, and the prompt information; when the similarity is less than a similarity threshold, determining that the first text verification has failed; when the similarity is greater than or equal to the similarity threshold, determining that the first text verification has passed.

[0012] According to another aspect of the present disclosure, an electronic device is provided, comprising: a memory storing execution instructions; and a processor executing the execution instructions stored in the memory, so that the processor executes the training data generating method of any embodiment of the present disclosure.

[0013] According to another aspect of the present disclosure, a readable storage medium is provided, in which execution instructions are stored. When the execution instructions are executed by a processor, the training data generating method of any embodiment of the present disclosure is implemented.

[0014] According to another aspect of the present disclosure, a computer program product is provided, including a computer program, wherein when the computer program is executed by a processor, the training data generating method of any embodiment of the present disclosure is implemented. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] The accompanying drawings illustrate exemplary embodiments of the present disclosure and together with the description serve to explain the principles of the present disclosure. These drawings are included to provide a further understanding of the present disclosure and are incorporated in and constitute a part of this specification.

[0016] Figure 1 It is a schematic diagram of an application scenario of the training data generation method according to an embodiment of the present disclosure.

[0017] Figure 2 It is a flow chart of a training data generation method according to an embodiment of the present disclosure.

[0018] Figure 3 It is a schematic diagram of a training data generation process according to an embodiment of the present disclosure.

[0019] Figure 4 It is a schematic diagram of a process of obtaining graphic data according to an embodiment of the present disclosure.

[0020] Figure 5 It is a schematic diagram of a process of obtaining text data according to an embodiment of the present disclosure.

[0021] Figure 6 It is a schematic block diagram of the structure of a training data generating device according to an embodiment of the present disclosure.

[0022] Figure 7It is a schematic block diagram of the structure of an electronic device according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0023] The present disclosure is further described in detail below in conjunction with the accompanying drawings and examples. It is understood that the specific examples described herein are only used to explain the relevant content, rather than to limit the present disclosure. It should also be noted that, for ease of description, only the parts related to the present disclosure are shown in the accompanying drawings.

[0024] It should be noted that, in the absence of conflict, the embodiments and features of the embodiments in the present disclosure can be combined with each other. The technical solution of the present disclosure will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0025] Large Language Models (LLMs) can generate high-quality code based on some simple instructions, significantly improving productivity. However, natural language text cannot accurately express all development requirements. For example, in code generation scenarios involving graphical display, complex interaction, etc., it is difficult to accurately express development intent with text instructions alone. Therefore, it is necessary to provide more intuitive demand expressions through graphic instructions (such as renderings, line drawings, etc.). In order to solve the above problems, it is necessary to evolve the large language model into a model that can handle multimodal instructions, so that it can not only understand text instructions, but also parse and use graphic instructions to generate more accurate code. Undoubtedly, the core of training models lies in obtaining high-quality multimodal training data. At present, although there is a certain amount of front-end code for the model to learn, there is very little graphic data to match it. Constructing multimodal training data containing "graphics-text-code" is one of the main challenges in obtaining multimodal models.

[0026] To this end, the present disclosure proposes a training data generation method.

[0027] Figure 1 Schematic diagram of an application scenario of the training data generation method according to an embodiment of the present disclosure. Figure 1 As shown, in this application scenario, a server 100 and a terminal device 200 may be included. The server 100 and the terminal device 200 may be connected via a network or Bluetooth to exchange data. The server 100 may be a cloud server or a physical server, and the terminal device 200 may be a smart device such as a computer, a mobile phone, or a tablet. The server 100 is used to provide the basic data required to run the training data generation method, and the terminal device 200 executes the training data generation method of the present disclosure based on the basic data provided by the server 100.

[0028] For the convenience of description and to make the technical solutions of the specific implementation methods of the present disclosure easier to understand, before describing the code generation method implemented in the present disclosure, the technical terms involved in the specific implementation methods of the present disclosure are explained as follows: Target code: code used to render the target element, which records the style and function of the target element. Any code can become the target code when it is processed as training data. The target code can be a component code or page code in a public code library, etc. The target code includes functional logic code and style code. The functional logic code records the processing logic for implementing various functions in the target element, and the style design code records the design plan for implementing the specified style. The target code can also include some referenced external dependent libraries. For example, the element corresponding to the target code is a page, and the target code references an existing component, and the component code of the existing component is recorded in database A. Then the target code will record reference information to database A (such as the dependency identifier of the dependent library). At this time, database A is the dependent library of the target code.

[0029] Dependency library: It is an external database referenced by the target code, which records at least part of the style information and / or at least part of the functional information of the target element. Each target code can reference multiple dependency libraries, and there is no limit on the number of dependency libraries. When rendering the target element, it is necessary to call the corresponding data in the dependency library, so it is necessary to determine and download the dependency library to the rendering environment.

[0030] Target element: The display result obtained by rendering the target code, which can be a component or a page, etc. If the target element is a page, then the page can contain multiple components.

[0031] Graphic data: It is an image such as a screenshot or photo that contains the target element. Graphic data can record the style of the target element in the form of a graphic. When the target element is a component, the style includes the appearance of the component. When the target element is a page, the style includes the appearance of each component in the page and the layout of each component on the page.

[0032] Text data: It is natural language description information about the target element generated according to the target code and graphic data, and records the style and function of the target element in text form. The text data can be obtained through the first model, where the first model is a large language model.

[0033] Training data: is multimodal data including target code, graphic data and text data, and is the processing result of the method described in the present disclosure. The training data is used to train the multimodal model.

[0034] Figure 2 is a flow chart of a method for generating training data according to an embodiment of the present disclosure. Figure 2As shown, the present disclosure proposes a training data generation method M200, which obtains multimodal training data through steps S210 to S230, solving the problem of insufficient training data when training a multimodal model. The data of running the training data generation method M200 can be recorded in Figure 1 The method M200 is executed by the server 100 in the terminal device 200. It should be noted that the method M200 can be executed by the intelligent agent Agent carried in the terminal device 200. The Agent has the ability to autonomously run the method M200, and can call the large language model to assist in performing corresponding tasks in various processing links, such as calling the first model to generate text data.

[0035] In step S210, the target code is rendered into a target element, and graphic data of the target element is obtained.

[0036] The target code can be extracted from a public code library, which contains the style code and function code required to render the target element, or contains the identifier of a database that records the style and function of the target element, wherein the identifier of the database contained in the target code is used as the dependency identifier for the subsequent determination of the dependent library. The target code is the code used to render the target element. Any code can become the target code when it is processed as training data. The functional logic code of the target code records the processing logic for implementing various functions in the target element, and the style code records the design scheme for implementing the specified style. The target code can also include some referenced external dependent libraries. For example, the element corresponding to the target code is a page, and the target code references an existing component, and the component code of the existing component is recorded in database A. Then the target code will record reference information to database A (such as the dependency identifier of the dependent library). At this time, database A is the dependent library of the target code.

[0037] Since the target code is random, different codes may rely on different development environments when used as target codes. In order to avoid interference of other codes on the target code, it is necessary to provide a universally applicable and isolated development environment for the target code. Then, in this development environment, the target element is rendered in a conventional code rendering manner.

[0038] Furthermore, the rendered target element is subjected to operations such as screenshots, photographs, or screen recording to capture video frames containing the target element, to obtain graphic data containing the target element. The graphic data records the style of the target element in the form of a graphic. The target element can be an element such as a component or a page. If the target element is a component, the graphic data can record the appearance of the corresponding component, including color, shape, structure, etc.; if the target element is a page, the graphic data can record the appearance of each component in the page, as well as the layout of all components involved in the page.

[0039] Graphic data intuitively and accurately displays the style of the target element rendered by the target code in the form of graphics. In the scenario of controlling the multimodal model to generate code, graphic data is actually the expected result of the element style rendered by the code. Therefore, compared with text data in natural language form, graphic data can more comprehensively and accurately describe the style of the target element, especially the part of the target style that is difficult to describe in natural language.

[0040] In step S220, the target code and graphic data are analyzed to obtain text data.

[0041] The target code describes the style and function of the target element from the dimension of code, and the graphic data describes the style of the target element more accurately and comprehensively from the dimension of graphics. The text data is a description text generated by describing the style and function of the target element in natural language based on the analysis and understanding of the target code and graphic data. The text data can be generated manually or by the first model, and the method of obtaining it is not limited here.

[0042] If the target code and graphic data are analyzed by the first model, prompt information needs to be established in advance to put forward clear processing requirements to the first model, including the implementation logic of element style and element function, etc. Then, the target code, graphic data and prompt information are used as inputs of the first model, and the first model processes and analyzes the input data to obtain text data describing the style and function of the target element.

[0043] In step S230, training data including target code, graphic data and text data is obtained.

[0044] After obtaining the graphic data and text data, the image data, text data and target code are integrated into one training data. Obviously, the training data is multimodal (three modes such as code-graphics-text) training data.

[0045] By repeatedly processing multiple target codes using steps S210 to S230, a training data set can be obtained. The training data set is then used to train the multimodal model to obtain a multimodal model that can output a desired multimodal output result by analyzing various modal input data such as image data, text data, and target codes.

[0046] The training data generation method disclosed in the present invention renders the existing target code in an independent and non-interference development environment to obtain the graphic data of the target element. In view of the problems that may arise in the rendering process, such as various challenges in the download of dependent libraries and the rendering process of target elements, the present invention provides a series of solutions to ensure the success rate, feasibility and efficiency of rendering. In addition, the method comprehensively analyzes the target code and its corresponding graphic data by calling the first model to generate natural language text data describing the target element. Compared with the traditional manual writing method, these automatically generated text data have higher credibility, comprehensiveness and accuracy. The multimodal training data set constructed based on the above process integrates the target code, graphic data and natural language text data, providing strong data support for the training of multimodal models. This data set not only solves the problem of lack or insufficiency of training data, but also ensures the high quality and integrity of the training data provided, which greatly promotes the development and application of multimodal models. Through this method, the present invention overcomes the data bottleneck encountered in the previous multimodal model training, and at the same time improves the quality of training data, so that the model can more accurately learn and understand the relationship between different modes, and finally achieve more efficient model training and optimization.

[0047] Figure 3 Schematic diagram of the training data generation process according to an embodiment of the present disclosure. Figure 3 , intuitively showing that through Figure 2 The method for generating training data shown is a process for obtaining training data.

[0048] Find existing high-quality code in the public code library as the target code. The code can be component code or page code and other demonstrative front-end code, and can be developed in the React code framework.

[0049] The target code is the code used to render the target element. Any code can become the target code when it is processed as training data. The functional logic code of the target code records the processing logic for implementing various functions in the target element, and the style code records the design plan for implementing the specified style. The target code can also include some referenced external dependent libraries. For example, if the element corresponding to the target code is a page, and the target code references an existing component, and the component code of the existing component is recorded in database A, then the target code will record reference information to database A (such as the dependency identifier of the dependent library). At this time, database A is the dependent library of the target code.

[0050] The target code usually records the style and function of the target element, and the target element with certain functions and displaying the specified style can be obtained by rendering the target code. However, some target codes refer to the content recorded in the external database, which records at least part of the style or at least part of the function of the target element. In order to render the target element, it is necessary to locate and download the external database referenced by the target code. First, according to the reference information in the target code, the identifier of the referenced external database is determined. These external databases referenced by the target code are the dependent libraries of the target code, and their identifiers are the dependent identifiers. According to the code syntax rules, there are usually database identifiers referenced by fixed reference syntax in fixed positions in the target code (such as the first few lines of the code), and these database identifiers are used as dependent identifiers. Then, the corresponding dependent libraries are located according to these dependent identifiers. Among them, the dependent libraries and their numbers of each target code are different. If the element described by the target code is a page, then there may be multiple components that reuse existing components, and there may be multiple dependent libraries; if the element described by the target code is a component, then usually at most one existing component is reused, then the dependent library may be one or a few, and there is no restriction on the dependent library and its number. The dependency library is an external database referenced by the target code, which records at least part of the style information and / or at least part of the functional information of the target element. Each target code can reference multiple dependency libraries, and there is no limit on the number of dependency libraries. When rendering the target element, it is necessary to call the corresponding data in the dependency library, so it is necessary to determine and download the dependency library to the rendering environment.

[0051] Furthermore, it is necessary to download these dependent libraries to a development environment that can support the rendering of the code. Due to the randomness of the target code, the present disclosure does not fix the development environment, but configures an appropriate environment for the target code. According to the code syntax rules of the target code, determine the code framework for developing the target code, such as the React code framework. In order to simulate the real development environment, a standardized initial framework is created. In this initial framework, the rest of the code content is not included, and only the basic tools necessary for rendering the target element are provided, which ensures the isolation between the target element and the rest of the elements, avoids the interference of the functions of the rest of the elements on the rendering process of the target element, and improves the reliability and scalability of the rendering results.

[0052] Furthermore, N dependent libraries and target codes are downloaded to the initial framework, where N can be 0 or any positive integer. If N is 0, it means that the target code does not reference any external database. Then, based on the dependent libraries, the target code is rendered in the initial framework to obtain the target element; the graphic data is obtained by taking a screenshot or taking a photo of the target element. The graphic data is an image of the target element, and the graphic data at least records the style of the target element.

[0053] Figure 4 FIG. 1 is a schematic diagram of a process for obtaining graphic data according to an embodiment of the present disclosure. Figure 4 , the process of obtaining graphic data is explained in detail.

[0054] When the target code and N dependent libraries are downloaded to the initial framework, if the N dependent libraries can be successfully downloaded to the initial framework, an information box such as "Download Successful" will be generated, and subsequent rendering actions can be performed at this time; wherein, the content of the information box can be set arbitrarily and is not restricted here. On the contrary, if some dependent libraries cannot be successfully downloaded to the initial framework, or some related problems arise after the download, a first feedback result will be generated. The first feedback result records the various problems that occurred when downloading the dependent libraries. Then, a processing solution for the dependent libraries is generated based on the first feedback result. The target code is the code used to render the target element. Any code can become the target code when it is processed as training data. The functional logic code of the target code records the processing logic for implementing various functions in the target element, and the style code records the design solution for implementing the specified style. The target code may also include some referenced external dependent libraries. For example, if the element corresponding to the target code is a page, and the target code references an existing component, and the component code of the existing component is recorded in database A, then the target code will record reference information to database A (such as the dependency identifier of the dependent library). At this time, database A is the dependent library of the target code.

[0055] Specifically, the second model can be used to analyze the first feedback result and multiple dependent libraries to determine a processing solution; wherein the second model can be a large language model, the second model can be different from the first model, or it can be the first model. The second model has the ability to process the problems in the first feedback result and generate a processing solution.

[0056] When the first feedback result is that there is a version conflict problem, it means that the existing library in the initial framework is incompatible with the current version of the dependent library currently being downloaded. Among them, if a certain target code has multiple dependent libraries, then these dependent libraries will be downloaded to the initial framework one by one in a preset order, then the dependent library that has been downloaded to the initial framework is the existing library. Furthermore, it is necessary to use the second model to determine the compatible version that the existing library can support for the current dependent library. If a compatible version exists, then "download a compatible version of the dependent library" is used as processing solution A.

[0057] On the contrary, if the existing library is not compatible with any version of the current dependent library, that is, there is no compatible version, then the second model uses "abandoning the target code" as solution B. In other words, the multiple dependent libraries involved in the target code are not compatible with each other, and these dependent libraries cannot be downloaded at the same time in the same initial framework. Each dependent library records at least part of the style information or at least part of the functional information required to render the target element. Without any dependent library, the target code cannot be rendered as a target element, and graphic data about the target code cannot be generated. Therefore, it is necessary to abandon the target code to avoid abnormal training data and reduce the time spent on abnormal training data.

[0058] If the first feedback result is that there is a derivative dependency problem, it means that the target code's dependent library also has dependencies on other databases. For example, the target code depends on database A, and part of the content in database A is recorded in database B, then database B is the derivative library of the target code; if only database A is downloaded, the target element cannot be rendered due to the missing content recorded in database B, so the derivative library also needs to be downloaded to the initial framework. Based on the above, the second model will determine the derivative library of the target code based on the database identifier referenced by the dependent library, and then generate a processing solution C for "downloading the derivative library".

[0059] After determining the processing scheme, the dependent library is reprocessed according to the processing scheme. For example, for processing scheme A, a compatible version of the dependent library is downloaded to the initial framework, and the content of the dependent library incompatible with the existing library in the initial framework is deleted to achieve the effect of covering the incompatible version of the dependent library with the compatible version of the dependent library. For processing scheme B, the processing process of the target code is directly ended, and the remaining codes are traversed again. For processing scheme C, the derivative library is located and downloaded to the initial framework to supplement the content in the initial framework. According to the processing scheme output by the second model, it is ensured that the initial framework of the available target code has complete and compatible multiple dependent libraries, providing a basic environment for subsequent rendering of target elements. In addition, the target code with rendering problems is cleared in a timely manner to avoid the time spent on meaningless rendering.

[0060] The processing solution generated by the second model identifies, repairs and iterates the problems caused during the database download process, improving the rendering success rate of the target code. The code can render less than 3,000 elements from the initial 30,000 codes to 18,000 elements. This method has a strong error repair capability and ensures the stability and efficiency of code rendering.

[0061] Further, it is determined whether the number of renderings exceeds the number threshold. If it does not exceed the number threshold, the target code is rendered in the initial frame. Further, it is determined whether the rendering is successful. If the rendering is successful, a target element that meets the style and function requirements in the target code is obtained. Then, a screenshot or photo of the target element is taken to obtain graphic data containing the target element.

[0062] If the rendering of the target code fails, a second feedback result will be generated. The second feedback result records the problems that occurred when rendering the target element, such as syntax errors in the target code. Then, the third model is controlled to analyze the second feedback result, the target code, and the dependent library, determine the modification plan, and then repair the target code or dependent library according to the modification plan. Further, the number of renderings is increased by one based on the original number, and then it is determined whether the number of renderings exceeds the number threshold. If it does not exceed the number threshold, the target code is rendered again in the initial framework. If the rendering is successful, the graphic data of the target element is obtained in the aforementioned manner; if the rendering fails, a new second feedback result is generated, and the third model is called again to repair the target code and dependent library. Until the number of renderings exceeds the number threshold, the process of generating training data according to the target code is terminated, and the time spent in the problem code is reduced; or, until the graphic data of the target element is obtained within the range of the number threshold. Among them, the number threshold can be set according to demand, for example, five times; the third model can be a large language model, which has the ability to repair the target code or dependent library according to the second feedback result, and the third model can be the same model as the second model or the first model.

[0063] Through the above steps, the graphic data of the target code is obtained. The graphic data can record the style of the target element in the form of a graphic, including the appearance of the components involved, and / or the layout of each component. Taking the target element as the system login page as an example, it at least includes a "user name" component, a "password" component, a "verification code" component and a "confirmation" component; wherein the "user name" component includes a user name prompt box and a user name filling field, the "password" component includes a password prompt box and a password filling field, etc., the "verification code" component includes a verification code prompt box and other appearances, a verification code display area and a verification code filling field, etc., and the "confirmation" component includes a confirmation button and other appearances. In addition, the "user name" component, the "password" component, and the "verification code" component are arranged in the center of the page from top to bottom, and the "confirmation" component is located in the lower right corner of the page. From this example, it can be seen that the style of the target element includes the appearance of each component and the layout of each component.

[0064] It should be noted that the first feedback result and the second feedback information mentioned above are first recorded by the Agent in the error log, and then the Agent calls the corresponding model to perform analysis and repair according to the content in the error log.

[0065] Figure 5 Schematic diagram of the process of obtaining text data according to an embodiment of the present disclosure. Figure 5 , the process of obtaining text data is explained.

[0066] Further, the function of the target element is determined according to the target code, and the style of the target element is determined according to the target code and the graphic data, and prompt information that can express these functions and styles is created. The prompt information issues processing requirements to the first model in a language that can be recognized by the first model, and the target code and the graphic data are limited to the processing objects of the first model, and then the prompt information, the target code and the graphic data are provided to the first model as input data. The first model outputs the first text after analyzing and processing the prompt information, the target code and the graphic data. The target code is the code used to render the target element. Any code can become the target code when it is processed as training data. The functional logic code of the target code records the processing logic for implementing various functions in the target element, and the style code records the design scheme for implementing the specified style. The target code can also include some referenced external dependent libraries. For example, if the element corresponding to the target code is a page, and the target code references an existing component, and the component code of the existing component is recorded in database A, then the target code will record reference information to database A (such as the dependency identifier of the dependent library), and at this time, database A is the dependent library of the target code.

[0067] In order to improve the consistency between the first text and the input data and ensure the validity and availability of the training data, the prompt information, target code, graphic data and the first text are also used as new input data for the first model. At the same time, a new processing requirement including "determining the similarity between the first text and the prompt information, target code and graphic data" is formed, and the new processing requirement is also used as the content of the new input data of the first model. Then, the new input data is input into the first model, and the first model determines the similarity between the first text and the prompt information, target code and graphic data. If the similarity is greater than the similarity threshold, it means that the first text can meet the processing requirements of the prompt information, and its description content has a high similarity with the style and function of the target code and the style of the graphic data. At this time, the first text is verified to be qualified and the first text is used as training data.

[0068] Otherwise, it is proved that the first text cannot meet the processing requirements of the prompt information, or the content described therein is significantly different from at least one of the style and function of the target code and the style of the graphic data, and thus the verification thereof fails.

[0069] When the verification fails, a verification suggestion is generated. Then the verification suggestion and the input data are input into the first model again, and the first model is controlled to re-analyze and process the input data according to the verification suggestion to generate a new first text. Then, the aforementioned process of determining the similarity is repeated until the similarity is greater than the similarity threshold, and then the first text is used as training data. The specific value of the similarity threshold can be set based on experience or the fit accuracy requirement for the first text, and is not limited here.

[0070] After obtaining the text data, the graphic data, text data and target code are integrated to obtain the training data. The Agent saves these training data in a structured manner to form the standard data set format required for multimodal learning, ensuring that the information of each part can be efficiently and accurately used in the subsequent training process of the multimodal model. The training data includes multiple modal forms of code dimension, text dimension and graphic dimension. When training the multimodal model, the graphic data and text data can be used as the input of the multimodal model. According to the difference between the code output by the multimodal model and the target code, the multimodal model can be processed by adjusting parameters, so that it can output code similar to or identical to the target code. After obtaining the training data, it can be used to train the multimodal model, so that the model can be applied to code generation scenarios that are difficult to describe in natural language.

[0071] The training data generation method disclosed in the present invention renders the existing target code in an independent, non-interference development environment to obtain the graphic data of the target element; and, during the rendering process, it proposes solutions to various problems in the process of downloading dependent libraries and problems in the process of rendering target elements, so that the rendering process of target elements is feasible and efficient. In addition, the first model is called to perform a comprehensive analysis of the target code and graphic data to obtain text data in a natural language for describing the target element, which has higher credibility, comprehensiveness and accuracy than manually written text data. Based on the foregoing, the constructed multimodal training data containing target code, graphic data and text data provides data support for the training of the multimodal model, overcomes the training problem of no training data or little training data, and the provided training data is credible and complete.

[0072] Figure 6 FIG. 1 is a schematic block diagram of a training data generating device according to an embodiment of the present disclosure. Figure 6 As shown, the present disclosure provides a training data generating device 600, which can be set in Figure 1 In the terminal device 200 shown in FIG. 1 , various data required for its operation can be recorded in Figure 1The apparatus 600 includes: a graphic data acquisition module 610, a text data acquisition module 620 and a training data generation module 630.

[0073] Among them, the graphic data acquisition module 610 is used to render the target code into the target element and obtain the graphic data of the target element, the target code records the style and function of the target element in the form of code, and the graphic data presents the style of the target element in the form of pictures; the text data acquisition module 620 is used to analyze the target code and graphic data to obtain text data, and the text data describes the style and function of the target element in text form; and the training data generation module 630 is used to obtain training data including the target code, graphic data and text data.

[0074] The training data generating device of the present disclosure may be in the form of computer software, and each module of the training data generating device may be in the form of a computer software module.

[0075] The various modules of the training data generating device 600 are configured to implement the various steps of the training data generating method. Therefore, the execution principle and process thereof can refer to the description of the method above and will not be described in detail here.

[0076] Figure 7 The electronic device 1000 can execute the training data generation method of any embodiment described above in the present disclosure.

[0077] like Figure 7 As shown, the hardware structure of the electronic device 1000 can be implemented using a bus architecture. The bus architecture can include any number of interconnected buses and bridges, depending on the specific application and overall design constraints of the hardware. The bus 1100 connects various circuits including one or more processors 1200, memory 1300 and / or hardware modules together. The bus 1100 can also connect various other circuits 1400 such as peripherals, voltage regulators, power management circuits, external antennas, etc.

[0078] The bus 1100 may be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Component (EISA) bus, etc. The bus may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, the figure only uses one connecting line, but does not mean that there is only one bus or one type of bus.

[0079] The present disclosure also provides a readable storage medium, in which a computer program is stored, and the computer program is used to implement the above method when executed by a processor. "Readable storage medium" can be any device that can contain, store, communicate, propagate or transmit a program for use in an instruction execution system, device or equipment or in combination with these instruction execution systems, devices or equipment. More specific examples of readable storage media include the following: an electrical connection portion (electronic device) with one or more wirings, a portable computer disk box (magnetic device), a random access memory (RAM), a read-only memory (ROM), an erasable and editable read-only memory (EPROM or flash memory), an optical fiber device, and a portable read-only memory (CDROM), etc.

[0080] The present disclosure also provides a computer program product. The method of the present disclosure can be implemented in whole or in part by software, hardware, firmware, or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer programs or instructions. When the computer program or instruction is loaded and executed, the process or function of the present disclosure is executed in whole or in part.

[0081] The computer program or instructions may be stored in a readable storage medium or transmitted from one readable storage medium to another readable storage medium, for example, the computer program or instructions may be transmitted from one website, computer, server or data center to another website, computer, server or data center by wired or wireless means. The readable storage medium may be any available medium that can be accessed or a data storage device such as a server, data center, etc. that integrates one or more available media. The available medium may be a magnetic medium, such as a floppy disk, a hard disk, or a magnetic tape; it may also be an optical medium, such as a digital video disk; it may also be a semiconductor medium, such as a solid state drive. The computer readable storage medium may be a volatile or non-volatile storage medium, or may include both volatile and non-volatile types of storage media.

[0082] Those skilled in the art will appreciate that the embodiments of the present disclosure may be provided as methods, systems, or computer program products. Therefore, the present disclosure may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present disclosure may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0083] The present disclosure is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to the present disclosure. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0084] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0085] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0086] In the description of this specification, the description with reference to the terms "one embodiment / method", "some embodiments / methods", "example", "specific example", or "some examples" etc. means that the specific features, structures, or characteristics described in conjunction with the embodiment / method or example are included in at least one embodiment / method or example of the present disclosure. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment / method or example. Moreover, the specific features, structures, or characteristics described may be combined in any one or more embodiments / methods or examples in a suitable manner. In addition, those skilled in the art may combine and combine different embodiments / methods or examples described in this specification and features of different embodiments / methods or examples, unless they are contradictory.

[0087] In addition, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of the features. In the description of the present disclosure, "plurality" means at least two, such as two, three, etc., unless otherwise clearly and specifically defined.

[0088] Those skilled in the art should understand that the above embodiments are only for the purpose of clearly illustrating the present disclosure, and are not intended to limit the scope of the present disclosure. For those skilled in the art, other changes or modifications may be made based on the above disclosure, and these changes or modifications are still within the scope of the present disclosure.

Claims

1. A method for generating training data, characterized in that: include: Rendering the target code into a target element to obtain graphic data of the target element, wherein the target code records the style and function of the target element in the form of code, and the graphic data presents the style of the target element in the form of a picture; Analyzing the target code and the graphic data to obtain text data, wherein the text data describes the style and function of the target element in text form; as well as Obtain training data including target code, graphic data, and text data.

2. The training data generation method according to claim 1, characterized in that: Rendering the target code into a target element and obtaining graphic data of the target element includes: Identify a dependency identifier in the target code, where the dependency identifier is used to mark a dependency library, where the dependency library records at least part of the style information and / or at least part of the function information of the target element; Generate an initial framework, wherein the initial framework provides a blank development environment for the target element; According to the dependency identifier, download the dependency library to the initial framework; Rendering the target element in the initial frame according to the dependent library and the target code; and The graphic data containing the target element is acquired, where the graphic data includes a photo or a screenshot of the target element.

3. The training data generation method according to claim 2, characterized in that: Downloading the dependent library to the initial framework includes: Monitoring a first feedback result about the dependent library, where the first feedback result records a problem that occurs when the dependent library is downloaded to the initial framework; Analyze the first feedback result and the dependent library to determine a solution to resolve the first feedback result; and The dependent library is processed according to the processing scheme so that the dependent library can be downloaded to the initial framework.

4. The training data generation method according to claim 3, characterized in that: Determine a solution to the first feedback result, including: In the case where the first feedback result is that there is a version conflict problem, determining a compatible version that the existing library can support for the dependent library, the existing library being the dependent library that has been downloaded to the initial framework; and When the compatible version exists, a processing solution for downloading the dependent library of the compatible version is generated; or, when the compatible version does not exist, the target code is abandoned.

5. The training data generation method according to claim 3, characterized in that: Determine a solution to the first feedback result, including: In the case where the first feedback result is that there is a derivative dependency problem, determining a derivative library related to the dependent library, the derivative library being a database referenced by the dependent library; and A processing scheme for downloading the derivative library is generated.

6. The training data generation method according to claim 2, characterized in that: Rendering the target element in the initial frame includes: monitoring second feedback information about the target element, where the second feedback information is a problem that occurs when rendering the target element; According to the second feedback information, modify the target code and / or the dependent library; Based on the modified dependent library, rendering the modified target code; and Rendering the target element in the initial frame; or, when the number of rendering times is greater than a threshold, abandoning the target code.

7. The training data generation method according to claim 1, characterized in that: Get text data, including: Generate prompt information including the style and function of the target element, wherein the prompt information is a processing requirement issued to the first model, and the first model has the ability to generate text data according to the prompt information; Synchronizing the target code, the graphic data and the prompt information to the first model, and controlling the first model to output a first text; and The first text is verified, and the verified first text is used as the text data.

8. The training data generation method according to claim 7, characterized in that: Verifying the first text includes: Calling the first model to analyze the similarity between the first text and the target code, the graphic data and the prompt information; When the similarity is less than a similarity threshold, it is determined that the first text verification fails; when the similarity is greater than or equal to the similarity threshold, it is determined that the first text verification passes.

9. An electronic device, characterized in that: include: A memory storing execution instructions; as well as A processor, wherein the processor executes the execution instructions stored in the memory, so that the processor executes the training data generation method according to any one of claims 1 to 8.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the training data generating method according to any one of claims 1 to 8 is implemented.

Images