Method, apparatus, computing device cluster and storage medium for generating web page code

Abstract

The present application discloses a method, apparatus, computing device cluster and storage medium for generating web page code, belonging to the field of computer technology. The method includes: obtaining a first web page image, where the first web page image is an image corresponding to the design draft of the first web page; performing image segmentation on the first web page image to obtain a plurality of components; predicting, through a relationship prediction network, the parent-child relationships among the plurality of components; and generating the code of the first web page, where the code of the first web page includes code for implementing the parent-child relationships among the plurality of components. The web page code generated by the present application can correctly reflect the parent-child relationships among the various components in the first web page image, thereby improving the accuracy of the web page code of the generated first web page.

Description

Technical Field

[0001] This application relates to the field of computer technologies, and particularly to a method, an apparatus, a computing device cluster, and a storage medium for generating web page code. Background Art

[0002] In the web page development process, a static web page design drawing is first created, and then the web page design drawing is converted into web page code.

[0003] In related technologies, with the maturity of automation and artificial intelligence technologies, many code generation tools have begun to attempt to solve the problem of automatically generating web page code from web page design drawings. However, these tools can usually only recognize individual components in the design drawings. For example, these tools may process icons, text, and buttons in the design drawings separately, and there is room for improvement in the accuracy of the web page code generated by related technologies. Summary of the Invention

[0004] This application provides a method, an apparatus, a computing device cluster, and a storage medium for generating web page code. The web page code generated by this application can correctly reflect the parent-child relationships between various components in the first web page image, thereby improving the accuracy of the web page code of the generated first web page. The technical solutions are as follows.

[0005] In a first aspect, a method for generating web page code is provided. The method includes the following steps.

[0006] Obtain a first web page image, where the first web page image is an image corresponding to the design draft of the first web page; perform image segmentation on the first web page image to obtain multiple components; predict the parent-child relationships between the multiple components through a relationship prediction network; generate the code of the first web page, where the code of the first web page includes code for implementing the parent-child relationships between the multiple components.

[0007] In this application, the parent-child relationships between multiple components are predicted through a relationship prediction network, and then the web page code of the first web page is generated. Since the web page code of the first web page includes the code for implementing the parent-child relationships between multiple components, the web page code generated in this application can correctly reflect the parent-child relationships between the various components in the first web page image, and the accuracy of the web page code is relatively high. Moreover, the method provided in this application can be used to generate web page codes in various programming languages, such as HTML (hypertext markup language), JavaScript (an interpreted scripting language), and so on. The method provided in this application has universality. Additionally, in this application, the web page code of the first web page is not directly generated by a neural network, but the parent-child relationships are predicted by a relationship prediction network and then the web page code of the first web page is generated. This reduces the dependence of the output of the relationship prediction network on the quality of the generated web page code, making the parent-child relationships output by the relationship prediction network more accurate and standardized.

[0008] Optionally, the relationship prediction network includes a feature extraction layer and a prediction layer; the predicting the parent-child relationships between multiple components through the relationship prediction network includes: performing feature extraction on each of the multiple components through the feature extraction layer to obtain multiple first image features; and performing feature extraction on the first web page image through the feature extraction layer to obtain a second image feature; based on the multiple first image features and the second image feature, predicting, through the prediction layer, the parent component of each component among the multiple components, and / or predicting the child component of each component among the multiple components.

[0009] In this application, predicting the parent component and / or child component of each component based on the image feature of each component and the image feature of the first web page provides a component prediction method. Moreover, the relationship prediction network adopted in this application can be a classification network or a regression network. In this application, the classification network or the regression network is used to predict the parent-child relationships between multiple components and then generate the web page code. Compared with generating the web page code using a large language model, since the large language model needs to use a large number of learning samples to fix a large number of network parameters, while the classification network or the regression network adopted in this application reduces the model training cost, and the classification network or the regression network requires fewer training samples and fewer network parameters to be adjusted. Additionally, the relationship prediction network in this application includes a feature extraction layer and a prediction layer. The structure of the relationship prediction network is simple and convenient to deploy. The relationship prediction network is extremely efficient in generating the parent-child relationships between multiple components. The structure of the relationship prediction network is clear and the format is concise, and it has high scalability.

[0010] Optionally, the method further includes: for each component among the multiple components, identifying the position feature and size feature of the component in the first web page image; the predicting, by a prediction layer, the parent component of each component among the multiple components and / or the child component of each component among the multiple components based on the multiple first image features and the second image features includes: splicing, by the prediction layer, the position feature, size feature, and first image feature of the component to obtain a first splicing feature; and predicting, by the prediction layer based on the multiple first splicing features corresponding to the multiple components and the second image features, the parent component of each component among the multiple components and / or the child component of each component among the multiple components.

[0011] In this application, a first splicing feature will also be obtained by combining the first image feature, position feature, and size feature of each component, and then the parent component and / or child component of each component will be predicted based on the multiple first splicing features corresponding to the multiple components and the second image features. This application provides a component prediction method, which makes the prediction result output by the relationship prediction network more accurate by using more feature information.

[0012] Optionally, generating the code of the first web page includes: generating component code for each component among the multiple components, where the component code includes code for implementing the parent-child relationship corresponding to the component; and assembling the multiple component codes of the multiple components to obtain the web page code of the first web page.

[0013] In this application, the component code of each component will be generated first and then assembled to obtain the web page code of the first web page. This application splits the task of generating the web page code of the first web page into the task of generating the component codes of multiple components, reducing the difficulty of generating the web page code.

[0014] Optionally, generating component code for each component among the multiple components includes: for each component among the multiple components, determining the padding attribute and margin attribute of the component, where the padding attribute is used to represent the distance from the component to its child component, and the margin attribute is used to represent the distance from the component to its parent component; and generating component code through a front-end code generator, where the component code includes code for implementing the padding attribute and margin attribute of each component.

[0015] In this application, the parent-child relationship between the multiple predicted components is converted into the padding and margin attributes of each component, and then the component code of each component is generated. The component code includes code for implementing the padding and margin attributes of each component. Therefore, the generated first web page code can correctly reflect the distance between the multiple components, the logical relationship of the generated first web page is more compact, and the generated first web page can correctly express the design logic of the first web page image.

[0016] Optionally, the method further includes: for each of the multiple components, identifying other CSS (cascading style sheets) attributes of the component, where the component code further includes code for implementing the other CSS attributes of the component, and the other CSS attributes include attributes other than the padding attribute and the margin attribute in the CSS attributes.

[0017] In this application, component code will be generated based on the padding attribute, margin attribute, and other CSS attributes of each component, and then the multiple component codes corresponding to the multiple components will be combined to obtain web page code. Therefore, the method for generating web page code provided in this application can generate various first web pages, and the overall layout and style of the first web page are affected by the rich CSS attributes of each component.

[0018] Optionally, the method further includes: obtaining a sample web page image, where the sample web page image is an image corresponding to the design draft of the sample web page; performing image segmentation on the sample web page image to obtain multiple sample components; predicting the parent-child relationship between the multiple sample components through a relationship prediction network to obtain a first parent-child relationship; training the relationship prediction network based on the gap between the first parent-child relationship and the second parent-child relationship, where the second parent-child relationship is the parent-child relationship between the multiple sample components obtained by annotation.

[0019] In this application, the relationship prediction network will be trained based on the gap between the first parent-child relationship and the second parent-child relationship. That is, this application provides a training method for a supervised relationship prediction network, reducing the training difficulty of the relationship prediction network.

[0020] Optionally, the relationship prediction network includes a feature extraction layer and a prediction layer. The step of predicting the parent-child relationship between the multiple sample components through the relationship prediction network to obtain a first parent-child relationship includes: for a masking process, performing a masking operation on one of the multiple sample components in the sample web page image to obtain a masked image; through performing multiple masking processes, obtaining multiple masked images corresponding one by one to the multiple sample components; respectively performing feature extraction on the multiple masked images through the feature extraction layer to obtain multiple third image features, and performing feature extraction on the sample web page image through the feature extraction layer to obtain a fourth image feature; based on the multiple third image features and the fourth image feature, predicting the first parent-child relationship through the prediction layer.

[0021] In this application, a relationship prediction network will be trained using a masking operation. For each sample component among multiple sample components, the relationship prediction network can learn not only the ability to predict parent / child components from the masked image corresponding to the current sample component and the sample web page image, but also the visual features, position features, and size features of the current sample component. That is, through the training method of the masking operation, the model ability of the relationship prediction network is improved, and the trained relationship prediction network can be better applied to the inference process.

[0022] In a second aspect, a device for generating web page code is provided. The device for generating web page code has the function of implementing the method behavior described in the first aspect above. The device for generating web page code includes at least one module, and this at least one module is used to implement the method provided in the first aspect above. The at least one module includes an acquisition module, an image segmentation module, a prediction module, and a generation module.

[0023] The acquisition module is used to acquire a first web page image, and the first web page image is an image corresponding to the design draft of the first web page; the image segmentation module is used to perform image segmentation on the first web page image to obtain multiple components; the prediction module is used to predict the parent-child relationships between the multiple components through the relationship prediction network; the generation module is used to generate the code of the first web page, and the code of the first web page includes the code for implementing the parent-child relationships between the multiple components.

[0024] Optionally, the relationship prediction network includes a feature extraction layer and a prediction layer; the prediction module is further used to perform feature extraction on each of the multiple components through the feature extraction layer to obtain multiple first image features; and, perform feature extraction on the first web page image through the feature extraction layer to obtain a second image feature; based on the multiple first image features and the second image feature, predict the parent component of each component among the multiple components and / or predict the child component of each component among the multiple components through the prediction layer.

[0025] Optionally, the device further includes an identification module. The identification module is used to identify the position features and size features of each component in the first web page image for each of the multiple components; the prediction module is further used to splice the position features, size features, and first image features of the component through the prediction layer to obtain a first spliced feature; based on the multiple first spliced features corresponding to the multiple components and the second image feature, predict the parent component of each component among the multiple components and / or predict the child component of each component among the multiple components through the prediction layer.

[0026] Optionally, the generation module is further used to generate component code for each of the multiple components, and the component code includes the code for implementing the corresponding parent-child relationship of the component; assemble the multiple component codes of the multiple components to obtain the web page code of the first web page.

[0027] Optionally, the generating module is further configured to determine, for each of the multiple components, the padding attribute and the margin attribute of the component, where the padding attribute is used to characterize the distance from the sub-components of the component, and the margin attribute is used to characterize the distance from the parent component of the component; generate component code through a front-end code generator, and the component code includes code for implementing the padding attribute and the margin attribute of each component.

[0028] Optionally, the identifying module is further configured to identify, for each of the multiple components, other CSS (cascading style sheets) attributes of the component, and the component code further includes code for implementing the other CSS attributes of the component, where the other CSS attributes include attributes other than the padding attribute and the margin attribute in the CSS attributes.

[0029] Optionally, the obtaining module is further configured to obtain a sample web page image, where the sample web page image is an image corresponding to the design draft of the sample web page; the image segmentation module is further configured to perform image segmentation on the sample web page image to obtain multiple sample components; the prediction module is further configured to predict the parent-child relationship between the multiple sample components through a relationship prediction network to obtain a first parent-child relationship; the training module is further configured to train the relationship prediction network based on the gap between the first parent-child relationship and the second parent-child relationship, where the second parent-child relationship is the parent-child relationship between the multiple sample components obtained by annotation.

[0030] Optionally, the relationship prediction network includes a feature extraction layer and a prediction layer; the prediction module is further configured to, for a masking process, perform a masking operation on a sample component among the multiple sample components in the sample web page image to obtain a masked image; by performing multiple masking processes, obtain multiple masked images corresponding one by one to the multiple sample components; perform feature extraction on the multiple masked images respectively through the feature extraction layer to obtain multiple third image features, and perform feature extraction on the sample web page image through the feature extraction layer to obtain a fourth image feature; based on the multiple third image features and the fourth image feature, predict the first parent-child relationship through the prediction layer.

[0031] In a third aspect, a computing device is provided, where the computing device includes a processor, and the processor is coupled to a memory, and the memory is used to store a computer program for executing the method provided in the first aspect above. The processor is configured to execute the computer program stored in the memory to implement the method described in the first aspect above.

[0032] Optionally, the computing device may further include a communication bus, and the communication bus is used to establish a connection between the processor and the memory.

[0033] Fourthly, a computing device cluster is provided, including at least one computing device, and each computing device includes a processor, with the processor being coupled to a memory.

[0034] The processor of the at least one computing device is configured to execute instructions stored in the memory, so that the computing device cluster executes the method described in the first aspect above.

[0035] Fifthly, a computer-readable storage medium is provided, in which instructions are stored, and when the instructions run on a computing device, the computing device is caused to execute the method described in the first aspect above.

[0036] Sixthly, a computer program product including instructions is provided, and when the instructions run on a computing device, the computing device is caused to execute the method described in the first aspect above.

[0037] The technical effects obtained in the second to sixth aspects above are similar to those obtained by the corresponding technical means in the first aspect, and will not be elaborated here. In addition, the device for generating web page code mentioned in the second aspect above may be the computing device mentioned in the third aspect, or the computing device cluster mentioned in the fourth aspect above. Description of the Drawings

[0038] Figure 1 is a schematic diagram of a web page code generation architecture provided by an embodiment of the present application;

[0039] Figure 2 is a flowchart of a web page code generation method provided by an embodiment of the present application;

[0040] Figure 3 is a schematic diagram of an inference method of a relationship prediction network provided by an embodiment of the present application;

[0041] Figure 4 is a flowchart of a web page code generation method provided by another embodiment of the present application;

[0042] Figure 5 is a schematic diagram of a training method of a relationship prediction network provided by an embodiment of the present application;

[0043] Figure 6 is a schematic diagram of another training method of a relationship prediction network provided by an embodiment of the present application;

[0044] Figure 7 is a schematic diagram of the structure of a web page code generation device provided by an embodiment of the present application;

[0045] Figure 8It is a schematic structural diagram of a computing device provided by an embodiment of the present application;

[0046] Figure 9 It is a schematic structural diagram of a computing device cluster provided by an embodiment of the present application;

[0047] Figure 10 It is a schematic structural diagram of a computing device cluster provided by another embodiment of the present application. Detailed implementation manners

[0048] To make the objectives, technical solutions, and advantages of the embodiments of the present application clearer, the following will further describe the embodiments of the present application in detail with reference to the accompanying drawings.

[0049] First, some terms involved in the embodiments of the present application will be briefly introduced.

[0050] Web page code generation process: In the web page development process, it refers to the process of first creating a static web page design drawing and then converting the web page design drawing into web page code. In the traditional web page code generation process, the division of labor between designers and developers is relatively clear. Designers use drawing tools such as Photoshop, Sketch, Figma, Adobe XD, etc. to create static web page design drawings, and then developers manually convert the web page design drawings into HTML and JavaScript code. However, this process has the following pain points.

[0051] 1. Repetitive labor: Developers need to manually implement each element (such as buttons, forms, pictures) in the web page design drawing as code, consuming a lot of time and energy. Especially when the web page design drawing changes frequently, developers need to modify it repeatedly. 2. Communication barriers: There may be problems with poor communication between designers and developers, resulting in the code implementation effect not matching the expectation or requiring multiple modifications, increasing the development cycle. 3. Version issues: Synchronization management between the version update of the web page design drawing and code generation also brings difficulties. Especially in multi-person collaboration projects, each time the web page design drawing is updated, it is necessary to synchronously allocate the development tasks of multiple developers. 4. Difficulty in responsive design: The adaptation of the generated web page code to different device sizes often requires developers to handle manually, further increasing the workload of front-end development.

[0052] In recent years, with the maturity of automation and artificial intelligence technologies, many researches and products have started to attempt to solve the problem of automatic generation from web design diagrams to code. However, although the automatic generation tools for web code in related technologies can significantly improve development efficiency and shorten the transition time from web design to web development, they still have some problems and limitations. It is difficult for related technologies to accurately identify the relationships between various components on the web design diagram and generate corresponding code, while the present application provides a technology that can accurately generate web code based on the web design diagram.

[0053] Figure 1 The figure shows a schematic diagram of the generation architecture of web code provided by an exemplary embodiment of the present application. Figure 1 The generation architecture of the web code in the figure includes three stages: an image segmentation stage 10, a relationship prediction stage 11, and a code generation stage 12.

[0054] Exemplarily, as Figure 1 shown, in the image segmentation stage 10, image segmentation is performed on the obtained first web image 101 to obtain multiple components 102 in the first web image 101. The first web image 101 is an image corresponding to the design draft of the first web page, that is, the first web image 101 is the web design diagram of the first web page.

[0055] The relationship prediction stage 11 is a stage for performing relationship prediction based on a relationship prediction network. In the relationship prediction stage 11, the first web image 101 and multiple components 102 are input into the relationship prediction network, and the parent-child relationships 105 between the multiple components are predicted through the relationship prediction network. Optionally, the parent-child relationships 105 between the multiple components include the hierarchical relationships between the multiple components 102. Exemplarily, if a div component contains two txt components, then the parent component of the two txt components is the div component. In one embodiment, the first web image 101 and multiple components 102 are input into the relationship prediction network, and the parent component of each component is predicted through the relationship prediction network, and / or the child component of each component is predicted, and the parent-child relationships of the multiple components 102 as a whole are obtained by integrating the parent component and / or child component of each component.

[0056] Optionally, the relationship prediction network is a hierarchical implicit neural network. Exemplarily, as Figure 1As shown in the figure, the relationship prediction network includes a feature extraction layer and a prediction layer, and both the feature extraction layer and the prediction layer have at least one hidden layer. In the relationship prediction stage 11, multiple components 102 and the first web page image 101 output by the image segmentation stage 10 are input into the feature extraction layer. The feature extraction layer performs feature extraction on each of the multiple components 102 respectively to obtain multiple first image features 103, and each first image feature 103 is used to characterize the image feature of a component; and, the feature extraction layer performs feature extraction on the first web page image 101 to obtain a second image feature 104. In the embodiments of the present application, the feature extraction layer is used to extract image features, and the embodiments of the present application do not limit the type of neural network used for performing feature extraction. Optionally, the feature extraction layer can be a convolutional neural network (CNN), a residual network (Resnet), etc.

[0057] The multiple first image features 103 and the second image feature 104 are input into the prediction layer, and the prediction layer predicts the parent-child relationships 105 between the multiple components. In the embodiments of the present application, the prediction layer is used to predict the parent-child relationships 105 between the multiple components, and the embodiments of the present application do not limit the type of neural network used for performing relationship prediction. Optionally, the prediction layer can be a fully connected neural network (FCNN), etc. The prediction layer outputs the parent-child relationships 105 between the multiple components to the code generation stage 12.

[0058] In the code generation stage 12, the web page code of the first web page is generated, and the web page code of the first web page includes the code for implementing the parent-child relationships 105 between the multiple components. In one embodiment, in the code generation stage 12, the component code of each component among the multiple components 102 of the first web page is generated, and the multiple component codes corresponding to the multiple components are combined to obtain the web page code of the first web page. The component code of each component is used to implement the parent-child relationship corresponding to the current component. Exemplarily, the component code of each component is used to implement the padding property and the margin property of the current component. The padding property is used to characterize the distance from the current component to its child components, and the margin property is used to characterize the distance from the current component to its parent component.

[0059] Optionally, the above web page code generation architecture is applied to a web page code generation tool. The web page code generation tool can be an extension component of a front-end web page designer and be embedded in the front-end web page designer, or the web page code generation tool can be an independent code generation tool for users to use.

[0060] Optionally, the above-mentioned web page code generation architecture is applied to a computer device, which includes at least one of a terminal and a server. The device types of terminal devices include at least one of smartphones, smart watches, vehicle-mounted terminals, wearable devices, smart TVs, tablets, e-book readers, players, laptop computers, and desktop computers. Terminal devices include, but are not limited to, mobile phones, computers, intelligent voice interaction devices, smart home appliances, vehicle-mounted terminals, aircraft, etc.

[0061] In some embodiments, the server can be an independent physical server, a server cluster or a distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks, and big data and artificial intelligence platforms.

[0062] Figure 2 The flowchart of the web page code generation method provided by an exemplary embodiment of the present application is shown. Taking the method as being executed by a computer device as an example, optionally, the computer device includes Figure 1 the web page code generation architecture shown, and the method includes the following steps.

[0063] Step 220, obtain a first web page image.

[0064] The first web page image is an image corresponding to the design draft of the first web page, and the first web page image can also be referred to as the web page design diagram of the first web page. In the embodiments of the present application, the first web page image will be obtained, and the web page code of the first web page will be generated based on the first web page image. In one embodiment, the first web page image is an image drawn by a drawing tool. The first web page image is a web page image drawn by a designer using a drawing tool. Optionally, the drawing tool includes Photoshop, Sketch, Figma, Adobe XD, etc. In another embodiment, the first web page image is an image generated by a neural network model. The neural network model generates the first web page image that conforms to the received design sketch. Optionally, the neural network model also receives a text description as a generation constraint, so that the generated first web page image not only conforms to the design sketch, but also conforms to the input text description. Optionally, the neural network model is a ControlNet model.

[0065] The first web page image includes multiple components. The types of the multiple components on the first web page image can be the same or different. For example, the first web page image includes two input boxes for users to enter text and a scroll bar for changing the browsing position. Optionally, the multiple components can include a navigation bar, buttons, input boxes, text areas, radio buttons, check boxes, drop-down menus, labels, modal windows, carousels, tables, lists, dividers, breadcrumb navigations, sidebars, pop-ups, tooltips, progress bars, and collapsible panels, etc.

[0066] In one embodiment, the first web page image is stored in a local image library. When preparing to generate the web page code corresponding to the first web page image, the first web page image will be read from the local image library. Optionally, the local image library is an image library located in the terminal device used by the designer, and the local image library is an offline image library to ensure the security of the first web page image. In another embodiment, the first web page image is stored in a local area server, and the local area server is a server in a local area network. When preparing to generate the web page code corresponding to the first web page image, the first web page image will be downloaded from the local area server. At this time, the first web page image is shared in the local area server, which is convenient for multiple designers to collaboratively draw the first web page image, and is also convenient for modifying the first web page image and performing version updates on the first web page image, etc.

[0067] Step 240, perform image segmentation on the first web page image to obtain multiple components.

[0068] By performing image segmentation on the first web page image, multiple components on the first web page image are segmented. The multiple components are sub-images on the first web page image. Optionally, the distribution of the multiple components on the first web page image includes at least one of juxtaposed distribution and nested distribution. Juxtaposed distribution refers to a distribution method in which two or more components are independent of each other. Nested distribution refers to a distribution method in which there is a nested relationship between two or more components. For example, a sidebar component is inserted into an input box component. Optionally, perform image segmentation on the first image to segment all components on the first image.

[0069] In one embodiment, an image segmentation operation is performed on the first web page image through a neural network model. Optionally, the neural network model can be a segment anything model (SAM) or a YOLO model.

[0070] Step 260, predict the parent-child relationship between multiple components through a relationship prediction network.

[0071] The parent-child relationship among multiple components includes the hierarchical relationship among multiple components. Schematically, if a div component contains two txt components, then the parent component of the two txt components is the div component. At this time, it is considered that the level of the div component is shallower than that of the two txt components, and the levels of the two txt components are deeper than that of the div component.

[0072] Through the relationship prediction network, predict the parent-child relationship among the multiple components obtained by segmenting the first web page image. Optionally, the relationship prediction network is a hierarchical implicit neural network model. In one embodiment, through the relationship prediction network, predict the parent component and / or child component of each component among the multiple components obtained by segmenting the first web page image, and then integrate to obtain the overall parent-child relationship of the multiple components. Optionally, the relationship prediction network is a classification network. For the i-th component, the relationship prediction network predicts the probability that each component among the multiple components is the parent component of the i-th component, and determines the component that meets the parent component probability condition as the parent component of the i-th component. For example, determine the component with a predicted probability greater than 80% as the parent component of the i-th component, or if the probability corresponding to the component with the highest predicted probability is greater than 80%, then determine the component with the highest probability as the parent component of the i-th component. Optionally, the number of parent components of each component can be zero, one, or more. Optionally, for a component, if the parent component of the component cannot be predicted, it is considered that the parent component of the component is empty.

[0073] Optionally, the relationship prediction network is a classification network. For the i-th component, the relationship prediction network predicts the probability that each component among the multiple components is the child component of the i-th component, and determines the component that meets the child component probability condition as the child component of the i-th component. For example, determine the component with a predicted probability greater than 80% as the child component of the i-th component, or if the probability corresponding to the component with the highest predicted probability is greater than 80%, then determine the component with the highest probability as the child component of the i-th component. Optionally, the number of child components of each component can be zero, one, or more. Optionally, for a component, if the child component of the component cannot be predicted, it is considered that the child component of the component is empty.

[0074] Based on the parent component or child component of each component obtained above, construct a parent-child relationship sequence table. The parent-child relationship sequence table records the level of each component among the multiple components of the first web page image and the parent-child components associated with each component. The parent-child relationship sequence table is used to generate the web page code of the first web page.

[0075] In the embodiments of the present application, the relationship prediction network can be a classification or regression network. That is, in the embodiments of the present application, a classification or regression network is used to predict the parent-child relationships between multiple components and then generate web page code. In the related art, a large language model is often used to generate web page code. The large language model needs to use a large number of learning samples to fix a large number of network parameters. In contrast, the classification or regression network adopted in the embodiments of the present application reduces the model training cost, and the training samples required by the classification or regression network and the network parameters to be adjusted are less.

[0076] Step 280, generate the code of the first web page. The code of the first web page includes the code for implementing the parent-child relationships between multiple components.

[0077] In one embodiment, the code of the first web page is generated by a front-end code generator. The generated code of the first web page includes the code for implementing the parent-child relationships between multiple components. In one embodiment, component code is generated for each of the multiple components. The component code of each component includes the code for implementing the corresponding parent-child relationship of each component. Furthermore, the web page code of the first web page is obtained by assembling the multiple component codes of the multiple components.

[0078] Optionally, the front-end code generator can be a generator based on a low-code / no-code platform. The front-end code generator provides a visual interface that allows users to build a front-end web page by dragging and dropping components, configuring attributes, etc., and then automatically generate the corresponding code of the web page. For example, the front-end code generator can be Webflow (a browser plugin that helps users generate web page code without writing code), Bubble (a visual development tool designed to help non-technical personnel quickly build and deploy web pages without writing complex code), Wix (an online website building tool), etc.

[0079] In summary, in the embodiments of the present application, multiple components are segmented from the first web page image in the image segmentation stage, and the parent-child relationships between the multiple components are predicted by a relationship prediction network, and then the web page code of the first web page is generated. Since the web page code of the first web page includes the code for implementing the parent-child relationships between multiple components, the web page code generated in the embodiments of the present application can correctly reflect the parent-child relationships between the components in the first web page image, and the accuracy of the web page code is relatively high. Moreover, the method provided in the embodiments of the present application can be used to generate web page code in various programming languages, such as programming languages like HTML, JavaScript, etc. The method provided in the embodiments of the present application has universality.

[0080] Inference process of the relationship prediction network

[0081] Based on Figure 2 In the optional embodiments shown,Figure 3 A schematic diagram showing the inference method of the relationship prediction network mentioned in step 260 is as follows Figure 3 As shown, the relationship prediction network 30 is a hierarchical implicit neural network. The relationship prediction network 30 includes a feature extraction layer 31 and a prediction layer 32.

[0082] Obtain the first web page image 301 and multiple components 302 obtained by performing image segmentation on the first web page image 301. Input the first web page image 301 and the multiple components 302 into the feature extraction layer 31. Through the feature extraction layer 31, perform feature extraction on the multiple components 302 respectively to obtain multiple first image features 303, and through the feature extraction layer 31, perform feature extraction on the first web page image 301 to obtain a second image feature 304. Input the multiple first image features 303 and the second image feature 304 into the prediction layer 32, and through the prediction layer 32, predict the parent component and / or child component 305 of each component among the multiple components.

[0083] In one embodiment, for each component among the multiple components 302, the first image feature 303 and the second image feature 304 of the current component are concatenated through the prediction layer 32 to obtain the component representation of the current component, and based on the component representation of the current component, predict the parent component and / or child component of the current component among the multiple components 302.

[0084] In one embodiment, for each component among the multiple components 302, the position feature, size feature, and the first image feature 303 of the current component are concatenated through the prediction layer 32 to obtain a first concatenated feature; based on the multiple first concatenated features corresponding to the multiple components 302 and the second image feature 304, through the prediction layer 32, predict the parent component and / or child component of each component among the multiple components 302.

[0085] Optionally, for each component among the multiple components 302, the first concatenated feature and the second image feature 304 of the current component are concatenated through the prediction layer 32 to obtain the component representation of the current component, and based on the component representation of the current component, predict the parent component and / or child component of the current component among the multiple components 302.

[0086] Optionally, in the embodiments of the present application, for each component among the multiple components 302, identify the position feature and size feature of each component in the first web page image 301. Optionally, the operation of identifying the position feature and size feature is performed after the image segmentation stage 10 and before the relationship prediction stage 11. Optionally, by identifying the CSS attributes of the multiple components 302, obtain the position feature and size feature of each component.

[0087] In one embodiment, after obtaining a plurality of first image features 303 and second image features 304 through the feature extraction layer 31, a feature alignment operation is further performed. Through the feature alignment operation, the plurality of first image features 303 and second image features 304 are aligned, facilitating the subsequent fusion of features by the prediction layer 32. The feature alignment operation can eliminate the differences between data and improve the performance and generalization ability of the relationship prediction network 30. Optionally, the feature alignment operation includes operations such as normalization, standardization, and feature scaling.

[0088] Figure 4 The flowchart of the method for generating web page code provided by an exemplary embodiment of the present application is shown. Taking the method being executed by a computer device as an example, optionally, the computer device includes Figure 1 The web page code generation architecture shown. The method includes the following steps.

[0089] Step 410, obtain a first web page image.

[0090] The first web page image is the image corresponding to the design draft of the first web page, and the first web page image can also be referred to as the web page design diagram of the first web page. In the embodiments of the present application, a first web page image will be obtained, and web page code for the first web page will be generated based on the first web page image.

[0091] Step 420, perform image segmentation on the first web page image to obtain a plurality of components.

[0092] By performing image segmentation on the first web page image, a plurality of components on the first web page image are segmented. The plurality of components are subgraphs on the first web page image. In one embodiment, an image segmentation operation is performed on the first web page image through a neural network model. Optionally, the neural network model can be a Segment Anything Model or a YOLO model.

[0093] Step 430, for each component among the plurality of components, identify other CSS attributes of the component.

[0094] CSS attributes are used to describe the layout and style of components on a web page. CSS attributes include the component type, component class (class), component style (style), etc. of each component. Other CSS attributes include attributes other than the padding attribute and the margin attribute in CSS attributes. Optionally, other CSS attributes include all attributes other than the padding attribute and the margin attribute in CSS attributes.

[0095] In one embodiment, for each of the multiple components, image recognition is performed to obtain some or all of the attributes in the other CSS attributes of each component. And / or, in another embodiment, for each of the multiple components, the description text of each component is obtained, and by performing text recognition on the description text of each component, some or all of the attributes in the other CSS attributes of each component are obtained. Optionally, the attributes of each component obtained through image recognition and text recognition are combined to obtain the other CSS attributes of each component.

[0096] In one embodiment, for each of the multiple components, CSS attribute recognition is performed on each component to obtain the CSS file corresponding to each component, and the CSS file of each component includes all CSS attributes. In one embodiment, for each of the multiple components, image recognition is performed to obtain some or all of the attributes in all the CSS attributes of each component. And / or, in another embodiment, for each of the multiple components, the description text of each component is obtained, and by performing text recognition on the description text of each component, some or all of the attributes in all the CSS attributes of each component are obtained. Optionally, the attributes of each component obtained through image recognition and text recognition are combined to obtain all the CSS attributes of each component.

[0097] In one embodiment, the identified other CSS attributes can determine the position coordinates and size of the component. Here, the position coordinates and size can be used as the position feature and size feature respectively, and applied to the operation of splicing the position feature, size feature and the first image feature 303 of the current component in the prediction layer 32 above.

[0098] Step 440, predict the parent-child relationship between multiple components through a relationship prediction network.

[0099] Through a relationship prediction network, predict the parent-child relationship between multiple components obtained by segmenting the first web page image. In one embodiment, through a relationship prediction network, predict the parent component of each component among the multiple components obtained by segmenting the first web page image, and then integrate to obtain the parent-child relationship of the multiple components as a whole.

[0100] And / or, through a relationship prediction network, predict the child components of each component among the multiple components obtained by segmenting the first web page image, and then integrate to obtain the parent-child relationship of the multiple components as a whole.

[0101] Step 450, generate component code for each of the multiple components. The component code includes code for implementing the corresponding parent-child relationship of the component and code for implementing the other CSS attributes of the component.

[0102] In one embodiment, for each of a plurality of components, the padding attribute and the margin attribute of each component are determined. The padding attribute is used to characterize the distance from the current component to its child components, and the margin attribute is used to characterize the distance from the current component to its parent component. Component code is generated by a front-end code generator, and the component code includes code for implementing the padding attribute and the margin attribute corresponding to the current component.

[0103] The parent-child relationships among the plurality of components describe the parent component and child components corresponding to each component. For example, the child components of component A include component B, and the parent component of component A includes component C. For component A, the padding attribute of component A is determined, where the padding attribute is the distance between component A and component B, and the margin attribute of component A is determined, where the margin attribute is the distance between component A and component C. Since the parent component and child components of each component have been predicted in the foregoing, therefore, the padding attribute and the margin attribute of each component can be determined here, and relatively accurate padding attributes and margin attributes of each component are obtained.

[0104] In one embodiment, based on the position coordinates of the plurality of components and the parent-child relationship corresponding to each component, the padding attribute and the margin attribute of each component are determined. Through the parent-child relationship corresponding to each component, the child components and the parent component of each component can be determined. Furthermore, by combining the position coordinates of each component among the plurality of components, the distance between each component and its child components, and the distance between each component and its parent component can be obtained, and then the padding attribute and the margin attribute of each component are determined. Optionally, the position coordinates of the plurality of components are determined based on other CSS attributes of the plurality of components in the foregoing.

[0105] In one embodiment, other CSS attributes of each identified component include attributes other than the padding attribute and the margin attribute in the CSS attributes. Component code is generated by a front-end code generator, and the component code includes code for implementing other CSS attributes corresponding to the current component.

[0106] In another embodiment, CSS attribute identification is performed on each component to obtain the CSS file corresponding to each component. The CSS file of each component includes all CSS attributes, that is, the CSS file also includes the padding attribute and the margin attribute. In the embodiments of the present application, the padding attribute and the margin attribute determined by the predicted parent-child relationship are taken as the standard, and the padding attribute and the margin attribute in the CSS file are modified, and component code is generated based on the CSS file by a front-end code generator.

[0107] Step 460: Assemble the component codes of the plurality of components to obtain the web page code of the first web page.

[0108] Through a front - end code generator, multiple component codes of multiple components are assembled to obtain the web page code of the first web page. Optionally, the web page code of the first web page generated by the front - end code generator is also subject to manual review or pre - execution by a pre - execution tool to ensure the security and reliability of the generated web page code of the first web page.

[0109] In summary, in the above - mentioned embodiments, the parent - child relationships between the multiple predicted components are converted into the inner and outer margin attributes of each component, and then the component code of each component is generated. The component code includes the code for implementing the inner and outer margin attributes of each component. Therefore, the generated first web page code can correctly reflect the distances between the multiple components, that is, the finally generated first web page can correctly reflect the parent - child relationships (hierarchical relationships) between the multiple components. The generated first web page is more compact, and the first web page can reflect the design logic of the first web page image.

[0110] Training process of the relationship prediction network

[0111] Based on Figure 2 In the optional embodiment shown, Figure 5 FIG. shows a schematic diagram of the training method of the relationship prediction network in step 260. Figure 5 The training method of the relationship prediction network shown is executed by a computer device, which can be the same as or different from the computer device of the web page code generation architecture shown. Figure 1 shown.

[0112] As Figure 5 shown, when training the relationship prediction network 50, a sample web page image 501 is obtained. The sample web page image 501 is an image corresponding to the design draft of the sample web page. Optionally, the sample web page image 501 is a web page image drawn by a designer using a drawing tool or an image in an open - source image set.

[0113] Performing image segmentation on the sample web page image 501 to obtain multiple sample components 502. Predicting the parent - child relationships between the multiple sample components 502 through the relationship prediction network 50 to obtain the first parent - child relationship 506; obtaining the second parent - child relationship 507 by annotating the parent - child relationships between the multiple sample components 502. Training the relationship prediction network 50 based on the gap between the first parent - child relationship 506 and the second parent - child relationship 507.

[0114] For example, the first parent-child relationship 506 includes the parent components of each sample component obtained by prediction, and the second parent-child relationship 507 includes the parent components of each sample component obtained by annotation. Based on the error between the first parent-child relationship 506 and the second parent-child relationship 507, the network parameters of the relationship prediction network 50 are updated. Optionally, the differences between the predicted parent components and the annotated parent components of each sample component are accumulated to obtain a loss value, and the relationship prediction network 50 is trained based on the loss value. Similarly, the first parent-child relationship 506 may also include the child components of each sample component obtained by prediction, and the second parent-child relationship 507 includes the child components of each sample component obtained by annotation.

[0115] Exemplarily, as Figure 5 shown, the relationship prediction network 50 includes a feature extraction layer 51 and a prediction layer 52. After performing an image segmentation operation on the sample web page image 501 to obtain a plurality of sample components 502, a masking operation will be performed. For a masking process, a masking operation is performed on one sample component among the plurality of sample components 502 in the sample web page image 501 to obtain a masked image; by performing a plurality of masking processes, a plurality of masked images 503 corresponding to the plurality of sample components 502 one by one are obtained.

[0116] The feature extraction layer 51 respectively performs feature extraction on the plurality of masked images 503 to obtain a plurality of third image features 504, and the feature extraction layer 51 performs feature extraction on the sample web page image 501 to obtain a fourth image feature 505; based on the plurality of third image features 504 and the fourth image feature 505, the first parent-child relationship 506 is predicted through the prediction layer 52.

[0117] Optionally, in the prediction layer 52, for each sample component among the plurality of sample components 502, the third image feature 504 and the fourth image feature 505 of the current sample component are concatenated through the prediction layer 52 to obtain the component representation of the current sample component, and the parent component and / or child component of the current sample component among the plurality of sample components 502 are predicted based on the component representation of the current sample component.

[0118] In the embodiment of the present application, a masking technique will be used to train the relationship prediction network 50. For each sample component among the plurality of sample components 502, the relationship prediction network 50 can learn not only the ability to predict parent / child components from the masked image corresponding to the current sample component and the sample web page image 501, but also the picture features, position features, and size features of the current sample component. Through the training method of the masking operation, the model ability of the relationship prediction network 50 is improved, and the trained relationship prediction network 50 can be better applied to the inference process.

[0119] In one embodiment, a relationship prediction network is trained in a self-paced learning manner, where self-paced learning will gradually increase components in each training process, thereby gradually increasing the training difficulty. Figure 6 FIG. shows a schematic diagram of a method for training a relationship prediction network provided by an exemplary embodiment of the present application. Figure 6 The shown method for training the relationship prediction network is executed by a computer device, which may be the same as or different from the computer device having the Figure 1 web page code generation architecture shown.

[0120] Exemplarily, multiple training processes will be executed during the self-paced learning process. For example, Figure 6 as shown, in the i-th training process, the (i - 1)-th sample web page image is obtained. When i equals 1, the (i - 1)-th sample web page image is an empty web page image, and the empty web page image is used to represent an image with no components in the initial state. The i-th sample component is added to the (i - 1)-th sample web page image 601 to obtain the i-th sample web page image 602. The i-th sample web page image 602 is input into the relationship prediction network 60, and the i-th sample parent-child relationship 603 is predicted through the relationship prediction network 60. The i-th sample parent-child relationship 603 includes the parent-child relationships of all components in the i-th sample web page image 602.

[0121] Based on the gap between the predicted i-th sample parent-child relationship 603 and the i'-th sample parent-child relationship 604, the relationship prediction network 60 is trained. The i'-th sample parent-child relationship 604 includes the true parent-child relationships of all components in the i-th sample web page image 602. It should be noted that in the embodiments of the present application, the sample components added in each training process are selected. The added i-th sample component can be a sub-component of an existing component, a parent component of an existing component, or a component that has no parent-child relationship with an existing component. That is, the parent-child relationship corresponding to each added sample component is known. Thus, the true parent-child relationships of all components in the i-th sample web page image are naturally obtained. Through the self-paced learning method provided by the present application, no additional annotation operation is required, and no additional annotation is needed for the parent-child relationships of all components in the i-th sample web page image.

[0122] Optionally, the added i-th sample component in one training process may include one or more sample components, and the number of sample components added in each training process may be the same or different. Optionally, components are gradually added in each training process until the number of components in the web page image reaches a set upper limit.

[0123] In the above embodiments, a method for training the relationship prediction network 60 through self-paced learning is provided. The training method through self-paced learning does not require additional annotation operations or the use of additional annotation resources. Moreover, through the self-paced learning method, the training difficulty of the relationship prediction network 60 increases gradually, which has two advantages. The first advantage is that the training speed of the relationship prediction network 60 will be significantly accelerated. The second advantage is that since the training difficulty of the relationship prediction network 60 each time is related to the previous training process, it is very difficult for the relationship prediction network 60 to enter the overfitting and local optimal points.

[0124] In one embodiment, Figure 5 and Figure 6 shows two training methods for the relationship prediction network. Optionally, one of them can be selected Figure 5 or Figure 6 to execute the training method, or both can be selected Figure 5 and Figure 6 shown to train the relationship prediction network. When both are selected, the training method shown in Figure 5 can be executed first to train the relationship prediction network, or the training method shown in Figure 6 can be executed first to train the relationship prediction network.

[0125] The embodiments of the present application also provide a device for generating web page code. The device for generating web page code can be implemented by software, hardware, or a combination of both to become part or all of a computer device. Refer to Figure 7 As shown in the figure, the device includes: an acquisition module 701, an image segmentation module 702, a prediction module 703, and a generation module 704.

[0126] The acquisition module 701 is used to acquire a first web page image, and the first web page image is an image corresponding to the design draft of the first web page.

[0127] The image segmentation module 702 is used to perform image segmentation on the first web page image to obtain multiple components.

[0128] The prediction module 703 is used to predict the parent-child relationships between multiple components through the relationship prediction network.

[0129] The generation module 704 is used to generate the code of the first web page, and the code of the first web page includes the code for implementing the parent-child relationships between multiple components.

[0130] In an alternative embodiment, the relationship prediction network includes a feature extraction layer and a prediction layer. The prediction module 703 is further configured to perform feature extraction on multiple components respectively through the feature extraction layer to obtain multiple first image features; and perform feature extraction on the first web page image through the feature extraction layer to obtain a second image feature. Based on the multiple first image features and the second image feature, the prediction layer is used to predict the parent component of each component among the multiple components, and / or predict the child component of each component among the multiple components.

[0131] In an alternative embodiment, the apparatus further includes an identification module 705. The identification module 705 is further configured to, for each component among the multiple components, identify the position feature and size feature of the component in the first web page image. The prediction module 703 is further configured to splice the position feature, size feature, and first image feature of the component through the prediction layer to obtain a first splicing feature. Based on the multiple first splicing features corresponding to the multiple components and the second image feature, the prediction layer is used to predict the parent component of each component among the multiple components, and / or predict the child component of each component among the multiple components.

[0132] In an alternative embodiment, the generation module 704 is further configured to generate a component code for each component among the multiple components. The component code includes code for implementing the parent-child relationship corresponding to the component. The multiple component codes of the multiple components are assembled to obtain the web page code of the first web page.

[0133] In an alternative embodiment, the generation module 704 is further configured to, for each component among the multiple components, determine the padding attribute and margin attribute of the component. The padding attribute is used to characterize the distance from the component to its child component, and the margin attribute is used to characterize the distance from the component to its parent component. A component code is generated through a front-end code generator. The component code includes code for implementing the padding attribute and margin attribute of each component.

[0134] In an alternative embodiment, the identification module 705 is further configured to, for each component among the multiple components, identify other Cascading Style Sheets (CSS) attributes of the component. The component code further includes code for implementing other CSS attributes of the component. The other CSS attributes include attributes other than the padding attribute and margin attribute in the CSS attributes.

[0135] In an alternative embodiment, the obtaining module 701 is further configured to obtain a sample web page image, where the sample web page image is an image corresponding to the design draft of the sample web page. The image segmentation module 702 is further configured to perform image segmentation on the sample web page image to obtain a plurality of sample components. The prediction module 703 is further configured to predict the parent-child relationship between the plurality of sample components through a relationship prediction network to obtain a first parent-child relationship. The apparatus further includes a training module 706, and the training module 706 is configured to train the relationship prediction network based on the gap between the first parent-child relationship and the second parent-child relationship, where the second parent-child relationship is the parent-child relationship between the plurality of sample components obtained by annotation.

[0136] In an alternative embodiment, the relationship prediction network includes a feature extraction layer and a prediction layer; the prediction module 703 is further configured to, for a masking process, perform a masking operation on a sample component among the plurality of sample components in the sample web page image to obtain a masked image; by performing a plurality of masking processes, obtain a plurality of masked images corresponding one by one to the plurality of sample components. Respectively perform feature extraction on the plurality of masked images through the feature extraction layer to obtain a plurality of third image features, and perform feature extraction on the sample web page image through the feature extraction layer to obtain a fourth image feature. Based on the plurality of third image features and the fourth image feature, predict the first parent-child relationship through the prediction layer.

[0137] In summary, in the embodiment of the present application, a plurality of components are segmented from the first web page image in the image segmentation stage, the parent-child relationship between the plurality of components is predicted through a relationship prediction network, and the web page code of the first web page is generated. The web page code includes code for implementing the parent-child relationship between the plurality of components. Furthermore, the web page code generated in the embodiment of the present application can correctly reflect the parent-child relationship between the components in the first web page image, and the accuracy of the web page code is relatively high. Moreover, the method provided in the embodiment of the present application can be used to generate web page codes in various programming languages, such as programming languages like HTML, JavaScript, etc. The method provided in the embodiment of the present application has universality.

[0138] Among them, the obtaining module 701, the image segmentation module 702, the prediction module 703, the generating module 704, the recognition module 705, and the training module 706 can all be implemented by software or can be implemented by hardware. Exemplarily, next, taking the obtaining module 701 as an example, the implementation manner of the obtaining module 701 is introduced. Similarly, the implementation manners of the image segmentation module 702, the prediction module 703, the generating module 704, the recognition module 705, and the training module 706 can refer to the implementation manner of the obtaining module 701.

[0139] As an example of a software functional unit, the obtaining module 701 may include code running on a computing instance. The computing instance may include at least one of a physical host (computing device), a virtual machine, and a container. Further, the above computing instance may be one or more. For example, the obtaining module 701 may include code running on multiple hosts / virtual machines / containers. It should be noted that the multiple hosts / virtual machines / containers for running the code may be distributed in the same region or in different regions. Further, the multiple hosts / virtual machines / containers for running the code may be distributed in the same availability zone (AZ) or in different AZs, and each AZ includes one data center or multiple geographically proximate data centers. Usually, one region may include multiple AZs.

[0140] Similarly, the multiple hosts / virtual machines / containers for running the code may be distributed in the same virtual private cloud (VPC) or in multiple VPCs. Usually, one VPC is set within one region. For cross-region communication between two VPCs within the same region and between VPCs in different regions, a communication gateway needs to be set in each VPC, and the interconnection between VPCs is achieved through the communication gateway.

[0141] As an example of a hardware functional unit, the obtaining module 701 may include at least one computing device, such as a server. Alternatively, the obtaining module 701 may also be a device implemented using an application-specific integrated circuit (ASIC) or a programmable logic device (PLD). Among them, the above PLD may be implemented by a complex programmable logic device (CPLD), a field-programmable gate array (FPGA), a generic array logic (GAL), or any combination thereof.

[0142] The multiple computing devices included in the obtaining module 701 may be distributed in the same region or in different regions. The multiple computing devices included in the obtaining module 701 may be distributed in the same availability zone (AZ) or in different AZs. Similarly, the multiple computing devices included in the obtaining module 701 may be distributed in the same virtual private cloud (VPC) or in multiple VPCs. Among them, the multiple computing devices may be any combination of computing devices such as servers, application-specific integrated circuits (ASICs), programmable logic devices (PLDs), complex programmable logic devices (CPLDs), field-programmable gate arrays (FPGAs), and generic array logic (GALs).

[0143] It should be noted that, in other embodiments, the obtaining module 701 may be used to perform any step in the method for generating web page code, the image segmentation module 702 may be used to perform any step in the method for generating web page code, the prediction module 703 may be used to perform any step in the method for generating web page code, the generating module 704 may be used to perform any step in the method for generating web page code, the recognition module 705 may be used to perform any step in the method for generating web page code, and the training module 706 may be used to perform any step in the method for generating web page code. The steps to be implemented by the obtaining module 701, the image segmentation module 702, the prediction module 703, the generating module 704, the recognition module 705, and the training module 706 can be specified as needed. By respectively implementing different steps in the method for generating web page code through the obtaining module 701, the image segmentation module 702, the prediction module 703, the generating module 704, the recognition module 705, and the training module 706, all functions of the web page code generating device are realized.

[0144] It should be noted that: when the web page code generating device provided in the above embodiment generates web page code, only the division of the above functional modules is used for illustration. In practical applications, the above functions can be allocated to different functional modules as needed, that is, the internal structure of the device is divided into different functional modules to complete all or part of the functions described above. In addition, the web page code generating device provided in the above embodiment and the embodiment of the web page code generating method belong to the same concept. For the specific implementation process, please refer to the method embodiment, which will not be elaborated here.

[0145] The embodiment of the present application also provides a computing device. Please refer to Figure 8 Figure, the computing device 800 includes a bus 802, a processor 804, a memory 806, and a communication interface 808. The processor 804, the memory 806, and the communication interface 808 communicate with each other through the bus 802. The computing device 800 may be a server or a terminal device. It should be understood that the present application does not limit the number of processors and memories in the computing device 800.

[0146] The bus 802 can be a peripheral component interconnect (PCI) bus, an extended industry standard architecture (EISA) bus, or the like. The bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 8 only one line is used in Figure 8 , but it does not mean that there is only one bus or one type of bus. The bus 802 can include a path for transmitting information between various components of the computing device 800 (e.g., the memory 806, the processor 804, the communication interface 808).

[0147] The processor 804 can include any one or more of processors such as a central processing unit (CPU), a graphics processing unit (GPU), a microprocessor (MP), or a digital signal processor (DSP).

[0148] The memory 806 can include volatile memory, such as random access memory (RAM). The processor 804 can also include non-volatile memory, such as read-only memory (ROM), flash memory, a hard disk drive (HDD), or a solid state drive (SSD).

[0149] The memory 806 stores executable program code, and the processor 804 executes the executable program code to respectively implement the functions of the foregoing acquisition module, image segmentation module, prediction module, generation module, recognition module, and training module, thereby implementing the method for generating web page code. That is, the memory 806 stores instructions for executing the method for generating web page code.

[0150] The communication interface 808 uses a transceiver module such as, but not limited to, a network interface card or a transceiver to implement communication between the computing device 800 and other devices or a communication network.

[0151] Embodiments of the present application also provide a computing device cluster. The computing device cluster includes at least one computing device. The computing device can be a server, such as a central server, an edge server, or a local server in a local data center. In some embodiments, the computing device can also be a terminal device such as a desktop computer, a laptop computer, or a smart phone.

[0152] As Figure 9 shown, the computing device cluster includes at least one computing device 800. Instructions for executing the method for generating web page code can be stored in the memory 806 of one or more of the computing devices 800 in the computing device cluster. Similar to the computing device 800, the computing device 800 includes a bus 802, a processor 804, a memory 806, and a communication interface 808. The processor 804, the memory 806, and the communication interface 808 communicate with each other through the bus 802.

[0153] In some possible implementation manners, partial instructions for executing the method for generating web page code can also be stored separately in the memory 806 of one or more of the computing devices 800 in the computing device cluster. In other words, a combination of one or more computing devices 800 can jointly execute the instructions for executing the method for generating web page code.

[0154] It should be noted that the memories 806 in different computing devices 800 in the computing device cluster can store different instructions, which are respectively used to execute partial functions of the web page code generation device. That is, the instructions stored in the memories 806 of different computing devices 800 can implement the functions of one or more of the acquisition module, the image segmentation module, the prediction module, the generation module, the recognition module, and the training module.

[0155] In some possible implementation manners, one or more computing devices in the computing device cluster can be connected through a network. Among them, the network can be a wide area network or a local area network, etc. Figure 10 shows a possible implementation manner. As Figure 10 shown, two computing devices 800A and 800B are connected through a network. Specifically, they are connected to the network through the communication interfaces in each computing device. In this type of possible implementation manners, the memory 806 in the computing device 800A stores instructions for executing the functions of the acquisition module, the image segmentation module, and the prediction module. At the same time, the memory 806 in the computing device 800B stores instructions for executing the functions of the generation module, the recognition module, and the training module.

[0156] Figure 10The connection method between the computing device clusters shown can be considered. Since the method for generating web page code provided in this application needs to perform a large number of model training operations, it is considered to hand over the functions implemented by the generation module, the recognition module, and the training module to the computing device 800B for execution.

[0157] It should be understood that Figure 10 the functions of the computing device 800A shown in can also be completed by multiple computing devices 800B. Similarly, the functions of the computing device 800B can also be completed by multiple computing devices 800A.

[0158] The embodiments of this application also provide another computing device cluster. The connection relationship between the computing devices in this computing device cluster can be similarly referred to Figure 9 and Figure 10 the connection method of the computing device cluster described. The difference is that the same instructions for executing the method for generating web page code can be stored in the memory 806 of one or more computing devices 800 in this computing device cluster.

[0159] In some possible implementation manners, the memory 806 of one or more computing devices 800 in this computing device cluster can also store partial instructions for executing the method for generating web page code respectively. In other words, a combination of one or more computing devices 800 can jointly execute the instructions for executing the method for generating web page code.

[0160] The embodiments of this application also provide a computer program product containing instructions. The computer program product can be software or a program product containing instructions that can run on a computing device or be stored in any available medium. When the computer program product runs on at least one computing device, it causes at least one computing device to execute the method for generating web page code.

[0161] The embodiments of this application also provide a computer-readable storage medium. The computer-readable storage medium can be any available medium that a computing device can store or a data storage device such as a data center containing one or more available media. The available medium can be a magnetic medium (for example, a floppy disk, a hard disk, a magnetic tape), an optical medium (for example, a DVD), or a semiconductor medium (for example, a solid-state drive), etc. The computer-readable storage medium includes instructions that direct a computing device to execute the method for generating web page code.

[0162] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware, or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer instructions are loaded and executed on a computer, the processes or functions described in the embodiments of the present application are generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center in a wired manner (such as coaxial cable, optical fiber, digital subscriber line (DSL)) or a wireless manner (such as infrared, wireless, microwave, etc.). The computer-readable storage medium can be any available medium that can be accessed by a computer, or a data storage device such as a server or data center that includes one or more integrated available media. The available medium can be a magnetic medium (such as a floppy disk, hard disk, magnetic tape), an optical medium (such as a digital versatile disc (DVD)), or a semiconductor medium (such as a solid state disk (SSD)), etc. It should be noted that the computer-readable storage medium mentioned in the embodiments of the present application can be a non-volatile storage medium, in other words, it can be a non-transitory storage medium.

[0163] It should be understood that the "plurality" mentioned herein refers to two or more. In the description of the embodiments of the present application, unless otherwise specified, " / " means "or". For example, A / B can mean A or B; the "and / or" herein is merely a description of the association relationship of associated objects, indicating that three relationships can exist. For example, A and / or B can mean: A exists alone, A and B exist simultaneously, and B exists alone. In addition, in order to facilitate a clear description of the technical solutions of the embodiments of the present application, in the embodiments of the present application, terms such as "first" and "second" are used to distinguish the same items or similar items with basically the same functions and roles. Those skilled in the art can understand that the terms "first", "second", etc. do not limit the quantity and execution order, and the terms "first", "second", etc. do not necessarily limit differences.

[0164] It should be noted that the information involved in the embodiments of this application (including but not limited to user device information, user personal information, etc.), data (including but not limited to data for analysis, stored data, displayed data, etc.), and signals are all authorized by the user or fully authorized by all parties, and the collection, use, and processing of relevant data need to comply with the relevant laws, regulations, and standards of relevant countries and regions.

[0165] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them; although the present application has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that they can still modify the technical solutions recorded in the foregoing embodiments, or perform equivalent replacements on some of the technical features; and these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the protection scope of the technical solutions of the embodiments of the present application.

Claims

1. A method for generating a webpage code, characterized in that: The method comprises: Acquire a first webpage image, where the first webpage image is an image corresponding to a design draft of the first webpage; Performing image segmentation on the first web page image to obtain a plurality of components; Predicting the parent-child relationship between the multiple components through a relationship prediction network; Generate code for the first webpage, wherein the code for the first webpage includes code for implementing a parent-child relationship between the plurality of components; The training process of the relationship prediction network includes: Performing feature extraction on the plurality of mask images respectively through the feature extraction layer in the relationship prediction network to obtain a plurality of third image features, and performing feature extraction on the sample web page image to obtain a fourth image feature; each mask image is obtained by performing a mask operation on a sample component in the sample web page image, and the sample web page image is an image corresponding to the design draft of the sample web page; Through the prediction layer in the relationship prediction network, for each sample component in the sample web page image, the third image feature and the fourth image feature of each sample component are spliced ​​to obtain a component representation of each sample component; based on the component representation of each sample component, the parent component and / or child component of each sample component in the sample web page image is predicted to obtain a first parent-child relationship; The relationship prediction network is trained based on the gap between the first parent-child relationship and the second parent-child relationship, wherein the second parent-child relationship is the parent-child relationship between multiple sample components in the labeled sample web page image.

2. The method according to claim 1, characterized in that The relationship prediction network includes a feature extraction layer and a prediction layer; the parent-child relationship between the multiple components is predicted by the relationship prediction network, including: Performing feature extraction on the plurality of components respectively through the feature extraction layer to obtain a plurality of first image features; and performing feature extraction on the first web page image through the feature extraction layer to obtain a second image feature; Based on the multiple first image features and the second image features, the parent component of each component in the multiple components is predicted through the prediction layer, and / or the child component of each component in the multiple components is predicted.

3. The method according to claim 2, characterized in that The method further comprises: For each component of the plurality of components, identifying and obtaining a position feature and a size feature of the component in the first web page image; The predicting, based on the multiple first image features and the second image features, obtaining a parent component of each component in the multiple components through the prediction layer, and / or predicting a child component of each component in the multiple components, includes: Splicing the position feature, the size feature and the first image feature of the component through the prediction layer to obtain a first splicing feature; Based on the multiple first splicing features and the second image features corresponding to the multiple components, the parent component of each component in the multiple components is predicted through the prediction layer, and / or the child component of each component in the multiple components is predicted.

4. The method according to any one of claims 1 to 3, characterized in that: The code for generating the first webpage includes: Generate component code for each component of the plurality of components, the component code including code for implementing a parent-child relationship corresponding to the component; The plurality of component codes of the plurality of components are assembled to obtain the webpage code of the first webpage.

5. The method according to claim 4, characterized in that The generating component code for each component of the plurality of components comprises: For each component of the plurality of components, determine an inner margin attribute and an outer margin attribute of the component, wherein the inner margin attribute is used to represent a distance from a child component of the component, and the outer margin attribute is used to represent a distance from a parent component of the component; The component code is generated by a front-end code generator, and the component code includes codes for implementing the inner margin attribute and the outer margin attribute of each component.

6. The method according to claim 5, characterized in that The method further comprises: For each of the multiple components, other Cascading Style Sheets (CSS) properties of the component are identified, and the component code also includes codes for implementing the other CSS properties of the component, wherein the other CSS properties include properties in the CSS properties except the inner margin property and the outer margin property.

7. A device for generating a webpage code, characterized in that: The device comprises: an acquisition module, used to acquire a first web page image, wherein the first web page image is an image corresponding to a design draft of the first web page; An image segmentation module, used for performing image segmentation on the first web page image to obtain multiple components; A prediction module, used for predicting the parent-child relationship between the multiple components through a relationship prediction network; A generating module, configured to generate code for the first webpage, wherein the code for the first webpage includes code for implementing a parent-child relationship between the plurality of components; The prediction module is further used to perform feature extraction on the plurality of mask images respectively through the feature extraction layer in the relationship prediction network to obtain a plurality of third image features, and to perform feature extraction on the sample web page image to obtain a fourth image feature; each mask image is obtained by performing a mask operation on a sample component in the sample web page image, and the sample web page image is an image corresponding to the design draft of the sample web page; The prediction module is further used to, for each sample component in the sample web page image, concatenate the third image feature and the fourth image feature of each sample component through the prediction layer in the relationship prediction network to obtain a component representation of each sample component; predict the parent component and / or child component of each sample component in the sample web page image based on the component representation of each sample component to obtain a first parent-child relationship; A training module is used to train the relationship prediction network based on the gap between the first parent-child relationship and the second parent-child relationship, wherein the second parent-child relationship is the parent-child relationship between multiple sample components in the sample web page image obtained by annotation.

8. A computing device cluster, characterized in that: comprising at least one computing device, each computing device comprising a processor and a memory; The processor of the at least one computing device is configured to execute instructions stored in the memory of the at least one computing device, so that the computing device cluster executes the method according to any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that: The method comprises computer program instructions. When the computer program instructions are executed by a computing device cluster, the computing device cluster performs the method according to any one of claims 1 to 6.

10. A computer program product comprising instructions, characterized in that When the instructions are executed by a computing device cluster, the computing device cluster executes the method according to any one of claims 1 to 6.

Images