Method, device, equipment, storage medium and program product for generating front-end code

Abstract

The present disclosure relates to the technical field of code generation, and particularly provides a front-end code generation method and device, equipment, a storage medium and a program product. The method comprises: obtaining a multi-scale feature enhancement map of a front-end page image, and obtaining a pre-constructed pixel semantic similarity model; processing the multi-scale feature enhancement map using the pixel semantic similarity model to obtain a component text description of the front-end page image; wherein the component text description comprises category labels of multiple components in the front-end page image; activating the front-end page image using the component text description to obtain an activated front-end page image; and generating front-end code of the front-end page image using the activated front-end page image. The present disclosure can automatically label the front-end page image, accurately identify and locate the multiple components contained in the front-end page image, and thus improve the accuracy of generating the front-end code.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to the technical field of code generation, and in particular, to a front-end code generation method and device, equipment, a storage medium and a program product. BACKGROUND

[0002] As the core foundation of building web applications (referred to as web applications), mobile applications and cross-platform applications, the application prospect of front-end code is constantly expanding with the development of technology and industrial demand, showing a trend of diversification, intelligence and light weight.

[0003] In related technologies, front-end code can usually be generated based on a requirement description document, and front-end code can also be generated based on edge information of components in a front-end page image. However, these front-end code generation methods lack understanding of the overall layout of the front-end page, ignore the association between components, and the code generation effect is not good. SUMMARY

[0004] The present disclosure is proposed in view of the above problems. The present disclosure provides a front-end code generation method and device, equipment, a storage medium and a program product.

[0005] According to one aspect of the present disclosure, a front-end code generation method is provided, comprising:

[0006] obtaining a multi-scale feature enhancement map of a front-end page image, and obtaining a pre-constructed pixel semantic similarity model;

[0007] processing the multi-scale feature enhancement map using the pixel semantic similarity model to obtain a component text description of the front-end page image; wherein the component text description includes category labels of multiple components in the front-end page image;

[0008] activating the front-end page image using the component text description to obtain an activated front-end page image;

[0009] generating front-end code of the front-end page image using the activated front-end page image.

[0010] According to another aspect of the present disclosure, a front-end code generation device is provided, comprising:

[0011] an acquisition module configured to obtain a multi-scale feature enhancement map of a front-end page image, and obtain a pre-constructed pixel semantic similarity model;

[0012] a processing module configured to process the multi-scale feature enhancement map using the pixel semantic similarity model to obtain a component text description of the front-end page image; wherein the component text description includes category labels of multiple components in the front-end page image;

[0013] The processing module is configured to activate the front-end page image using the component text description, to obtain an activated front-end page image.

[0014] The generating module is configured to generate front-end code of the front-end page image using the activated front-end page image.

[0015] In yet another aspect of the example embodiments of the present disclosure, an electronic device is provided, which includes a memory, a processor, and a computer program stored in the memory, and the processor executes the computer program to implement the method according to the example embodiments of the present disclosure.

[0016] In yet another aspect of the example embodiments of the present disclosure, a computer-readable storage medium is provided, which stores a computer program, and the computer program is executed by a processor to implement the method according to the example embodiments of the present disclosure.

[0017] In yet another aspect of the example embodiments of the present disclosure, a computer program product is provided, which includes a computer program, and the computer program is executed by a processor to implement the method according to the example embodiments of the present disclosure.

[0018] As will be described in detail below, according to the method, device, equipment, storage medium and program product for generating front-end code according to the embodiments of the present disclosure, the method includes obtaining a multi-scale feature enhancement map of a front-end page image, and obtaining a pixel semantic similarity model constructed in advance; processing the multi-scale feature enhancement map using the pixel semantic similarity model to obtain a component text description of the front-end page image; wherein the component text description includes class labels of multiple components in the front-end page image; activating the front-end page image using the component text description to obtain an activated front-end page image; and generating front-end code of the front-end page image using the activated front-end page image. After the component text description is automatically generated based on the multi-scale feature enhancement map covering feature information on multiple different scales using the pixel semantic similarity model, the front-end page image is automatically labeled using the component text description, precise identification and positioning of the multiple components included in the front-end page image are achieved, and the accuracy of the generated front-end code is improved.

[0019] It is to be understood that both the foregoing general description and the following detailed description are exemplary, and are intended to provide further explanation of the subject technology. BRIEF DESCRIPTION OF DRAWINGS

[0020] The above and other objects, features and advantages of the present disclosure will become more apparent from the following detailed description when taken in conjunction with the accompanying drawings in which:

[0021] Figure 1 An architecture schematic diagram of a generation system of front-end code is shown according to an example embodiment of the present disclosure;

[0022] Figure 2 A flow schematic diagram of a generation method of front-end code is shown according to an example embodiment of the present disclosure;

[0023] Figure 3 A structure schematic diagram of a generation apparatus of front-end code is shown according to an example embodiment of the present disclosure;

[0024] Figure 4 A structure schematic diagram of an electronic device is shown according to an example embodiment of the present disclosure;

[0025] Figure 5 A structure schematic diagram of a computer system is shown according to an example embodiment of the present disclosure. DETAILED DESCRIPTION

[0026] In order to make the objectives, technical solutions and advantages of the present disclosure more apparent, the following will describe example embodiments according to the present disclosure in detail with reference to the accompanying drawings. Obviously, the described embodiments are only some of the embodiments of the present disclosure, rather than all the embodiments of the present disclosure, and it should be understood that the present disclosure is not limited to the example embodiments described herein.

[0027] It should be understood that each step described in the method embodiments of the present disclosure can be executed in different orders and / or in parallel. In addition, the method embodiments can include additional steps and / or omit the execution of the steps shown. The scope of the present disclosure is not limited in this respect.

[0028] The term “comprising” and its variations as used herein are open-ended, that is, “including but not limited to”. The term “based on” is “based, at least in part, on”. The term “one embodiment” means “at least one embodiment”; the term “another embodiment” means “at least one additional embodiment”; the term “some embodiments” means “at least some embodiments”. Related definitions of other terms will be given in the following description. It should be noted that the concepts of “first”, “second”, etc. mentioned in the present disclosure are only used to distinguish different devices, modules or units, and do not limit the order or interdependence of the functions performed by these devices, modules or units.

[0029] It should be noted that the modification of "one", "multiple" mentioned in the present disclosure is illustrative but not restrictive, and those skilled in the art should understand that unless otherwise explicitly indicated in the context, it should be understood as "one or more".

[0030] The names of the messages or information exchanged between the plurality of devices in the embodiments of the present disclosure are only for illustrative purposes, and are not used to limit the scope of the messages or information.

[0031] In the related art, the method for generating front-end code based on a requirement description document can provide front-end key information, business logic and process information according to user development requirements, and provide front-end key information and generation prompts to a development model, so that the development model outputs software front-end component code and verification scripts. However, this method only analyzes front-end development requirements through pure text of the requirement description, and the style of the front-end page component is easily affected by the code library and component library in the development model, resulting in poor reliability; the overall layout information of the front-end page is lacking, and the association between components is ignored.

[0032] The method for generating front-end code based on edge information of components in a front-end page image can be that the edge information of the front-end page image is first extracted to determine the edge information, and then the edge information is recognized to determine component position and category information, and according to the information, the front-end page image is cut according to components to generate front-end page code corresponding to each component. However, this method introduces edge detection, path search and feature extraction methods, only detects the edge information of the components in the front-end page image, lacks understanding of the overall layout of the front-end page, and ignores the association between components; there can be multiple different paths in path search, and the closure of the edge is uncertain.

[0033] Therefore, in order to solve the above problems, the present disclosure provides a front-end code generation method, device, equipment, storage medium and program product, which trains a pixel semantic similarity model by using front-end page image data, obtains the semantic similarity relationship between pixels and text in the front-end page image, globally extracts feature information of components in the front-end page image, and generates front-end code by using semantic segmentation results. The method can fully combine the details and textures of images in different scales, better understand the position of the components in the overall space of the front-end page and the relationship with other components, and more accurately and reliably generate front-end page code with the aid of the semantic association relationship between pixels.

[0034] The front-end code generation method provided by the embodiments of the present disclosure can be executed by an electronic device or a chip applied to an electronic device. The above-mentioned electronic device can be a terminal or a service platform.

[0035] Exemplarily, the terminal described above can include one or more of a mobile phone, a tablet computer, a wearable device, a vehicle-mounted device, a notebook computer, an ultra-mobile personal computer (UMPC), a netbook, a Personal Digital Assistant (PDA), and a wearable device based on augmented reality (AR) and / or virtual reality (VR) technology, and the like, and the exemplary embodiments of the present disclosure do not specifically limit this.

[0036] Exemplarily, the service platform described above can be a standalone physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server providing cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, Content Delivery Network (CDN), and basic cloud computing services such as big data and artificial intelligence platforms, and the like, and the exemplary embodiments of the present disclosure do not limit this.

[0037] The method for generating front-end code provided by the embodiments of the present disclosure is applied to a front-end code generation system, Figure 1 The architecture schematic diagram of the front-end code generation system provided by the exemplary embodiments of the present disclosure is shown. As shown in the figure, Figure 1 The front-end code generation system 100 can include an image preprocessing module 101, a pixel semantic similarity module 102, and a front-end code generation module 103.

[0038] The image preprocessing module 101 uses a multi-scale feature extraction method to explore the semantic similarity between pixels from multiple dimensions of the front-end page image. The pixel semantic similarity module 102 uses a semantic similarity method for image and text pairs to extract the similarity of image and text pairs about components in the front-end page image from multi-modal data combined with natural language processing and image processing, and obtains more accurate component position information and category information for front-end code generation by maximizing the similarity relationship between component pixels and corresponding text. The above two modules further improve the recognition and positioning accuracy of components in the front-end page image from the semantic point of view, and lay a solid foundation for front-end page code generation. The front-end code generation module 103 uses a code generation model to generate front-end page code, and uses the front-end page image with component semantic information output by the first two modules as training data to improve the accuracy of the model in generating front-end page code.

[0039] The embodiments of the present disclosure fully consider the natural language processing data of the requirement document description class and the UI interface image given by the design draft in actual application, utilize the multi-modal data processing to fully combine the information provided by the two kinds of data, especially extract the rich features contained in the image, and assist the model in component recognition and page understanding by analyzing the semantic similarity relationship of the image-text pair of pixels. In addition, the image is scaled when the input image is trained, so that the network learns the features of the same input data under different angles and scales, fully extracts the rich semantic information contained in the image, and only uses edge information as a starting point for component recognition and segmentation. Semantic extraction for multiple modal data and multiple scale images is realized, the diversity of front-end page components is fully considered, and the accuracy of front-end page component code automatic generation is greatly improved.

[0040] In contrast, the current scheme is more suitable for complex and variable actual application scenarios, can support multiple types of input data for semantic extraction at the same time, obtain more accurate component color, size and other style information, and generate more reliable front-end page code.

[0041] Figure 2 The flowchart of the front-end code generation method provided by the exemplary embodiments of the present disclosure is shown. As shown in Figure 2 The front-end code generation method includes:

[0042] S201, obtaining a multi-scale feature enhancement map of a front-end page image, and obtaining a pre-constructed pixel semantic similarity model;

[0043] S202, processing the multi-scale feature enhancement map using the pixel semantic similarity model to obtain a component text description of the front-end page image; wherein the component text description includes class labels of multiple components in the front-end page image;

[0044] S203, activating the front-end page image using the component text description to obtain an activated front-end page image;

[0045] S204, generating front-end code of the front-end page image using the activated front-end page image.

[0046] Specifically, the multi-scale feature enhancement map can be obtained by a multi-scale feature extraction model, which extracts and enhances features of the front-end page image at multiple different scales. Here, the multi-scale feature extraction model can be a pre-constructed feature extraction model, and the specific network architecture of the model is not limited by the embodiments of the present disclosure.

[0047] The different scales can include an original size, a reduced size and an enlarged size of the front-end page image, and the multi-scale feature enhancement map can include deep feature information of the front-end page image in the original size, the reduced size and the enlarged size, so that richer and effective features are obtained, and more comprehensive and accurate component information in the front-end page image is obtained.

[0048] The pixel semantic similarity model can be a pre-constructed neural network model. The training data of the pixel semantic similarity model can include component text descriptions of the front-end page image and the multi-scale feature enhancement map. In the training stage, the pixel semantic similarity model can be used to learn the similarity between the image (i.e., the pixel) and the text pair about the component in the front-end page image, and the position information and the category information of the component are obtained by maximizing the similarity relationship between the component pixel and the corresponding text.

[0049] In the inference stage, the pixel semantic similarity model can obtain the component text description of the front-end page image based on the multi-scale feature enhancement map covering feature information in multiple different scales. Here, the component text description can include category labels of multiple components in the front-end page image, and the component text description can be used to automatically annotate multiple components contained in the front-end page image, so as to accurately identify and locate the multiple components contained in the front-end page image.

[0050] At this time, the front-end page image can be activated by using the component text description to obtain an activated front-end page image. Here, the "activation" specifies the component position in the front-end page image that matches the component text description, which is similar to "highlighting" the component, and clearly indicates the specific area of the component in the front-end page image, so that the activated front-end page image can carry component semantic information. Further, the front-end code of the front-end page image is generated by using the activated front-end page image, which improves the accuracy of generating the front-end code.

[0051] According to the technical scheme of the example embodiments of the present disclosure, the multi-scale feature enhancement map of the front-end page image is obtained, and the pre-constructed pixel semantic similarity model is obtained. The pixel semantic similarity model is used to process the multi-scale feature enhancement map to obtain the component text description of the front-end page image. The component text description includes category labels of multiple components in the front-end page image. The front-end page image is activated by using the component text description to obtain an activated front-end page image. The front-end code of the front-end page image is generated by using the activated front-end page image. After the component text description is automatically generated by the pixel semantic similarity model based on the multi-scale feature enhancement map covering feature information in multiple different scales, the front-end page image is automatically annotated by using the component text description, the multiple components contained in the front-end page image are accurately identified and located, and the accuracy of generating the front-end code is improved.

[0052] In some embodiments, the multi-scale feature enhancement map of the front-end page image is obtained, comprising:

[0053] obtaining a multi-scale feature map and a multi-scale category activation map of the front-end page image;

[0054] reorganizing and splicing the multi-scale feature map and the multi-scale category activation map to obtain a multi-scale feature enhancement map.

[0055] Specifically, when extracting the image features of the front-end page image, the single-scale feature map has obvious defects. For example, the "small-scale" feature map can only capture details such as text edges and icon pixels, but cannot understand the global correlation of "a certain button belongs to the navigation bar" and "a certain text block belongs to the content area". For another example, the "large-scale" feature map can only identify the page layout structure in the front-end page image, but will lose key details such as "whether the text is clear" and "whether the icon is complete". Here, "small-scale" refers to the reduced size after reducing the original size of the front-end page image, and "large-scale" refers to the enlarged size after enlarging the original size of the front-end page image.

[0056] The above multi-scale feature map can include an original feature map, a reduced feature map, and an enlarged feature map, and the above multi-scale category activation map can include an original category activation map, a reduced category activation map, and an enlarged category activation map. Among them, the size of the original feature map and the original category activation map is the same as the original size of the front-end page image, the size of the reduced feature map and the reduced category activation map is the same as the reduced size of the front-end page image, and the size of the enlarged feature map and the enlarged category activation map is the same as the enlarged size of the front-end page image.

[0057] Here, the reduced size, the original size, and the enlarged size can be 0.5 times, 1 times, and 1.5 times of the front-end page image, respectively. The image input scales of the multi-scale feature extraction model are set to 0.5 times, 1.0 times, and 1.5 times of the front-end page image, respectively, and operations such as horizontal flipping are used to make the network learn the features of the same input data at different angles, increase the diversity of the training data, and generate more stable category activation maps.

[0058] The reason for using these three scales is that the 0.5 times scale mainly captures global features and rough semantic information; the 1.0 times scale maintains the original size of the image and can capture the most natural features and semantic information; and the 1.5 times scale helps to capture more detailed local features and detailed information of small objects.

[0059] The input images of the three scales pass through the convolution layer, the pooling layer, and the full connection layer to obtain the feature maps and the category activation maps of the corresponding three scales. The feature maps of the three scales are: , a reduced feature map , and an enlarged feature map ; the category activation maps of the three scales are respectively: an original category activation map , a reduced category activation map , and an enlarged category activation map .

[0060] Based on this, in the process of obtaining the multi-scale feature map and the multi-scale category activation map of the front-end page image, the front-end page image can be first subjected to size reduction processing and size enlargement processing respectively to obtain a reduced front-end page image and an enlarged front-end page image. Then, the multi-scale feature extraction model is used to perform feature extraction on the reduced front-end page image, the original front-end page image and the enlarged front-end page image respectively to obtain a reduced feature map, an original feature map and an enlarged feature map, and a reduced category activation map, an original category activation map and an enlarged category activation map, that is, to obtain the multi-scale feature map and the multi-scale category activation map. The multi-scale feature map and the multi-scale category activation map are reorganized and spliced to obtain a multi-scale feature enhancement map.

[0061] Exemplarily, the multi-scale feature map and the multi-scale category activation map are reorganized and spliced to obtain a multi-scale feature enhancement map, including:

[0062] The original feature map, the reduced feature map, the enlarged feature map, the original category activation map, the reduced category activation map and the enlarged category activation map are grouped to obtain a plurality of feature map combinations; each feature map combination includes a feature map or a category activation map of an original size, a reduced size and an enlarged size;

[0063] The feature information of the plurality of feature map combinations is fused to obtain a multi-scale feature enhancement map.

[0064] Specifically, the plurality of feature map combinations can include three combinations, the first combination is a reduced feature map , an enlarged feature map and an original category activation map ; the second combination is an original feature map , a reduced category activation map and an enlarged category activation map ; and the third combination is an original category activation map , a reduced category activation map and an enlarged category activation map .

[0065] Then, a 1×1 convolution kernel was used to linearly transform the 0.5x, 1.0x, and 1.5x image channels, fixing them to C2 channels without changing the spatial size of the feature map for channel feature recombination. Finally, each combination simultaneously contains information at three scales. Subsequently, semantic similarity relationships between pixels were mined for each combination to refine the class activation map, and the refined class activation map feature information was summarized. The summarized result is shown in the following formula, which is also the multi-scale feature enhancement map.

[0066] (1)

[0067] in, This represents a multi-scale feature enhancement map, with subscripts indicating the scale. s It is only used to distinguish parameters and has no actual meaning, and is related to k The value of is irrelevant. Indicates the first k Reduced category activation map in the combination, Indicates the first k Enlarged category activation maps in various combinations k It is an integer greater than or equal to 1 and less than or equal to 3.

[0068] Based on this, the embodiments of this disclosure recombine the features of different channels, integrate the low-level feature information of pixels, enable the network to learn richer and more effective features, obtain more comprehensive and accurate front-end page component information for the code generation model, and thus generate more accurate front-end page code.

[0069] In some embodiments, the pixel semantic similarity model includes a text encoder, an image encoder, and a semantic similarity calculation module;

[0070] During the training phase of the pixel semantic similarity model, the text encoder is used to obtain the text feature vector of the component text description, the image encoder is used to obtain the image feature vector of the multi-scale feature enhancement map, and the semantic similarity calculation module is used to calculate the similarity between the text feature vector and the image feature vector.

[0071] Specifically, the input to the pixel semantic similarity model is the component text description and the multi-scale feature enhancement map. During the training phase, the pixel semantic similarity model maps the paired component text descriptions and multi-scale feature enhancement maps into one-dimensional vectors. That is, the text encoder can be used to obtain the text feature vector of the component text description, and the image encoder can be used to obtain the image feature vector of the multi-scale feature enhancement map. The semantic similarity calculation module is used to perform semantic alignment by utilizing the similarity relationship between the two and to supervise the activation of components in the front-end page image.

[0072] Include the category labels of multiple components contained in the front-end page image as part of the text description, in the format: wherein, represents the class label of the i th component, and a set of text descriptions containing all class labels are established as a text supervision auxiliary model to explore a more complete region of the foreground object.

[0073] The pixel semantic similarity model can be a contrastive language-image pretraining model. The pixel semantic similarity model can minimize the loss value by maximizing the relationship between the foreground component and the text description, so as to promote the activation range of the foreground component, realize the alignment of the foreground component and the text description in the front-end page image, and improve the positioning and recognition accuracy of the front-end page component.

[0074] Therefore, the pixel semantic similarity model can maximize the image-text pair semantic similarity of pixels in the front-end page image by a multi-modal method to improve the positioning and classification accuracy of the front-end page component. Meanwhile, a multi-scale feature extraction model is combined to magnify or reduce the front-end page image to sufficiently capture the color, texture, space and other features of the page, so that the information at different scales can complement each other, which is helpful for more accurate description of the component.

[0075] In some embodiments, in the training stage of the pixel semantic similarity model, the loss of the pixel semantic similarity model is determined by the similarity between the text feature vector and the image feature vector; and the calculation formula of the loss of the pixel semantic similarity model is:

[0076] (2)

[0077] wherein, represents the loss of the pixel semantic similarity model; represents the text feature vector of the class label of the i th component in the component text description; represents the image feature vector of the i th component in the multi-scale feature enhancement map; represents the similarity between the text feature vector and the image feature vector; represents whether the i th component is contained in the front-end page image, represents that the i th component is contained in the front-end page image, represents that the i th component is not contained in the front-end page image.

[0078] In some embodiments, the calculation formula of the similarity between the text feature vector and the image feature vector is:

[0079] (3)

[0080] (4)

[0081] (5)

[0082] wherein, denotes the norm of a vector in the cosine function, denotes a feature calculation function related to the category label of the i-th component, i denotes the category label of the i-th component, denotes a feature vector of a front-end page image, i denotes a multi-scale feature enhancement map of the i-th component, I denotes a feature vector of a front-end page image, denotes a multi-scale feature enhancement map of the i-th component, i denotes a feature calculation function related to the multi-scale feature enhancement map of the i-th component. i It should be noted that in formula (1), can be used to calculate the multi-scale feature enhancement map of the front-end page image (i.e., the overall multi-scale feature enhancement map), or can be used to calculate the multi-scale feature enhancement map of the i-th component (i.e., the local multi-scale feature enhancement map). When calculating the multi-scale feature enhancement map of the i-th component,

[0083] i i

[0084] The above mainly introduces the scheme provided by the embodiments of the present disclosure. It can be understood that, in order to realize the above functions, the electronic device contains the hardware structure and / or software module corresponding to each function. Those skilled in the art should easily realize that, in combination with the units and algorithm steps of each example described in the embodiments disclosed in the present text, the present disclosure can be realized in the form of hardware or the combination of hardware and computer software. Whether a certain function is realized in the form of hardware or computer software driving hardware depends on the specific application and design constraints of the technical solution. The skilled person can use different methods to realize the described functions for each specific application, but such implementation should not be considered beyond the scope of the present disclosure.

[0085] ​​​​​​​The embodiments of the present disclosure can divide the functional units of the electronic device according to the above method examples. For example, each functional module can be divided according to each function, or two or more functions can be integrated into one processing module. The integrated module can be implemented in the form of hardware or in the form of a software functional module. It should be noted that the division of the modules in the embodiments of the present disclosure is illustrative, and is only a logical functional division. When actually implemented, another division mode can be used.

[0086] In the case of dividing each functional module according to each function, the present example embodiment provides a front-end code generation device. The front-end code generation device can be an electronic device or a chip applied to an electronic device. Figure 3 A structural schematic diagram of the front-end code generation device provided by the present example embodiment is shown. As shown in Figure 3 The device 300 includes:

[0087] The processing module 301 is configured to obtain a multi-scale feature enhancement map of a front-end page image and a pre-constructed pixel semantic similarity model.

[0088] The processing module 301 is further configured to process the multi-scale feature enhancement map by using the pixel semantic similarity model to obtain a component text description of the front-end page image. The component text description includes category labels of multiple components in the front-end page image.

[0089] The processing module 301 is further configured to activate the front-end page image by using the component text description to obtain an activated front-end page image.

[0090] The generation module 302 is configured to generate a front-end code of the front-end page image by using the activated front-end page image.

[0091] In some embodiments, the processing module 301 is configured to obtain a multi-scale feature map and a multi-scale category activation map of a front-end page image, and reorganize and splice the multi-scale feature map and the multi-scale category activation map to obtain a multi-scale feature enhancement map.

[0092] In some embodiments, the multi-scale feature map includes an original feature map, a reduced feature map, and an enlarged feature map, and the multi-scale category activation map includes an original category activation map, a reduced category activation map, and an enlarged category activation map. The size of the original feature map and the original category activation map is the same as the original size of the front-end page image, the size of the reduced feature map and the reduced category activation map is the same as the reduced size of the front-end page image, and the size of the enlarged feature map and the enlarged category activation map is the same as the enlarged size of the front-end page image.

[0093] The processing module 301 is further configured to group the original feature map, the reduced feature map, the enlarged feature map, the original category activation map, the reduced category activation map and the enlarged category activation map to obtain a plurality of feature map combinations; each of the feature map combinations includes the feature map or the category activation map of the original size, the reduced size and the enlarged size; and fuse feature information of the plurality of feature map combinations to obtain a multi-scale feature enhancement map.

[0094] In some embodiments, the pixel semantic similarity model includes a text encoder, an image encoder and a semantic similarity calculation module.

[0095] In the training phase of the pixel semantic similarity model, the text encoder is configured to obtain a text feature vector of the component text description, the image encoder is configured to obtain an image feature vector of the multi-scale feature enhancement map, and the semantic similarity calculation module is configured to calculate a similarity between the text feature vector and the image feature vector.

[0096] In some embodiments, in the training phase of the pixel semantic similarity model, a loss of the pixel semantic similarity model is determined by the similarity between the text feature vector and the image feature vector; and a calculation formula of the loss of the pixel semantic similarity model is as follows:

[0097] (2)

[0098] wherein, represents a loss of the pixel semantic similarity model; represents a text feature vector of a category label of an i-th component in the component text description; i represents an image feature vector of the i-th component in the multi-scale feature enhancement map; represents a similarity between the text feature vector and the image feature vector; i represents whether the i-th component is contained in the front-end page image, represents that the front-end page image contains the i-th component, represents that the front-end page image does not contain the i-th component. i i i

[0099] In some embodiments, a calculation formula of the similarity between the text feature vector and the image feature vector is as follows:

[0100] (3)

[0101] ​​​​​ (4)

[0102] (5)

[0103] in, This represents the magnitude of the vector in the cosine function. Indicates the relationship with the first i The feature calculation function related to the category label of each component. Indicates the relationship with the first i Category labels for each component, I This represents the feature vector of the front-end page image. Indicates the first i Multi-scale feature enhancement maps of each component. Indicates the relationship with the first i Feature computation functions related to the multi-scale feature enhancement map of each component.

[0104] This disclosure also provides an electronic device, including a memory, a processor, and a computer program stored in the memory, wherein the processor executes the computer program to implement the steps of the methods disclosed in this disclosure.

[0105] Figure 4 A schematic diagram of the structure of an electronic device provided in an exemplary embodiment of this disclosure is shown. For example... Figure 4 As shown, the electronic device 400 includes at least one processor 401 and a memory 402 coupled to the processor 401, which can perform the corresponding steps in the methods disclosed in the embodiments of this disclosure.

[0106] The processor 401 can also be referred to as a central processing unit (CPU), which can be an integrated circuit chip that has the processing capability of signals. Each step in the method disclosed in the embodiments of the present disclosure can be completed by the integrated logic circuit of hardware or the instructions in the form of software in the processor 401. The processor 401 can be a general-purpose processor, a digital signal processor (DSP), an ASIC, a field-programmable gate array (FPGA) or other programmable logic device, a discrete gate or transistor logic device, a discrete hardware component. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor. The steps of the method disclosed in conjunction with the embodiments of the present disclosure can be directly embodied as hardware code processing for execution, or executed by a combination of hardware and software modules in the code processing. The software module can be located in the memory 402, such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory or an electrically erasable programmable memory, a register, and other mature storage media in the art. The processor 401 reads the information in the memory 402, and completes the steps of the above method in conjunction with the hardware thereof.

[0107] In addition, various operations / processes according to the present disclosure, when implemented by software and / or firmware, can be downloaded from a storage medium or a network to a computer system with a dedicated hardware structure, for example, Figure 5 The computer system 500 shown is installed with programs constituting the software, and when various programs are installed, the computer system can perform various functions, including functions such as those described above. Figure 5 A structural schematic diagram of a computer system provided by an exemplary embodiment of the present disclosure is shown.

[0108] The computer system 500 is intended to represent various forms of digital electronic computer devices, such as laptops, desktops, workstations, personal digital assistants, servers, blade servers, mainframes, and other appropriate computers. The electronic device can also represent various forms of mobile devices, such as personal digital processors, cellular telephones, smartphones, wearable devices, and other similar computing devices. The components shown in the figures, their connections and relationships, and their functions, are merely examples, and are not intended to limit the implementations of the present disclosure described and / or claimed herein.

[0109] As Figure 5As shown, the computer system 500 includes a computing unit 501 that can perform various appropriate actions and processes in accordance with a computer program stored in a read-only memory (ROM) 502 or a computer program loaded into a random access memory (RAM) 503 from a storage unit 508. Various programs and data required for the operation of the computer system 500 can also be stored in the RAM 503. The computing unit 501, the ROM 502, and the RAM 503 are connected to each other through a bus 504. An input / output (I / O) interface 505 is also connected to the bus 504.

[0110] A plurality of components in the computer system 500 are connected to the I / O interface 505, including an input unit 506, an output unit 507, a storage unit 508, and a communication unit 509. The input unit 506 can be any type of device that can input information to the computer system 500, and can receive inputted digital or character information, and generate key signal inputs related to user settings and / or function controls of the electronic device. The output unit 507 can be any type of device that can present information, and can include, but is not limited to, a display, a speaker, a video / audio output terminal, a vibrator, and / or a printer. The storage unit 508 can include, but is not limited to, a magnetic disk, an optical disk. The communication unit 509 allows the computer system 500 to exchange information / data with other devices through a network such as the Internet, and can include, but is not limited to, a modem, a network card, an infrared communication device, a wireless communication transceiver, and / or a chipset, e.g., a Bluetooth™ device, a WiFi device, a WiMax device, a cellular communication device, and / or the like.

[0111] The computing unit 501 can be various general and / or special purpose processing components with processing and computing capabilities. Some examples of the computing unit 501 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any appropriate processor, controller, microcontroller, etc. The computing unit 501 performs various methods and processes described above. For example, in some embodiments, the above-described methods disclosed by embodiments of the present disclosure can be implemented as a computer software program tangibly embodied in a machine-readable medium, e.g., the storage unit 508. In some embodiments, part or all of the computer program can be loaded and / or installed onto the electronic device via the ROM 502 and / or the communication unit 509. In some embodiments, the computing unit 501 can be configured to perform the above-described methods disclosed by embodiments of the present disclosure by any other appropriate means, e.g., by means of firmware.

[0112] The embodiment of the present disclosure further provides a computer readable storage medium, wherein when instructions in the computer readable storage medium are executed by a processor of an electronic device, the electronic device is enabled to perform the above method disclosed by the embodiment of the present disclosure.

[0113] The computer readable storage medium in the embodiment of the present disclosure can be a tangible medium, which can contain or store programs for use by or in connection with an instruction execution system, apparatus or device. The above computer readable storage medium can include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus or device, or any suitable combination of the above. More specifically, the above computer readable storage medium can include one or more wire-based electrical connections, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the above.

[0114] The above computer readable medium can be included in the above electronic device; or can exist separately without being assembled into the electronic device.

[0115] The embodiment of the present disclosure further provides a computer program product, comprising a computer program, wherein the computer program is executed by a processor to implement the above method disclosed by the embodiment of the present disclosure.

[0116] In the embodiments of the present disclosure, the computer program code for performing the operations of the present disclosure can be written in one or more programming languages or combinations thereof, including but not limited to object-oriented programming languages such as Java, Smalltalk, C++, and conventional procedural programming languages such as "C" language or similar programming languages. The program code can be executed entirely on a user computer, partially on a user computer, as an independent software package, partially on a user computer and partially on a remote computer, or entirely on a remote computer or server. In the case involving a remote computer, the remote computer can be connected to the user computer through any kind of network including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computer.

[0117] The flow diagrams and the block diagrams in the drawings are illustrations of architectures, functionalities, and operations of possible implementations of systems, methods, and computer program products according to various embodiments of present disclosure. In this regard, each block in the flow diagrams or block diagrams can represent a module, a segment, or a portion of code, which comprises one or more executable instructions for implementing the specified logical function(s). It should also be noted that in some alternative implementations, the functions noted in the block can occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. It will also be noted that each block of the block diagrams and / or flow diagrams, and combinations thereof, can be implemented by special purpose hardware-based systems that perform the specified functions or operations, or combinations of special purpose hardware and computer instructions.

[0118] The modules, components or units described in the embodiments of the present disclosure can be implemented by software or by hardware. In some cases, the name of the module, component or unit does not constitute a limitation on the module, component or unit itself.

[0119] The functions described above in this detailed description can be performed in at least part by one or more hardware logic components. For example, and without limitation, illustrative types of hardware logic components that can be used include Field-programmable Gate Arrays (FPGAs), Application-specific Integrated Circuits (ASICs), Application-specific Standard Products (ASSPs), System-on-a-chip systems (SOCs), Complex Programmable Logic Devices (CPLDs), etc.

[0120] The above description is merely some embodiments of the present disclosure and a description of principles of technology used. It should be understood by those skilled in the art that the disclosed scope of the present disclosure is not limited to the technical solutions formed by the specific combinations of the above technical features, and also covers other technical solutions formed by the combinations of the above technical features or equivalent features without departing from the above disclosed concept. For example, the technical solutions formed by the mutual replacement of the above features and the technical features disclosed in the present disclosure (but not limited to) having similar functions.

[0121] Although some specific embodiments of the present disclosure have been described in detail by way of examples, it should be understood that modifications can be made to the above embodiments without departing from the scope and spirit of the present disclosure. The scope of the present disclosure is defined by the appended claims.

Claims

1. A method for generating front-end code, characterized in that, include: Obtain multi-scale feature enhancement maps of front-end page images and obtain pre-built pixel semantic similarity models; The pixel semantic similarity model is used to process the multi-scale feature enhancement map to obtain the component text description of the front-end page image; wherein, the component text description includes the category labels of multiple components in the front-end page image; The front-end page image is activated using the component text description to obtain an activated front-end page image; The front-end code that generates the front-end page image using the activated front-end page image; The acquisition of the multi-scale feature enhancement map of the front-end page image includes: Obtain multi-scale feature maps and multi-scale class activation maps of the front-end page image; The multi-scale feature map and the multi-scale category activation map are recombined and stitched together to obtain a multi-scale feature enhancement map.

2. The method as described in claim 1, characterized in that, The multi-scale feature map includes an original feature map, a scaled-down feature map, and a magnified feature map; the multi-scale category activation map includes an original category activation map, a scaled-down category activation map, and a magnified category activation map; wherein, the original feature map and the original category activation map are the same size as the original size of the front-end page image, the scaled-down feature map and the scaled-down category activation map are the same size as the scaled-down size of the front-end page image, and the magnified feature map and the magnified category activation map are the same size as the magnified size of the front-end page image; The step of recombining and splicing the multi-scale feature map and the multi-scale category activation map to obtain a multi-scale feature enhancement map includes: The original feature map, the reduced feature map, the enlarged feature map, the original category activation map, the reduced category activation map, and the enlarged category activation map are grouped to obtain multiple sets of feature map combinations; wherein each set of feature map combinations includes the original size, the reduced size, and the enlarged size of the feature map or category activation map; The feature information of the multiple sets of feature maps is fused to obtain a multi-scale feature enhancement map.

3. The method as described in claim 1, characterized in that, The pixel semantic similarity model includes a text encoder, an image encoder, and a semantic similarity calculation module; During the training phase of the pixel semantic similarity model, the text encoder is used to obtain the text feature vector of the component text description, the image encoder is used to obtain the image feature vector of the multi-scale feature enhancement map, and the semantic similarity calculation module is used to calculate the similarity between the text feature vector and the image feature vector.

4. The method as described in claim 3, characterized in that, During the training phase of the pixel semantic similarity model, the loss of the pixel semantic similarity model is determined by the similarity between the text feature vector and the image feature vector; the formula for calculating the loss of the pixel semantic similarity model is as follows: in, This represents the loss of the pixel semantic similarity model; The first element in the component text description i The text feature vectors of the category labels of each component; The first feature in the multi-scale feature enhancement graph represents the... i Image feature vectors of each component; Indicates the similarity between text feature vectors and image feature vectors; Indicates whether the image on the front-end page contains the first... i Each component This indicates that the front-end page image contains the first... i Each component This indicates that the front-end page image does not contain the first... i Each component.

5. The method as described in claim 4, characterized in that, The formula for calculating the similarity between the text feature vector and the image feature vector is as follows: in, This represents the magnitude of the vector in the cosine function. Indicates the relationship with the first i The feature calculation function related to the category label of each component. Indicates the relationship with the first i Category labels for each component, I This represents the feature vector of the front-end page image. Indicates the first i Multi-scale feature enhancement maps of each component. Indicates the relationship with the first i Feature computation functions related to the multi-scale feature enhancement map of each component.

6. A front-end code generation device, characterized in that, include: The acquisition module is used to acquire multi-scale feature enhancement maps of front-end page images and to acquire pre-built pixel semantic similarity models; The processing module is used to process the multi-scale feature enhancement map using the pixel semantic similarity model to obtain the component text description of the front-end page image; wherein, the component text description includes category labels of multiple components in the front-end page image; The processing module is used to activate the front-end page image using the component text description to obtain an activated front-end page image; A generation module is used to generate front-end code for the front-end page image using the activated front-end page image; The acquisition module is also used to acquire multi-scale feature maps and multi-scale category activation maps of the front-end page image; and to reassemble and stitch the multi-scale feature maps and multi-scale category activation maps to acquire multi-scale feature enhancement maps.

7. An electronic device comprising a memory, a processor, and a computer program stored in the memory, characterized in that, The processor executes the computer program to implement the steps of the method according to any one of claims 1 to 5.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that, When executed by a processor, the computer program implements the steps of the method described in any one of claims 1 to 5.

9. A computer program product, comprising a computer program, characterized in that, When executed by a processor, the computer program implements the steps of the method described in any one of claims 1 to 5.

Images