Model pre-training method and device, HTML page processing method and device

Abstract

Embodiments of the present specification provide a model pre-training method and device, an HTML page processing method and device, wherein a model pre-training method comprises: preprocessing HTML code in a training sample to obtain HTML code features; inputting the HTML code features and page images in the training sample into a to-be-trained model for feature processing, and outputting a feature processing result; the HTML code and the page images correspond to a same HTML page; the to-be-trained model comprises an embedding layer and an encoder, the embedding layer performs feature conversion on the HTML code features and the page images to output code feature vectors and image feature vectors, the encoder performs fusion calculation on the code feature vectors and the image feature vectors to output the feature processing result; and at least one pre-training task of pre-training the to-be-trained model is performed based on the feature processing result.

Description

TECHNICAL FIELD

[0001] The present document relates to the technical field of HTML, and particularly relates to a model pre-training method and device and an HTML page processing method and device. BACKGROUND

[0002] An HTML (Hyper Text Mark-up Language) page is a typical rich information document page, which is composed of text, images and other multi-modal information. Text, images, icons and buttons each have independent semantic information. The combination of multiple same or different types of blocks can express higher-dimensional semantic information. A page developer combines fine-grained blocks organically through structural layout to describe the page. Therefore, how to model the blocks and the layout reasonably becomes a key and difficulty of HTML page understanding. SUMMARY

[0003] One or more embodiments of the present specification provide a model pre-training method, comprising: pre-processing HTML code in a training sample to obtain HTML code features. The HTML code features and page images in the training sample are input into a to-be-trained model for feature processing, and a feature processing result is output. The HTML code and the page images correspond to the same HTML page. The to-be-trained model comprises an embedding layer and an encoder. The embedding layer performs feature conversion on the HTML code features and the page images to output code feature vectors and image feature vectors. The encoder performs fusion calculation on the code feature vectors and the image feature vectors to output the feature processing result. At least one pre-training task of pre-training the to-be-trained model is performed based on the feature processing result.

[0004] One or more embodiments of the present specification provide an HTML page processing method, comprising: pre-processing HTML code to be processed to obtain HTML code features. The HTML code features and page images to be processed are input into an HTML page processing model for page feature processing, and a page processing result is output. The HTML page processing model comprises an embedding layer, an encoder and a page processing layer. The embedding layer performs feature conversion on the HTML code features and the page images to output code feature vectors and image feature vectors. The encoder performs fusion calculation on the code feature vectors and the image feature vectors to output an encoding vector. The page processing layer performs page processing based on the encoding vector to output the page processing result.

[0005] One or more embodiments of the present specification provide a model pre-training apparatus, comprising: a preprocessing module configured to preprocess HTML code in a training sample to obtain HTML code features. A feature processing module is configured to input the HTML code features and page images in the training sample into a to-be-trained model for feature processing, and output a feature processing result. The HTML code and the page images correspond to the same HTML page. The to-be-trained model includes an embedding layer and an encoder. The embedding layer performs feature conversion on the HTML code features and the page images to output code feature vectors and image feature vectors. The encoder performs fusion calculation on the code feature vectors and the image feature vectors to output the feature processing result. A pre-training task execution module is configured to execute at least one pre-training task for pre-training the to-be-trained model based on the feature processing result.

[0006] One or more embodiments of the present specification provide an HTML page processing apparatus, comprising: a preprocessing module configured to preprocess HTML code to be processed to obtain HTML code features. A page feature processing module is configured to input the HTML code features and page images to be processed into an HTML page processing model for page feature processing, and output a page processing result. The HTML page processing model includes an embedding layer, an encoder, and a page processing layer. The embedding layer performs feature conversion on the HTML code features and the page images to output code feature vectors and image feature vectors. The encoder performs fusion calculation on the code feature vectors and the image feature vectors to output an encoding vector. The page processing layer performs page processing based on the encoding vector to output the page processing result.

[0007] One or more embodiments of the present specification provide a model pre-training device, comprising: a processor; and a memory configured to store computer executable instructions that, when executed, cause the processor to: preprocess HTML code in a training sample to obtain HTML code features. Input the HTML code features and page images in the training sample into a to-be-trained model for feature processing, and output a feature processing result. The HTML code and the page images correspond to the same HTML page. The to-be-trained model includes an embedding layer and an encoder. The embedding layer performs feature conversion on the HTML code features and the page images to output code feature vectors and image feature vectors. The encoder performs fusion calculation on the code feature vectors and the image feature vectors to output the feature processing result. At least one pre-training task for pre-training the to-be-trained model is executed based on the feature processing result.

[0008] One or more embodiments of the present specification provide an HTML page processing device, comprising: a processor; and a memory configured to store computer executable instructions that, when executed, cause the processor to: preprocess HTML code to be processed to obtain HTML code features. Input the HTML code features and a page image to be processed into an HTML page processing model for page feature processing, and output a page processing result. Wherein the HTML page processing model comprises an embedding layer, an encoder and a page processing layer; the embedding layer converts features of the HTML code and the page image, and outputs code feature vectors and image feature vectors. The encoder performs fusion calculation on the code feature vectors and the image feature vectors, and outputs an encoding vector. The page processing layer performs page processing based on the encoding vector, and outputs the page processing result.

[0009] One or more embodiments of the present specification provide a storage medium for storing computer executable instructions, which, when executed by a processor, implement the following processes: preprocessing HTML code in a training sample to obtain HTML code features. Input the HTML code features and a page image in the training sample into a to-be-trained model for feature processing, and output a feature processing result. The HTML code and the page image correspond to the same HTML page. The to-be-trained model comprises an embedding layer and an encoder. The embedding layer converts features of the HTML code and the page image, and outputs code feature vectors and image feature vectors. The encoder performs fusion calculation on the code feature vectors and the image feature vectors, and outputs the feature processing result. At least one pre-training task of pre-training the to-be-trained model is performed based on the feature processing result.

[0010] One or more embodiments of the present specification provide another storage medium for storing computer executable instructions, which, when executed by a processor, implement the following processes: preprocessing HTML code to be processed to obtain HTML code features. Input the HTML code features and a page image to be processed into an HTML page processing model for page feature processing, and output a page processing result. Wherein the HTML page processing model comprises an embedding layer, an encoder and a page processing layer; the embedding layer converts features of the HTML code and the page image, and outputs code feature vectors and image feature vectors. The encoder performs fusion calculation on the code feature vectors and the image feature vectors, and outputs an encoding vector. The page processing layer performs page processing based on the encoding vector, and outputs the page processing result. BRIEF DESCRIPTION OF DRAWINGS

[0011] In order to more clearly illustrate the technical solutions in the one or more embodiments of the present specification or the prior art, the drawings needed to be used in the embodiment or prior art description will be briefly introduced as follows. Obviously, the drawings in the following description are only some embodiments described in the present specification, and for those skilled in the art, other drawings can also be obtained from these drawings without creative labor.

[0012] Figure 1 A model pre-training method processing flowchart is provided for one or more embodiments of the present specification.

[0013] Figure 2 A schematic diagram of an applet page is provided for one or more embodiments of the present specification.

[0014] Figure 3 A schematic diagram of an HTML code is provided for one or more embodiments of the present specification.

[0015] Figure 4 A schematic diagram of a DOM tree is provided for one or more embodiments of the present specification.

[0016] Figure 5 A schematic diagram of the correspondence between the HTML code and the applet page is provided for one or more embodiments of the present specification.

[0017] Figure 6 An architecture block diagram of a to-be-trained model is provided for one or more embodiments of the present specification.

[0018] Figure 7 An architecture block diagram of an HTML page processing model is provided for one or more embodiments of the present specification.

[0019] Figure 8 An HTML page processing method processing flowchart is provided for one or more embodiments of the present specification.

[0020] Figure 9 A model pre-training device schematic diagram is provided for one or more embodiments of the present specification.

[0021] Figure 10 An HTML page processing device schematic diagram is provided for one or more embodiments of the present specification.

[0022] Figure 11 A structure schematic diagram of a model pre-training equipment is provided for one or more embodiments of the present specification.

[0023] Figure 12 A structure schematic diagram of an HTML page processing equipment is provided for one or more embodiments of the present specification. DETAILED DESCRIPTION

[0024] In order to make the person skilled in the art better understand the technical solutions in one or more embodiments of the present specification, the technical solutions in one or more embodiments of the present specification will be clearly and completely described below in conjunction with the drawings in one or more embodiments of the present specification. Obviously, the described embodiments are only a part of the embodiments of the present specification, not all. Based on one or more embodiments of the present specification, all other embodiments obtained by those skilled in the art without creative labor should belong to the protection scope of the present document.

[0025] The model pre-training method provided in the present specification comprises the following steps:

[0026] Referring to Figure 1 The model pre-training method provided in the present embodiment specifically comprises steps S102 to S106.

[0027] In step S102, the HTML code in the training sample is preprocessed to obtain HTML code features.

[0028] The model pre-training method provided in the present application takes HTML code and page images as training samples for pre-training the to-be-trained model. In the pre-training process, the HTML code is preprocessed, the HTML code features obtained by preprocessing are taken as pre-training inputs, the to-be-trained model is inputted to perform feature conversion processing of vectorization on the HTML code features and the page images, and on the basis of the obtained code feature vectors and image feature vectors, fusion calculation is performed on the code feature vectors of the code modality and the image feature vectors of the image modality, so as to realize the fusion of the HTML page features of the code modality and the image modality, and at least one pre-training task is performed based on the feature processing result output by the to-be-trained model, so as to realize the pre-training processing of the to-be-trained model. Through the pre-training processing, the to-be-trained model can efficiently learn the recognition and processing ability of the HTML page based on the fusion of the code modality and the image modality, and at the same time, the pre-training can improve the efficiency of the model training process and the effect of the model training.

[0029] The HTML code described in the present embodiment refers to the HTML code of the HTML page implemented by HTML. It should be noted that the HTML page includes an access page of a subprogram or an access page of an application program implemented by HTML and application program technology, and also includes a Web page implemented by HTML only. If the HTML page is an access page of a subprogram, the HTML code is the HTML code of the access page (subprogram page) of the subprogram.

[0030] The subprogram includes a program function module or an application component carried by the host application or a program function module or an application component installed by the host application, such as a small program in an application program. From the perspective of service, the subprogram has the ability to independently provide a self-closed loop service, such as a small program running in an application program and having the self-closed loop ability of performing corresponding service processing.

[0031] In a specific implementation, in the process of preprocessing the HTML code, a DOM tree is constructed according to a hierarchical relationship between elements of an HTML page contained in the HTML code, starting from the elements of the HTML page contained in the HTML code, and a node feature of a node contained in the DOM tree is taken as a part of an input of pre-training, so as to enhance the understanding ability of the to-be-trained model for the HTML page on the basis of the node feature of the node contained in the DOM tree through the pre-training process.

[0032] In practice, the HTML code may contain dozens or even hundreds of elements, therefore, in order to reduce the difficulty of the to-be-trained model for understanding the HTML page in the pre-training process and improve the pre-training efficiency, an optional implementation provided in the embodiment includes the following steps of preprocessing the HTML code in the training sample to obtain an HTML code feature:

[0033] (1) constructing a DOM tree based on the HTML code and pruning the DOM tree to obtain a pruned DOM tree.

[0034] It should be noted that the elements contained in the HTML code are in one-to-one correspondence with the nodes contained in the DOM tree, and the hierarchical relationship of the nodes contained in the DOM tree is in one-to-one mapping with the hierarchical relationship of the elements contained in the HTML code, for example, a div element contained in the HTML code is hierarchically adjacent to an image unit and is in an upper layer of the image unit, then a node corresponding to the div element in the DOM tree is hierarchically adjacent to a node corresponding to the image unit, and the node corresponding to the div element is also in an upper layer of the node corresponding to the image unit.

[0035] Here, the pruning of the DOM tree refers to retaining nodes corresponding to key elements in the DOM tree and deleting nodes corresponding to non-key elements in the DOM tree. The key element refers to an element carrying key information in the HTML code, and the non-key element refers to an element not carrying key information, wherein the key information refers to information irrelevant to the type or service data of the HTML page. The pruned DOM tree refers to the DOM tree after the nodes corresponding to the non-key elements are deleted.

[0036] In an optional implementation of the embodiment, the DOM tree is pruned in the following manner: for any node included in the DOM tree, it is detected whether the element corresponding to the node is a key element; if yes, no further processing is needed; if not, the node is deleted from the nodes included in the DOM tree.

[0037] (2) The nodes included in the pruned DOM tree are classified to obtain a classified DOM tree, and the node features of the classified nodes included in the classified DOM tree are determined as the HTML code features.

[0038] To further reduce the difficulty of understanding the HTML page for the to-be-trained model in the pre-training process and improve the pre-training efficiency, in an optional implementation of the embodiment, the nodes included in the pruned DOM tree obtained through pruning are classified, so that the categories of the nodes included in the pruned DOM tree can be reduced under the premise of retaining the element information corresponding to the nodes. In this way, the difficulty of understanding the HTML page is reduced by reducing the categories of the nodes. Specifically, the classification of the nodes included in the pruned DOM tree includes:

[0039] According to the element features or element information of the elements corresponding to the nodes included in the pruned DOM tree, the classification categories of the nodes are determined.

[0040] Based on the classification categories of the nodes, the node identifiers of the nodes in the pruned DOM tree are updated to obtain the classified DOM tree.

[0041] Based on this, the node features of the classified nodes included in the classified DOM tree are determined, and the obtained node features are input into the to-be-trained model as the HTML code features to pre-train the to-be-trained model. To structure the node features input into the to-be-trained model, so as to standardize the model pre-training input, the embodiment structures the node features. Specifically, in the process of extracting the node features of the classified nodes included in the classified DOM tree from the HTML code, the types of the classified nodes are distinguished, and the following optional implementations are used to structure the non-text nodes and the text nodes, respectively.

[0042] In an optional implementation of the embodiment, the extracting the node features of the classified nodes included in the classified DOM tree from the HTML code comprises: if the classified node is a non-text node, performing depth-first traversal on the classified DOM tree to obtain a node number of the non-text node; determining image coordinate information of an element corresponding to the non-text node in the page image, and obtaining a parent node number of a parent node of the non-text node; and taking the node number, the image coordinate information, and the parent node number as the node features of the non-text node.

[0043] Further, if the classified node is a text node, the following operations are performed: performing depth-first traversal on the classified DOM tree to obtain a node number of the text node; determining image coordinate information of the text node in the page image, and obtaining a parent node number of a parent node of the text node; splitting text information of an element corresponding to the text node into text tokens and assigning token numbers to the text tokens, and obtaining text features of the element; and taking the node number, the image coordinate information, the parent node number, the token numbers, and the text features as the node features of the text node.

[0044] For example, a mini-program page of a mini-program for recycling mobile phones is as shown in Figure 2 The code of the mini-program page is as shown in Figure 3 A DOM tree is constructed based on the HTML code, and for the obtained DOM tree, it is detected whether an element corresponding to a node included in the DOM tree is a key element; if yes, no processing is performed; if not, the node corresponding to a non-key element is deleted from the DOM tree to obtain a pruned DOM tree; then, element information of an element corresponding to a node included in the pruned DOM tree is determined to determine a classification category of the node, and a node identifier of the node in the pruned DOM tree is updated based on the classification category of the node, and the updated classified DOM tree is as shown in Figure 4

[0045] The nodes in the classified DOM tree correspond to the elements included in the HTML code, and the hierarchical relationship of the nodes included in the classified DOM tree is one-to-one mapped with the hierarchical relationship of the elements included in the HTML code. In order to visually display the correspondence between the nodes in the classified DOM tree and the elements in the HTML code, the correspondence between the nodes in the classified DOM tree and the elements in the HTML code is displayed by means of an HTML page to which the HTML code belongs, as shown in Figure 5 ​It should be noted that the nodes in the DOM tree, the pruned DOM tree and the classified DOM tree are similar, and the nodes in the DOM tree and the pruned DOM tree correspond to the elements contained in the HTML code, and the hierarchical relationship of the nodes contained in the DOM tree and the pruned DOM tree is mapped to the hierarchical relationship of the elements contained in the HTML code, which will not be described here.

[0046] Finally, the node features of the classified nodes contained in the classified DOM tree are determined, and the corresponding relationship of the features required for different classified nodes is as follows:

[0047] Categorization node Base feature Supplementary feature div Node number, image two-dimensional coordinate Parent node number img Node number, image two-dimensional coordinate Parent node number text Node number, image two-dimensional coordinate Parent node number token Node number, image two-dimensional coordinate Token number, font, font size, color

[0048] The node number is determined by traversing the classified DOM tree, and the traversal is performed in a depth-first manner during the traversal process. The two-dimensional coordinates refer to the two-dimensional coordinate information of the element corresponding to the classified node in the page image shown in Figure 2 The token node is obtained by splitting the text corresponding to the text node. The text corresponding to the text node is split into the form of token. The token is numbered, and the node corresponding to the token is the token classified node. The font, size and color of the token classified node after the text classified node is split are added to the token classified node by the CSS information.

[0049] In step S104, the HTML code features and the page image in the training sample are input into the to-be-trained model for feature processing, and a feature processing result is output.

[0050] In this embodiment, the to-be-trained model includes an embedding layer and an encoder. The embedding layer performs feature conversion on the HTML code features and the page image, and outputs code feature vectors and image feature vectors. The encoder performs fusion calculation on the code feature vectors and the image feature vectors, and outputs the feature processing result.

[0051] The HTML code and the page image correspond to the same HTML page, that is, the page image and the HTML code are the page image and the HTML code of the same HTML, such as the page image and the HTML code shown in Figure 2 Figure 3 ​The illustrated HTML code is the page image and HTML code of the same applet page (applet access homepage) of the applet for providing mobile phone recycling. Moreover, the page image can be obtained based on the rendering of the HTML code, such as calling the HTML page rendering component integrated by the host application of the subprogram, obtaining the page image based on the page rendering of the HTML code, or calling the third-party HTML page rendering tool to obtain the page image based on the page rendering of the HTML code.

[0052] The feature conversion refers to vector conversion of the HTML code feature and the page image in the embedding layer of the model to be trained, to vectorized expression. Optionally, the embedding layer includes a code feature encoder and an image feature encoder. The feature conversion of the HTML code feature and the page image includes:

[0053] The HTML code feature is input into the code feature encoder for code feature encoding processing to obtain the code feature vector, and the page image is input into the image feature encoder for image encoding processing to obtain the image feature vector.

[0054] In addition, the embedding layer can also include an adaptive pool (Adaptive Pool). After the page image is input into the image feature encoder for image encoding processing to obtain the encoding result, the adaptive pool is used to convert the encoding result into an image feature vector. Alternatively, the embedding layer can also split the input page image into a plurality of image units. After the plurality of image units are input into the image feature encoder for image encoding processing to obtain the encoding result, the adaptive pool is used to convert the encoding result into an image feature vector. Alternatively, the page image is split into a plurality of image units before being input into the embedding layer. After the plurality of image units are input into the image feature encoder for image encoding processing to obtain the encoding result, the adaptive pool is used to convert the encoding result into an image feature vector.

[0055] In the specific implementation process, the page image can be split into a plurality of image blocks (image units) of equal size, or the page image can be split into a plurality of image blocks (image units) of image elements according to the hierarchical relationship between the image elements in the page image.

[0056] For example, Figure 6A model architecture of a to-be-trained model is shown, the to-be-trained model is a BERT (Bidirectional Encoder Representation from Transformers) model, the to-be-trained model comprises an embedding layer and an encoder, the embedding layer is configured with two encoders, the left encoder is used for encoding processing of a page image of an input small program page, and the specific processing process is as follows:

[0057] Firstly, the page image is split into a plurality of image units (Patch_1-Patch_N), and then the image features of each image unit are input into the left encoder for encoding processing, and the encoding processing result is converted into an image feature vector through adaptive pooling, and the image feature vector is specifically represented as:

[0058] E_img=ImgEncoder(img)+E(index_img)+E(2d-position)+E(segment_img);

[0059] Wherein, E_img represents the image feature vector, ImgEncoder(img) represents the vector obtained by the image unit through the encoder, E(index_img) represents the feature vector corresponding to the image unit, E(2d-position) represents the feature vector corresponding to the two-dimensional coordinates of the image unit, and E(segment_img) represents the feature vector corresponding to the modal identifier, and the modal identifier takes a value of 0 to represent a modal, and the modal identifier takes a value of 1 to represent a text modal.

[0060] The right encoder is used for encoding processing of the HTML code features of the input small program page, and the specific processing process is as follows:

[0061] The HTML code features are input into the right encoder for encoding processing, and the code feature vector of the HTML code features is output, and the code feature vector is specifically represented as:

[0062] E_dom=E(token)+E(DFS_id)+E(DFS_pid)+E(token_id)+E(token_font-size)+E(token_font-style)+E(token_font-color)+E(segment_dom);

[0063] Wherein, E_dom represents the code feature vector, E(token) represents the feature vector of token (character), E(DFS_id) represents the feature vector corresponding to the node number of each classification node obtained by traversing the classification DOM tree in the DFS (depth first traversal) traversal mode, E(DFS_pid) represents the feature vector corresponding to the node number of the parent node of each classification node obtained by traversing the classification DOM tree in the DFS (depth first traversal) traversal mode, E(token_id) represents the feature vector corresponding to the token number of token (character), E(token_font-size) represents the feature vector corresponding to the font size type of token (character), E(token_font-style) represents the feature vector corresponding to the font type of token (character), E(token_font-color) represents the feature vector corresponding to the font type of token (character), E(segment_dom) represents the feature vector corresponding to the modal identifier, and the modal identifier takes the value of 0 to represent the modal, and the modal identifier takes the value of 1 to represent the text modal.

[0064] The fusion calculation refers to multi-modal fusion calculation, and specifically refers to feature fusion calculation on the code modal code feature vector and the image modal image feature vector. In an optional implementation provided by the embodiment, the code feature vector and the image feature vector are fused and calculated, and the following method is specifically adopted to realize the fusion calculation: self-attention calculation is performed on the code feature vector and the image feature vector, and the obtained self-attention vector is taken as the feature processing result.

[0065] In the specific execution process, the encoder can perform calculation by using a self-attention algorithm (Self-Attention) in the process of performing self-attention calculation on the code feature vector and the image feature vector, and the encoder can adopt an encoder under a Transformer architecture.

[0066] Step S106, at least one pre-training task of pre-training the to-be-trained model is performed based on the feature processing result.

[0067] In this embodiment, a pre-training task for pre-training the to-be-trained model is created in advance. In the pre-training process, the training sample is input into the to-be-trained model, and the parameters of the to-be-trained model are adjusted by performing the pre-training task, so that the to-be-trained model can learn the relationship between the HTML code and the page image contained in the training sample. Specifically, in the pre-training process, the parameters of the embedding layer and the encoder can be adjusted according to the task execution result of the pre-training task; in addition, the training loss can also be determined according to the task execution result of the pre-training task and the training sample, so that the parameters of the embedding layer and the encoder are adjusted according to the training loss.

[0068] Optionally, the pre-training task includes at least one of the following: a semantic recognition task of predicting a mask element in the HTML code, a matching task of matching an element in the HTML code with an image unit in the page image, and an image area prediction task of predicting a position area of a mask image unit in the page image.

[0069] In the case where the pre-training task includes the semantic recognition task of predicting the mask element in the HTML code, the HTML code contained in the training sample also correspondingly contains or is provided with the mask element. The mask element refers to a mask processing result obtained by performing mask processing on a specific element in the HTML code, for example, randomly selecting 15% of the tokens in the HTML code to perform mask processing. The semantic recognition task refers to a prediction task of predicting the mask element in the HTML code by the to-be-trained model, as shown in pre-training task 1 in FIG. 6, and the parameters of the to-be-trained model are adjusted according to the task execution result of the semantic recognition task, so that the to-be-trained model can learn the semantic relationship between the elements in the HTML code through the semantic recognition task. Figure 6

[0070] In the case where the pre-training task includes the matching task of matching the element in the HTML code with the image unit in the page image, the training sample also correspondingly contains or is provided with the page image after the image unit is replaced (the image unit in the page image is replaced by another image unit), for example, the image unit in the page image is replaced at a rate of 50%. The matching task refers to a task of identifying whether the HTML code matches the page image with the replaced image unit by the to-be-trained model, as shown in pre-training task 2 in FIG. 6, and the parameters of the to-be-trained model are adjusted according to the task execution result of the matching task, so that the to-be-trained model can learn the matching relationship between the HTML code and the page image through the matching task. Figure 6

[0071] ​​In a case where the pre-training task comprises an image region prediction task of predicting a position region of a mask image unit of a page image in the page image, the training sample also comprises or is provided with a page image in which the positions of the image units are disturbed, for example, 15% of the image units are randomly selected, and the image two-dimensional coordinates and numbers of the image units are masked, and the region prediction task refers to predicting the original positions of the disturbed image units by the to-be-trained model, as shown in the pre-training task 3, and the parameters of the to-be-trained model are adjusted according to the task execution result of the region prediction task, so that the to-be-trained model can learn the relationship between local image information in the page image through the region prediction task. Figure 6

[0072] In actual application, after the pre-training of the to-be-trained model is completed, the to-be-trained model can also be supervisedly trained. In an optional implementation provided in the embodiment, after the pre-training is completed, a pre-labeled training sample is used to train the pre-trained model obtained after the pre-training of the model, to obtain an HTML page processing model.

[0073] In order to improve the usability of the HTML page processing model obtained through training in an actual HTML page processing scene, a page processing layer can also be configured in the HTML page processing model obtained through training. The output of the encoder is input into the page processing layer for corresponding HTML page processing. The type of HTML page processing implemented by the page processing layer can be configured according to the needs of the actual HTML page processing scene. For example, Figure 7 As shown in the HTML page processing model configured with the page processing layer, the page processing layer inputs the encoding vector output by the encoder into the configured page classifier for page classification processing, and outputs the page classification result; or the page processing layer inputs the encoding vector output by the encoder into the configured page text extraction module for page keyword extraction, and outputs the page keyword; or the page processing layer inputs the encoding vector output by the encoder into the configured page block recognition module for page block recognition and block information extraction of the page block, and outputs the block information.

[0074] An HTML page processing method provided in the specification is implemented as follows:

[0075] Referring to Figure 8 The HTML page processing method provided in the embodiment specifically comprises steps S802 to S804.

[0076] In step S802, the HTML code to be processed is pre-processed to obtain HTML code features.

[0077] ​The HTML code described in the embodiment refers to the HTML code of an HTML page implemented by using HTML. It should be noted that the HTML page includes an access page of a subprogram or an access page of an application implemented by using HTML and an application technology, and also includes a Web page implemented by using HTML only. If the HTML page is an access page of a subprogram, the HTML code is the HTML code of the access page (a subprogram page) of the subprogram.

[0078] The subprogram includes a program function module or an application component carried on a host application or a program function module or an application component installed and loaded by the host application, such as a small program in an application program. From the perspective of service, the subprogram has the capability of independently providing a self-closed loop service, such as a small program with the self-closed loop capability of performing corresponding service processing running in the application program.

[0079] In a specific implementation, in the process of preprocessing the HTML code, a DOM tree is constructed according to a hierarchical relationship between elements of an HTML page included in the HTML code, starting from the elements of the HTML page included in the HTML code, and a node feature of a node included in the DOM tree is taken as part of an input of HTML page processing. In practice, because the HTML code includes elements that can be as many as dozens or even hundreds, in order to reduce the difficulty of understanding the HTML page in the process of HTML page processing and improve the efficiency of HTML page processing, in an optional implementation provided in the embodiment, the HTML code to be processed is preprocessed to obtain an HTML code feature, including:

[0080] (1) A DOM tree is constructed based on the HTML code, and pruning processing is performed on the DOM tree to obtain a pruned DOM tree.

[0081] It should be noted that the elements included in the HTML code are in one-to-one correspondence with the nodes included in the DOM tree, and the hierarchical relationship of the nodes included in the DOM tree is in one-to-one mapping with the hierarchical relationship of the elements included in the HTML code. For example, a div element included in the HTML code is hierarchically adjacent to an image unit and is in an upper layer of the image unit, and a node corresponding to the div element in the DOM tree is hierarchically adjacent to a node corresponding to the image unit, and the node corresponding to the div element is also in an upper layer of the node corresponding to the image unit.

[0082] Here, the pruning of the DOM tree refers to retaining nodes corresponding to key elements in the DOM tree and deleting nodes corresponding to non-key elements in the DOM tree. The key element refers to an element in the HTML code that carries key information, and the non-key element refers to an element that does not carry key information, wherein the key information refers to information that is not related to the type of the HTML page or service data. The pruned DOM tree refers to the DOM tree after the nodes corresponding to the non-key elements are deleted.

[0083] In an optional implementation provided by the embodiment, the DOM tree is pruned in the following manner: for any node included in the DOM tree, it is detected whether the element corresponding to the node is a key element; if yes, no processing is performed; and if no, the node is deleted from the nodes included in the DOM tree.

[0084] (2) The nodes included in the pruned DOM tree are classified to obtain a classified DOM tree, and the node features of the classified nodes included in the classified DOM tree are determined as the HTML code features.

[0085] To further reduce the difficulty of understanding the HTML page and improve the processing efficiency of the HTML page, in an optional implementation provided by the embodiment, the nodes included in the pruned DOM tree obtained through pruning are classified, so that the categories of the nodes included in the pruned DOM tree can be reduced on the premise of retaining the element information corresponding to the nodes. In this way, the difficulty of understanding the HTML page is reduced by reducing the categories of the nodes. Specifically, the classification of the nodes included in the pruned DOM tree to obtain a classified DOM tree includes:

[0086] According to the element features of the elements corresponding to the nodes included in the pruned DOM tree, the classified categories of the nodes are determined;

[0087] Based on the classified categories of the nodes, the node identifiers of the nodes in the pruned DOM tree are updated to obtain the classified DOM tree.

[0088] Based on this, the node features of the classified nodes included in the classified DOM tree are determined, and the obtained node features are input into an HTML page processing model for page feature processing as the HTML code features. To structure the input node features and thus standardize the input of the HTML page processing model, the embodiment structures the node features. Specifically, in the process of extracting the node features of the classified nodes included in the classified DOM tree from the HTML code, the types of the classified nodes are distinguished, and the following optional implementations are respectively adopted for the non-text nodes and the text nodes for structured processing.

[0089] In an optional implementation of the embodiment, the extracting the node features of the classified nodes contained in the classified DOM tree from the HTML code comprises: if the classified node is a non-text node, obtaining a node number of the non-text node by performing depth-first traversal on the classified DOM tree; determining image coordinate information of an element corresponding to the non-text node in the page image, and obtaining a parent node number of a parent node of the non-text node; and taking the node number, the image coordinate information, and the parent node number as the node features of the non-text node.

[0090] Further, if the classified node is a text node, the following operations are performed: obtaining a node number of the text node by performing depth-first traversal on the classified DOM tree; determining image coordinate information of the text node in the page image, and obtaining a parent node number of a parent node of the text node; splitting text information of an element corresponding to the text node into text tokens and assigning token numbers to the text tokens, and obtaining text features of the element; and taking the node number, the image coordinate information, the parent node number, the token numbers, and the text features as the node features of the text node.

[0091] In step S804, the HTML code features and the page image to be processed are input into an HTML page processing model for page feature processing, and a page processing result is output.

[0092] In the embodiment, the HTML page processing model comprises an embedding layer, an encoder, and a page processing layer, as shown in FIG. 8. Figure 7 The embedding layer performs feature conversion on the HTML code features and the page image, and outputs code feature vectors and image feature vectors; the encoder performs fusion calculation on the code feature vectors and the image feature vectors, and outputs an encoding vector; and the page processing layer performs page processing based on the encoding vector, and outputs the page processing result.

[0093] The feature conversion refers to vector conversion of the HTML code features and the page image in the embedding layer of the HTML page processing model, and the conversion is expressed in a vectorized manner. Optionally, the embedding layer comprises a code feature encoder and an image feature encoder; the feature conversion of the HTML code features and the page image comprises: inputting the HTML code features into the code feature encoder for code feature encoding processing to obtain the code feature vectors, and inputting the page image into the image feature encoder for image encoding processing to obtain the image feature vectors, and taking the code feature vectors and the image feature vectors as the encoding vector.

[0094] In addition, the embedding layer can further include an adaptive pool. After the page image is input into the image feature encoder for image encoding processing to obtain an encoding result, the adaptive pool is used to convert the encoding result into an image feature vector. Alternatively, the embedding layer can split the input page image into a plurality of image units. After the plurality of image units are input into the image feature encoder for image encoding processing to obtain an encoding result, the adaptive pool is used to convert the encoding result into an image feature vector. Alternatively, before the embedding layer is input, the page image is split into a plurality of image units. After the plurality of image units are input into the image feature encoder for image encoding processing to obtain an encoding result, the adaptive pool is used to convert the encoding result into an image feature vector.

[0095] The fusion calculation refers to multi-modal fusion calculation, and specifically refers to feature fusion calculation on the code feature vector of the code modality and the image feature vector of the image modality. In an optional implementation provided by the embodiment, the code feature vector and the image feature vector are fused and calculated, and the fusion calculation is specifically implemented in the following manner: self-attention calculation is performed on the code feature vector and the image feature vector, and a self-attention vector obtained is used as the encoding vector. In the specific execution process, the encoder can perform calculation by using a self-attention algorithm in the process of performing self-attention calculation on the code feature vector and the image feature vector. The encoder can use an encoder under a Transformer architecture.

[0096] In an optional implementation provided by the embodiment, the page processing layer is configured with a page classifier. Correspondingly, the page processing based on the encoding vector includes: inputting the encoding vector into the page classifier for page classification processing, and using an output page classification result as the page processing result.

[0097] Specifically, the page classifier can be implemented by using a multi-classification algorithm, and the page classification result can be specifically classified into eight kinds of page classification results: page normal, loading exception, page without content, rendering tool exception, login authorization exception, page error, service unavailable, and other.

[0098] In a second optional implementation provided by the embodiment, the page processing layer is configured with a page text extraction module. Correspondingly, the page processing based on the encoding vector includes: inputting the encoding vector into the page text extraction module for page keyword extraction, and using an output page keyword as the page processing result.

[0099] In the process of extracting the page keywords, the page text extraction module can calculate the probability of each character being the first keyword of the page keyword and the probability of each character being the last keyword of the page keyword respectively, take the character with the highest probability of being the first keyword as the first keyword, take the character with the highest probability of being the last keyword as the last keyword, and take the first keyword, the last keyword and the character therebetween as the page keyword. In the process of calculating the probability of each character being the last keyword of the page keyword, the double-pointer algorithm can be used for calculation.

[0100] In the third optional implementation provided by the embodiment, the page processing layer is configured with a page block identification module. Correspondingly, the page processing based on the encoding vector includes: inputting the encoding vector into the page block identification module to perform page block identification and block information extraction of the page block, and taking the output block information as the page processing result.

[0101] In the process of performing page block identification and block information extraction of the page block, a binary classification algorithm can be used to determine the binary classification result of whether to perform information extraction on each page block. If the binary classification result of the page block is 1, the block information of the page block is extracted. If the binary classification result of the page block is 0, the block information of the page block is not extracted. The block information output by the page block identification module is the block information of all page blocks with a binary classification result of 1.

[0102] The model pre-training device provided in the embodiment is as follows:

[0103] In the above embodiment, a model pre-training method is provided, and a model pre-training device corresponding to the method is also provided. The following description is made with reference to the accompanying drawings.

[0104] Reference is made to Figure 9 which shows a model pre-training device provided in the embodiment.

[0105] Since the device embodiment corresponds to the method embodiment, the description is relatively simple, and the related parts can be seen in the above-mentioned corresponding description of the method embodiment. The device embodiment described below is only illustrative.

[0106] The model pre-training device provided in the embodiment includes:

[0107] The preprocessing module 902 is configured to pre-process the HTML code in the training sample to obtain HTML code features.

[0108] The feature processing module 904 is configured to input the HTML code feature and the page image in the training sample into a to-be-trained model for feature processing, and output a feature processing result; the HTML code and the page image correspond to a same HTML page; the to-be-trained model comprises an embedding layer and an encoder; the embedding layer performs feature conversion on the HTML code feature and the page image, and outputs a code feature vector and an image feature vector; and the encoder performs fusion calculation on the code feature vector and the image feature vector, and outputs the feature processing result.

[0109] The pre-training task execution module 906 is configured to execute at least one pre-training task of pre-training the to-be-trained model based on the feature processing result.

[0110] The HTML page processing apparatus provided in the specification implements, for example, the following.

[0111] In the above embodiment, an HTML page processing method is provided, and a corresponding HTML page processing apparatus is also provided, which will be described below with reference to the accompanying drawings.

[0112] Reference Figure 10 which shows a schematic diagram of an HTML page processing apparatus provided in the embodiment.

[0113] Since the apparatus embodiment corresponds to the method embodiment, the description is relatively simple, and the related parts can be seen in the above-provided corresponding description of the method embodiment. The apparatus embodiment described below is only schematic.

[0114] The HTML page processing apparatus provided in the embodiment comprises:

[0115] The preprocessing module 1002 is configured to pre-process HTML code to be processed, and obtain HTML code features.

[0116] The page feature processing module 1004 is configured to input the HTML code features and a page image to be processed into an HTML page processing model for page feature processing, and output a page processing result.

[0117] The HTML page processing model comprises an embedding layer, an encoder, and a page processing layer; the embedding layer performs feature conversion on the HTML code features and the page image, and outputs a code feature vector and an image feature vector; the encoder performs fusion calculation on the code feature vector and the image feature vector, and outputs an encoding vector; and the page processing layer performs page processing based on the encoding vector, and outputs the page processing result.

[0118] The model pre-training device provided in the specification implements the following, for example:

[0119] Based on the same technical concept, the model pre-training method described above, one or more embodiments of the specification also provide a model pre-training device for executing the model pre-training method provided above, Figure 11 The structural diagram of a model pre-training device provided by one or more embodiments of the specification.

[0120] The model pre-training device provided in the embodiment includes:

[0121] As shown in Figure 11 The model pre-training device can have large differences due to different configurations or performance, and can include one or more processors 1101 and memories 1102, and one or more storage applications or data can be stored in the memories 1102. Among them, the memory 1102 can be temporary storage or persistent storage. The application stored in the memory 1102 can include one or more modules (not shown in the figure), and each module can include a series of computer executable instructions in the model pre-training device. Further, the processor 1101 can be configured to communicate with the memory 1102, and execute a series of computer executable instructions in the memory 1102 on the model pre-training device. The model pre-training device can also include one or more power supplies 1103, one or more wired or wireless network interfaces 1104, one or more input / output interfaces 1105, one or more keyboards 1106, etc.

[0122] In one specific embodiment, the model pre-training device includes a memory and one or more programs, wherein one or more programs are stored in the memory, and one or more programs can include one or more modules, and each module can include a series of computer executable instructions in the model pre-training device, and the one or more processors are configured to execute the one or more programs include computer executable instructions for:

[0123] Preprocessing the HTML code in the training sample to obtain HTML code features;

[0124] The HTML code features and the page image in the training sample are input into a to-be-trained model for feature processing, and a feature processing result is output; the HTML code and the page image correspond to a same HTML page; the to-be-trained model comprises an embedding layer and an encoder; the embedding layer performs feature conversion on the HTML code features and the page image, and outputs code feature vectors and image feature vectors; and the encoder performs fusion calculation on the code feature vectors and the image feature vectors, and outputs the feature processing result.

[0125] At least one pre-training task of pre-training the to-be-trained model is performed based on the feature processing result.

[0126] The present specification provides an HTML page processing device, which implements, for example, the following:

[0127] Based on the same technical concept, the present specification one or more embodiments also provide an HTML page processing device for executing the model pre-training method provided above, Figure 12 A structural schematic diagram of an HTML page processing device provided by the present specification one or more embodiments.

[0128] The HTML page processing device provided by the present embodiment comprises:

[0129] As shown in Figure 12 The HTML page processing device can have great differences due to different configurations or performances, and can comprise one or more processors 1201 and memories 1202. The memories 1202 can store one or more storage applications or data. The memory 1202 can be temporary storage or persistent storage. The applications stored in the memory 1202 can comprise one or more modules (not shown in the figure), and each module can comprise a series of computer executable instructions in the HTML page processing device. Furthermore, the processor 1201 can be configured to communicate with the memory 1202 and execute a series of computer executable instructions in the memory 1202 on the HTML page processing device. The HTML page processing device can further comprise one or more power supplies 1203, one or more wired or wireless network interfaces 1204, one or more input / output interfaces 1205, one or more keyboards 1206, and the like.

[0130] In one specific embodiment, the HTML page processing device includes a memory, and one or more programs, wherein one or more programs are stored in the memory, and one or more programs can include one or more modules, and each module can include a series of computer executable instructions in the HTML page processing device, and the one or more programs configured to be executed by one or more processors include computer executable instructions for:

[0131] preprocessing the HTML code to be processed to obtain HTML code features;

[0132] inputting the HTML code features and the page image to be processed into an HTML page processing model for page feature processing, and outputting a page processing result;

[0133] The HTML page processing model includes an embedding layer, an encoder, and a page processing layer; the embedding layer performs feature conversion on the HTML code features and the page image to output code feature vectors and image feature vectors; the encoder performs fusion calculation on the code feature vectors and the image feature vectors to output an encoding vector; and the page processing layer performs page processing based on the encoding vector to output the page processing result.

[0134] The storage medium provided in the specification is implemented as follows:

[0135] According to the above description, a model pre-training method is provided based on the same technical concept, and one or more embodiments of the specification also provide a storage medium.

[0136] The storage medium provided in the embodiment is used to store computer executable instructions, and the computer executable instructions are executed by a processor to implement the following processes:

[0137] preprocessing the HTML code in the training sample to obtain HTML code features;

[0138] inputting the HTML code features and the page image in the training sample into a to-be-trained model for feature processing, and outputting a feature processing result; the HTML code and the page image correspond to the same HTML page; the to-be-trained model includes an embedding layer and an encoder, the embedding layer performs feature conversion on the HTML code features and the page image to output code feature vectors and image feature vectors, and the encoder performs fusion calculation on the code feature vectors and the image feature vectors to output the feature processing result;

[0139] Based on the feature processing result, at least one pre-training task of pre-training the to-be-trained model is performed.

[0140] It should be noted that the embodiment of the storage medium in the specification is based on the same inventive concept as the embodiment of the model pre-training method in the specification, and therefore the specific implementation of this embodiment can refer to the implementation of the corresponding method described above, and the repeated parts will not be described herein.

[0141] Another storage medium embodiment provided by the specification is as follows:

[0142] Based on the same technical concept, the storage medium is provided by one or more embodiments of the specification according to the HTML page processing method described above.

[0143] The storage medium provided by the embodiment is used to store computer executable instructions, and the computer executable instructions realize the following process when executed by a processor.

[0144] The pre-processing is performed on the HTML code to be processed to obtain HTML code features;

[0145] The HTML code features and the page image to be processed are input into an HTML page processing model for page feature processing, and a page processing result is output;

[0146] The HTML page processing model includes an embedding layer, an encoder, and a page processing layer; the embedding layer performs feature conversion on the HTML code features and the page image to output code feature vectors and image feature vectors; the encoder performs fusion calculation on the code feature vectors and the image feature vectors to output an encoding vector; and the page processing layer performs page processing based on the encoding vector to output the page processing result.

[0147] It should be noted that the embodiment of the storage medium in the specification is based on the same inventive concept as the embodiment of the HTML page processing method in the specification, and therefore the specific implementation of this embodiment can refer to the implementation of the corresponding method described above, and the repeated parts will not be described herein.

[0148] The above describes specific embodiments of the specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims can be performed in an order other than that described in the embodiments and still achieve desirable results. In addition, the processes depicted in the figures do not necessarily require the particular order shown or sequential order to achieve the desired results. In certain implementations, multitasking and parallel processing can be advantageous or possible.

[0149] In the 1930s, it was clear to distinguish whether an improvement in a technology was in hardware (e.g., improvement in circuit structure of diodes, transistors, switches, etc.) or in software (e.g., improvement in method flow). However, as technology has evolved, many improvements in method flow today can be considered as direct improvements in hardware circuit structure. Designers almost always obtain the corresponding hardware circuit structure by programming the improved method flow into the hardware circuit. Therefore, it cannot be said that an improvement in a method flow cannot be implemented by a hardware entity module. For example, a programmable logic device (PLD) (e.g., a field programmable gate array (FPGA)) is an integrated circuit whose logic function is determined by user programming of the device. A digital system is "integrated" on a PLD by the designer programming it, rather than by asking a chip manufacturer to design and fabricate a custom integrated circuit chip. Moreover, instead of manually fabricating an integrated circuit chip, this programming is now mostly implemented by "logic compiler" software, which is similar to software compilers used in program development, and the original code to be compiled is written in a specific programming language, called a hardware description language (HDL), of which there are many, such as ABEL (Advanced Boolean Expression Language), AHDL (Altera Hardware Description Language), Confluence, CUPL (Cornell University Programming Language), HDCal, JHDL (Java Hardware Description Language), Lava, Lola, MyHDL, PALASM, RHDL (Ruby Hardware Description Language), etc., the most commonly used being VHDL (Very-High-Speed Integrated Circuit Hardware Description Language) and Verilog. It should be clear to those skilled in the art that, by simply logically programming a method flow in one of the above hardware description languages and programming it into an integrated circuit, a hardware circuit implementing the logical method flow can be easily obtained.

[0150] The controller can be implemented in any suitable way, e.g. the controller can take the form of a microprocessor or processor and a computer readable medium storing computer readable program code, e.g. software or firmware, executable by the (micro)processor, logic gates, switches, an application specific integrated circuit (ASIC), a programmable logic controller and an embedded microcontroller, examples of controllers include but are not limited to the following microcontrollers: ARC 625D, Atmel AT91 SAM, Microchip PIC18F26K20 and Silicone Labs C8051F320, the memory controller can also be implemented as part of the control logic of the memory. Those skilled in the art will also know that in addition to being implemented in pure computer readable program code form, the controller can perfectly well be implemented by means of logic programmed into logic gates, switches, application specific integrated circuits, programmable logic controllers and embedded microcontrollers, etc. to perform the same functions. The controller can thus be considered as a hardware component, and the means comprised therein for performing various functions can be considered as structures within the hardware component. Alternatively, or even, the means for performing various functions can be considered as both a software module implementing a method and a structure within a hardware component.

[0151] The systems, apparatuses, modules or units illustrated by the above embodiments can be implemented by computer chips or entities, or products with certain functions. A typical implementation device is a computer. Specifically, the computer can be a personal computer, a laptop computer, a cellular phone, a camera phone, a smart phone, a personal digital assistant, a media player, a navigation device, an email device, a game console, a tablet computer, a wearable device, or a combination of any of these devices.

[0152] For the sake of description, the above apparatuses are described in various units with functions respectively. Of course, the functions of the units can be implemented in one or more software and / or hardware in the implementation of the embodiments of the present specification.

[0153] Those skilled in the art will appreciate that one or more embodiments of the present specification can be provided as a method, a system or a computer program product. Therefore, one or more embodiments of the present specification can take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware aspects. Moreover, the present specification can take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROMs, optical storage devices, etc.) containing computer usable program code.

[0154] The specification is presented with reference to flow diagrams and / or block diagrams of methods, apparatus (systems) and computer program products according to embodiments of the specification. It will be understood that each block of the flow diagrams and / or block diagrams, and combinations of blocks in the flow diagrams and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general purpose computer, special purpose computer, embedded processing element or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in the flow diagrams and / or block diagrams block or blocks. Figure 1 The flow diagrams and / or block diagrams in the specification can present a method, apparatus or computer program product according to embodiments of the specification. Flow diagrams and / or block diagrams can also present a method, apparatus or computer program product to achieve functions specified in flow diagrams and / or block diagrams block or blocks. Figure 1 The flow diagrams and / or block diagrams in the specification can present a method, apparatus or computer program product according to embodiments of the specification. Flow diagrams and / or block diagrams can also present a method, apparatus or computer program product to achieve functions specified in flow diagrams and / or block diagrams block or blocks.

[0155] These computer program instructions can also be stored in a computer-readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instructions which implement the flow diagrams and / or block diagrams block or blocks. Figure 1 The flow diagrams and / or block diagrams in the specification can present a method, apparatus or computer program product according to embodiments of the specification. Flow diagrams and / or block diagrams can also present a method, apparatus or computer program product to achieve functions specified in flow diagrams and / or block diagrams block or blocks. Figure 1 The flow diagrams and / or block diagrams in the specification can present a method, apparatus or computer program product according to embodiments of the specification. Flow diagrams and / or block diagrams can also present a method, apparatus or computer program product to achieve functions specified in flow diagrams and / or block diagrams block or blocks.

[0156] These computer program instructions can also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the flow diagrams and / or block diagrams block or blocks. Figure 1 The flow diagrams and / or block diagrams in the specification can present a method, apparatus or computer program product according to embodiments of the specification. Flow diagrams and / or block diagrams can also present a method, apparatus or computer program product to achieve functions specified in flow diagrams and / or block diagrams block or blocks. Figure 1 The flow diagrams and / or block diagrams in the specification can present a method, apparatus or computer program product according to embodiments of the specification. Flow diagrams and / or block diagrams can also present a method, apparatus or computer program product to achieve functions specified in flow diagrams and / or block diagrams block or blocks.

[0157] In a typical configuration, a computing device includes one or more processors (CPUs), input / output interfaces, network interfaces, and memory.

[0158] The memory can include non-persistent memory and / or storage mechanisms such as, for example, random access memory (RAM), non-volatile memory (NVM), and / or a persistent memory such as, for example, read-only memory (ROM) or flash memory. The memory is an example of computer-readable media.

[0159] Computer-readable media includes permanent and non-permanent, movable and non-movable media that can be implemented by any method or technology to store information. The information can be computer-readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic cassette, magnetic disk storage or other magnetic storage devices, or any other non-transmission medium that can be used to store information accessible to a computing device. According to the definition herein, computer-readable media does not include transitory media such as modulated data signals and carriers.

[0160] It should also be noted that the terms "comprising", "containing", or any other variant thereof are intended to cover non-exclusive inclusion, such that processes, methods, articles or devices that include a series of elements not only include those elements, but also include other elements not explicitly listed or inherent to such processes, methods, articles or devices. Without more limitations, the element defined by the statement "comprising a" does not exclude the presence of additional identical elements in the process, method, article or device that includes the element.

[0161] One or more embodiments of the present specification can be described in the general context of computer-executable instructions being executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, and the like that perform particular tasks or implement particular abstract data types. One or more embodiments of the present specification can also be practiced in a distributed computing environment, in which tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules can be located in local and remote computer storage media, including storage devices.

[0162] Each embodiment in the present specification is described in a progressive manner, and the same or similar parts between each embodiment can be referred to each other, and each embodiment focuses on the difference from other embodiments. In particular, for system embodiments, since they are basically similar to method embodiments, the description is relatively simple, and the relevant parts can be referred to the part of the method embodiment.

[0163] The above merely provides the example of the present document and is not intended to limit the present document. For those skilled in the art, the present document can have various modifications and changes. Any modification, equivalent replacement, improvement, etc. within the spirit and principle of the present document shall be included in the scope of claims of the present document.

Claims

1. A model pre-training method, comprising: constructing a DOM tree based on HTML code in a training sample and performing pruning processing to obtain a pruned DOM tree; performing classification processing on nodes included in the pruned DOM tree, and determining node features of classified nodes included in a classified DOM tree as HTML code features; the node features of non-text nodes in the classified nodes include: node numbers of the non-text nodes, image coordinate information of elements corresponding to the non-text nodes in a page image, and parent node numbers of parent nodes; the node features of text nodes in the classified nodes include: node numbers of the text nodes, image coordinate information of the text nodes in the page image, parent node numbers of parent nodes, token numbers of the text nodes, and text features of elements corresponding to the text nodes; inputting the HTML code features and the page image in the training sample into a to-be-trained model to perform feature processing, and outputting a feature processing result; the HTML code and the page image correspond to a same HTML page; the page image is obtained based on rendering of the HTML code; performing a plurality of pre-training tasks of pre-training the to-be-trained model based on the feature processing result, and adjusting parameters of embedding layers and encoders included in the to-be-trained model according to task execution results of the pre-training tasks; the pre-training tasks include: a semantic recognition task of predicting mask elements in the HTML code, and an image area prediction task of predicting a location area of a mask image unit of the page image in the page image.

2. The model pre-training method of claim 1, wherein the pruning processing comprises: for any node included in the DOM tree, detecting whether an element corresponding to the node is a key element; if not, deleting the node from the nodes included in the DOM tree.

3. The model pre-training method of claim 1, wherein the classification processing on the nodes included in the pruned DOM tree comprises: determining a classification category of a node according to element features of an element corresponding to the node; updating a node identifier of the node in the pruned DOM tree based on the classification category of the node to obtain the classified DOM tree.

4. The model pre-training method of claim 1, wherein the determination of the node features of the classified nodes included in the classified DOM tree comprises: if the classified node is a non-text node, obtaining a node number of the non-text node by performing depth-first traversal on the classified DOM tree; determining image coordinate information of an element corresponding to the non-text node in the page image, and obtaining a parent node number of a parent node of the non-text node; taking the node number, the image coordinate information, and the parent node number as the node features of the non-text node.

5. The model pre-training method of claim 4, if the classified node is a text node, the following operations are performed: obtaining a node number of the text node by performing depth-first traversal on the classified DOM tree; determining image coordinate information of the text node in the page image, and obtaining a parent node number of a parent node of the text node; splitting text information of an element corresponding to the text node into text tokens and assigning token numbers to the text tokens, and obtaining text features of the element; taking the node number, the image coordinate information, the parent node number, the token numbers, and the text features as node features of the text node.

6. The model pre-training method of claim 1, wherein the embedding layer comprises a code feature encoder and an image feature encoder; the embedding layer performs feature conversion on the HTML code features and the page image, comprising: inputting the HTML code features into the code feature encoder for code feature encoding processing to obtain code feature vectors, and inputting the page image into the image feature encoder for image encoding processing to obtain image feature vectors.

7. The model pre-training method of claim 6, further comprising: the encoder performs fusion calculation on the code feature vectors and the image feature vectors to output a feature processing result; wherein the fusion calculation on the code feature vectors and the image feature vectors comprises: performing self-attention calculation on the code feature vectors and the image feature vectors, and taking the obtained self-attention vectors as the feature processing result.

8. The model pre-training method of claim 1, further comprising: adjusting parameters of the embedding layer and the encoder according to a task execution result of the pre-training task; wherein the pre-training task comprises at least one of the following: a semantic recognition task of predicting a mask element in HTML code, a matching task of matching an element contained in HTML code with an image unit in the page image, and an image region prediction task of predicting a location region of a mask image unit in the page image.

9. The model pre-training method of claim 1, further comprising: using pre-labeled training samples to perform model training on a pre-training model obtained after model pre-training to obtain an HTML page processing model.

10. An HTML page processing method, comprising: constructing a DOM tree based on HTML code to be processed and performing pruning processing to obtain a pruned DOM tree; performing classification processing on nodes contained in the pruned DOM tree, and determining node features of classified nodes contained in a classified DOM tree as HTML code features; inputting the HTML code features and a page image to be processed into an HTML page processing model for page feature processing to output a page processing result; the HTML code and the page image correspond to the same HTML page; and the page image is obtained based on the HTML code. The embedding layer of the HTML page processing model converts features of the HTML code and the page image, and outputs a code feature vector and an image feature vector; the encoder of the HTML page processing model performs fusion calculation on the code feature vector and the image feature vector, and outputs an encoding vector; the page processing layer of the HTML page processing model is configured with a page block recognition module; the page processing layer inputs the encoding vector into the page block recognition module for page block recognition and block information extraction of the page block, and outputs the block information as the page processing result.

11. The HTML page processing method of claim 10, wherein the page processing layer is configured with a page classifier; and the page processing layer performs page processing based on the encoding vector to output the page processing result. Correspondingly, the page processing based on the encoding vector comprises: inputting the encoding vector into the page classifier for page classification processing, and outputting a page classification result as the page processing result.

12. The HTML page processing method of claim 10, the page processing layer is configured with a page text extraction module. The page processing layer performs page processing based on the encoding vector to output the page processing result. Correspondingly, the page processing based on the encoding vector comprises: inputting the encoding vector into the page text extraction module for page keyword extraction, and outputting a page keyword as the page processing result.

13. The HTML page processing method of claim 10, wherein the embedding layer comprises a code feature encoder and an image feature encoder. The feature conversion of the HTML code feature and the page image comprises: inputting the HTML code feature into the code feature encoder for code feature encoding processing to obtain the code feature vector, and inputting the page image into the image feature encoder for image encoding processing to obtain the image feature vector, and taking the code feature vector and the image feature vector as the encoding vector.

14. The HTML page processing method of claim 10, wherein the fusion calculation of the code feature vector and the image feature vector comprises: performing self-attention calculation on the code feature vector and the image feature vector, and taking a self-attention vector obtained as the encoding vector.

15. A model pre-training device, comprising: a preprocessing module configured to construct a DOM tree based on HTML code in a training sample and perform pruning processing to obtain a pruned DOM tree; performing classification processing on nodes contained in the pruned DOM tree, and determining node features of classified nodes contained in a classified DOM tree as HTML code features; The node features of the non-text nodes in the classified nodes include: node numbers of the non-text nodes, image coordinate information of elements corresponding to the non-text nodes in a page image, and parent node numbers of parent nodes; and the node features of the text nodes in the classified nodes include: node numbers of the text nodes, image coordinate information of the text nodes in the page image, parent node numbers of parent nodes, token numbers of the text nodes, and text features of elements corresponding to the text nodes. The feature processing module is configured to input the HTML code features and the page image in the training sample into a to-be-trained model for feature processing, and output feature processing results; the HTML code and the page image correspond to a same HTML page; and the page image is obtained based on rendering of the HTML code; The pre-training task execution module is configured to execute a plurality of pre-training tasks of pre-training the to-be-trained model based on the feature processing results, and adjust parameters of embedding layers and encoders included in the to-be-trained model according to task execution results of the pre-training tasks; the pre-training tasks include: a semantic recognition task of predicting mask elements in the HTML code, and an image area prediction task of predicting a location area of a mask image unit of the page image in the page image.

16. An HTML page processing apparatus, comprising: a preprocessing module configured to construct a DOM tree based on to-be-processed HTML code and perform pruning processing to obtain a pruned DOM tree; perform classification processing on nodes included in the pruned DOM tree, and determine node features of classified nodes included in a classified DOM tree as HTML code features; a page feature processing module configured to input the HTML code features and a to-be-processed page image into an HTML page processing model for page feature processing, and output page processing results; the HTML code and the page image correspond to a same HTML page; and the page image is obtained based on rendering of the HTML code; wherein an embedding layer of the HTML page processing model performs feature conversion on the HTML code features and the page image, and outputs code feature vectors and image feature vectors; an encoder of the HTML page processing model performs fusion calculation on the code feature vectors and the image feature vectors, and outputs an encoding vector; a page processing layer of the HTML page processing model is configured with a page block recognition module; and the page processing layer inputs the encoding vector into the page block recognition module for page block recognition and block information extraction of a page block, and outputs the block information as the page processing results.

17. A model pre-training device, comprising: a processor; and a memory configured to store computer executable instructions that, when executed, cause the processor to: construct a DOM tree based on HTML code in a training sample and perform pruning processing to obtain a pruned DOM tree; The nodes included in the pruned DOM tree are classified, and node features of the classified nodes included in the classified DOM tree are determined as HTML code features; The node features of the non-text nodes in the classified nodes include: node numbers of the non-text nodes, image coordinate information of elements corresponding to the non-text nodes in a page image, and parent node numbers of parent nodes; and the node features of the text nodes in the classified nodes include: node numbers of the text nodes, image coordinate information of the text nodes in the page image, parent node numbers of parent nodes, token numbers of the text nodes, and text features of elements corresponding to the text nodes; The HTML code features and the page image in the training sample are input into a to-be-trained model for feature processing, and a feature processing result is output; the HTML code and the page image correspond to a same HTML page; and the page image is obtained based on rendering of the HTML code; Based on the feature processing result, a plurality of pre-training tasks of pre-training the to-be-trained model are executed, and an embedding layer and an encoder included in the to-be-trained model are parameter adjusted according to a task execution result of the pre-training tasks; the pre-training tasks include: a semantic recognition task of predicting a mask element in the HTML code, and an image area prediction task of predicting a location area of a mask image unit of the page image in the page image.

18. An HTML page processing device, comprising: a processor; and a memory configured to store computer executable instructions which, when executed, cause the processor to: construct a DOM tree based on to-be-processed HTML code and perform pruning processing to obtain a pruned DOM tree; classify nodes included in the pruned DOM tree, and determine node features of the classified nodes included in the classified DOM tree as HTML code features; input the HTML code features and a to-be-processed page image into an HTML page processing model for page feature processing, and output a page processing result; the HTML code and the page image correspond to a same HTML page; and the page image is obtained based on rendering of the HTML code; wherein an embedding layer of the HTML page processing model performs feature conversion on the HTML code features and the page image, and outputs a code feature vector and an image feature vector; an encoder of the HTML page processing model performs fusion calculation on the code feature vector and the image feature vector, and outputs an encoding vector; a page processing layer of the HTML page processing model is configured with a page block recognition module; and the page processing layer inputs the encoding vector into the page block recognition module for page block recognition and block information extraction of a page block, and outputs the block information as the page processing result.

19. A storage medium for storing computer executable instructions which, when executed by a processor, implement the following processes: The DOM tree is constructed based on the HTML code in the training sample and pruning processing is performed to obtain a pruned DOM tree; The nodes contained in the pruned DOM tree are subjected to a categorization process, and the node features of the categorized nodes contained in the categorized DOM tree are determined as HTML code features; The node features of the non-text nodes in the classified nodes include: node numbers of the non-text nodes, image coordinate information of elements corresponding to the non-text nodes in a page image, and parent node numbers of parent nodes; and the node features of the text nodes in the classified nodes include: node numbers of the text nodes, image coordinate information of the text nodes in the page image, parent node numbers of parent nodes, token numbers of the text nodes, and text features of elements corresponding to the text nodes. The HTML code features and the page image in the training sample are input into a to-be-trained model for feature processing, and a feature processing result is output; the HTML code and the page image correspond to a same HTML page; and the page image is obtained based on rendering of the HTML code. Based on the feature processing result, a plurality of pre-training tasks of pre-training the to-be-trained model are executed, and parameters of embedding layers and encoders included in the to-be-trained model are adjusted according to task execution results of the pre-training tasks; the pre-training tasks include: a semantic recognition task of predicting mask elements in the HTML code, and an image area prediction task of predicting a location area of a mask image unit of the page image in the page image.

20. A storage medium for storing computer executable instructions, which, when executed by a processor, implement the following processes: A DOM tree is constructed based on to-be-processed HTML code and pruning processing is performed to obtain a pruned DOM tree; Nodes included in the pruned DOM tree are classified, and node features of classified nodes included in a classified DOM tree are determined as HTML code features; The HTML code features and a to-be-processed page image are input into an HTML page processing model for page feature processing, and a page processing result is output; the HTML code and the page image correspond to a same HTML page; and the page image is obtained based on rendering of the HTML code; wherein An embedding layer of the HTML page processing model performs feature conversion on the HTML code features and the page image, and outputs a code feature vector and an image feature vector; an encoder of the HTML page processing model performs fusion calculation on the code feature vector and the image feature vector, and outputs an encoding vector; a page processing layer of the HTML page processing model is configured with a page block recognition module; and the page processing layer inputs the encoding vector into the page block recognition module for page block recognition and block information extraction of a page block, and outputs the block information as the page processing result.

Images