Training of category prediction model, method and device for identifying interface element categories

Abstract

The embodiments of this specification provide a method and device for training a category prediction model and identifying the category of an interface element. In the training method, a sample image is obtained, and the sample image contains a number of interface elements. The text content contained in each interface element is obtained. The sample image and each text content are input into the category prediction model for prediction processing. The prediction processing includes: using a feature extractor to extract image features from the sample image. Using a first encoder based on the word vector of each word in each text content, determine the feature vector of each text content. Using a second encoder based on image features and the feature vector of each text content, determine the comprehensive feature representation of each interface element. Using a classifier based on each comprehensive feature representation, determine the first category prediction result of each interface element. Train the category prediction model based on at least each first category prediction result and each category label.

Description

Technical Field

[0001] One or more embodiments of the present specification relate to the field of image recognition technology, and in particular, to a method and device for training a category prediction model and identifying interface element categories. Background Art

[0002] In many application scenarios, in order to provide corresponding services, users are required to fill in forms. For example, when providing users with authentication services such as registration or login, users are required to fill in their identity information in the form; when providing users with related life services, users are required to fill in their address information in the form.

[0003] The more traditional way of filling out forms is for users to fill out forms manually. However, this method not only makes the user operation more complicated, but also prone to errors. Therefore, many websites or applications currently provide the function of automatically filling out form information. In order to use this function, it is necessary to identify the interface elements in the form, for example, to identify the category of the interface elements. However, the traditional method mainly relies on parsing the website source code to parse and identify the content, but due to the large differences in the forms of various websites, there is an urgent need for a more general recognition solution for the category of interface elements that relies on vision. Summary of the invention

[0004] One or more embodiments of the present specification describe a method and device for training a category prediction model and identifying interface element categories, which can simultaneously focus on the image features of an image and the correlation between the text contents of each interface element contained in the image, thereby more accurately identifying the category of the interface element.

[0005] In a first aspect, a training method for a category prediction model is provided, comprising:

[0006] Acquire a sample image, wherein the sample image includes a plurality of interface elements, each of which has a corresponding category label;

[0007] Get the text content contained in each interface element;

[0008] The sample image and the text contents are input into a category prediction model for prediction processing; the category prediction model includes a feature extractor, a first encoder, a second encoder and a classifier; the prediction processing includes: using the feature extractor to extract image features from the sample image; using the first encoder to determine the feature vector of each text content based on the word vector of each word in the text content; using the second encoder to determine the comprehensive feature representation of each interface element based on the image features and the feature vector of each text content; using the classifier to determine the first category prediction result of each interface element based on each comprehensive feature representation;

[0009] The category prediction model is trained based on at least the first category prediction results of each interface element and each category label.

[0010] In a second aspect, a method for identifying a category of an interface element is provided, comprising:

[0011] Acquire a target image, wherein the target image includes a target interface element;

[0012] Obtaining target content contained in the target interface element;

[0013] Obtaining a category prediction model, wherein the category prediction model is trained according to the method described in the first aspect;

[0014] The target image and the target content are input into the category prediction model for prediction processing to obtain the target category of the target interface element.

[0015] In a third aspect, a training device for a category prediction model is provided, comprising:

[0016] An acquisition unit, used for acquiring a sample image, wherein the sample image includes a plurality of interface elements, each of which has a corresponding category label;

[0017] The acquisition unit is further used to acquire each text content contained in each interface element;

[0018] A prediction processing unit is used to input the sample image and the text contents into a category prediction model for prediction processing; the category prediction model includes a feature extractor, a first encoder, a second encoder and a classifier; the prediction processing includes: using the feature extractor to extract image features from the sample image; using the first encoder to determine the feature vector of each text content based on the word vector of each word in the text content; using the second encoder to determine the comprehensive feature representation of each interface element based on the image features and the feature vector of each text content; using the classifier to determine the first category prediction result of each interface element based on each comprehensive feature representation;

[0019] A training unit is used to train the category prediction model based on at least the first category prediction results of each interface element and each category label.

[0020] In a fourth aspect, a device for identifying a category of an interface element is provided, comprising:

[0021] An acquisition unit, used to acquire a target image, wherein the target image includes a target interface element;

[0022] The acquisition unit is further used to acquire the target content contained in the target interface element;

[0023] The acquisition unit is further used to acquire a category prediction model, wherein the category prediction model is trained according to the method described in the first aspect;

[0024] The prediction processing unit is used to input the target image and the target content into the category prediction model for prediction processing to obtain the target category of the target interface element.

[0025] The training method and device of the category prediction model provided in one or more embodiments of this specification can train the category prediction model based on the image features of the sample image and the text content of each interface element contained in the sample image. In other words, the category prediction model can be trained based on a variety of information such as vision and text, which can greatly improve the accuracy of the model. In addition, during the prediction process, the present scheme first uses the first encoder to obtain the feature vector of each text content, and then inputs it into the second encoder, so that the second encoder can focus on the correlation between the text contents when determining the comprehensive feature representation of the interface element, rather than just the correlation between individual words. This can improve the accuracy of the feature representation of the interface element, and thus improve the accuracy of the category prediction results. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] In order to more clearly illustrate the technical solutions of the embodiments of this specification, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this specification. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0027] Figure 1 A schematic diagram of an implementation scenario of an embodiment disclosed in this specification;

[0028] Figure 2 A flow chart of a method for training a category prediction model according to an embodiment is shown;

[0029] Figure 3a A schematic diagram showing a sample image in one embodiment;

[0030] Figure 3b Show Figure 3a The category labels of the input boxes and prompts in ;

[0031] Figure 3c A schematic diagram showing a sample image in another embodiment;

[0032] Figure 4 A schematic diagram showing a training method of a category prediction model in one embodiment;

[0033] Figure 5 A flow chart of a method for identifying an interface element category according to an embodiment is shown;

[0034] Figure 6 A schematic diagram of a training device for a category prediction model according to an embodiment is shown;

[0035] Figure 7 A schematic diagram of a device for identifying interface element categories according to one embodiment is shown. DETAILED DESCRIPTION

[0036] The solution provided in this specification is described below in conjunction with the accompanying drawings.

[0037] Figure 1 The following is a schematic diagram of an implementation scenario of an embodiment disclosed in this specification. The implementation scenario involves the training of a category prediction model. The category prediction model may include a feature extractor, a first encoder, a second encoder, and a classifier.

[0038] Figure 1 In the training process of the category prediction model, a sample image is obtained, and the text content of each interface element in the sample image is obtained. The sample image and the text content are input into the category prediction model as two types of input information. Specifically, the sample image can be input into a feature extractor to extract image features, and each text content can be input into a first encoder to determine the feature vector of each text content. Afterwards, the image features and the feature vectors of each text content can be input into a second encoder together, so that the second encoder determines the comprehensive feature representation of each interface element by combining the correlation between each text content and the correlation between each text content and the image features. Finally, each comprehensive feature representation is input into a classifier to obtain a category prediction result for each interface element. The category prediction model is trained based on the category prediction results and category labels of each interface element.

[0039] Of course, in practical applications, the second encoder can also be used to obtain a comprehensive feature representation of the sample image. After the comprehensive feature representation of the sample image is input into the classifier, the direction prediction result of the sample image can be obtained. Therefore, the category prediction model can be trained based on the prediction error between the category prediction result and the category label, and the prediction error between the direction prediction result and the direction label.

[0040] The following combination Figure 1 Describe this plan in detail.

[0041] Figure 2 FIG. 1 is a flow chart of a method for training a category prediction model according to an embodiment. It is understood that the method can be executed by any device, equipment, platform, or device cluster with computing and processing capabilities. Figure 2As shown, the training process includes at least the following steps.

[0042] Step 202: Obtain a sample image, where the sample image includes a number of interface elements, each of which has a corresponding category label.

[0043] The sample image here may be, for example, a form image (also called a rich text image), which includes several interface elements. The interface element may include at least one of the following: a prompt, an input box (also called an interaction box), a button, and others.

[0044] Among them, for the input box, its corresponding category label may include a main category label and a subcategory label. In one example, the main category label of the input box may be, for example, input box. The subcategory label may include, but is not limited to, account number, password, mobile phone number, name, verification code, etc.

[0045] For other interface elements, the corresponding category labels may include only the main category label, or may include both the main category label and the subcategory label. For example, the category label corresponding to the prompt may include the main category label and the subcategory label.

[0046] Figure 3a A schematic diagram showing a sample image in one embodiment. Figure 3a In the sample image, the interface elements may include: prompts, input boxes, registration buttons and others.

[0047] Figure 3b Show Figure 3a The category labels of the input boxes and prompts in the figure. The main category label is shown in the upper left corner of each input box / prompt, which is represented by "0" and "1" respectively, where "0" represents the prompt and "1" represents the input box. The subcategory labels are shown in the lower right corner, which are represented by: "0", "18", "15", "21", "6", "31" and "30", where "0" represents the account number, "18" represents the password, "15" represents the name, "21" represents the mobile phone number, "6" represents the withdrawal password, "31" represents the WeChat account, and "30" represents the verification code.

[0048] In addition, the sample image also has a direction tag, which indicates the layout direction of the sample image. The direction tag may include any one of the following: center, right, and downward.

[0049] Step 204: Acquire the text content contained in each interface element.

[0050] In one example, the text content of each interface element included in the sample image may be read from a preset storage unit. The predicted storage unit is used to record the text content of each interface element included in each of the sample images.

[0051] Taking any sample image as an example, the text content of each interface element contained in the sample image can be obtained by performing text recognition on the sample image using optical character recognition (OCR) technology. OCR technology can refer to detecting and recognizing text in an image. For example, OCR technology can include two parts: text detection (TextDetection) and text recognition (TextRecognition). Text detection locates the area where text exists in the image, that is, finds the bounding box of a word or text line; text recognition recognizes the located text.

[0052] It should be understood that when using OCR technology to recognize sample images, two parts of content can be obtained for each interface element: a bounding box and text content, so that the respective category labels can be annotated based on the bounding boxes of each interface element.

[0053] In another example, OCR technology may be directly used to perform text recognition on the sample image to obtain the text content included in each interface element.

[0054] by Figure 3b For example, the text content included in each prompt and each input box may be as shown in Table 1.

[0055] Table 1

[0056] User Account Username must be 5 to 11 letters or numbers Login Password The password is 6-16 English letters or numbers Confirm Password Please enter your password again Real name Please enter your real name cell phone Please enter your phone number Withdrawal password Please enter the withdrawal password WeChat ID Please enter your WeChat ID Verification Code Please enter the verification code

[0057] Step 206: input the sample image and each text content into the category prediction model for prediction processing.

[0058] The category prediction model here may include: a feature extractor, a first encoder, a second encoder, and a classifier.

[0059] The above input process is first described below.

[0060] In one example, the inputting of the sample image and each text content into the category prediction model may include: masking a preset proportion of the text content in each text content with a preset phrase to obtain each text content after masking. The sample image and each text content after masking are input into the category prediction model. The above prediction proportion may be, for example, 10%.

[0061] It should be understood that in this method of inputting text content, the model is required to identify the category of the interface element based only on the position information, thereby strengthening the model's attention to the position information of the interface elements, thereby avoiding misjudgment caused by the model's excessive focus on the text content itself.

[0062] In another example, the above-mentioned inputting the sample image and each text content into the category prediction model may include: replacing a preset proportion of candidate content corresponding to each candidate content in the non-input box in each text content with the text content of the input box to obtain each text content after replacement. The sample image and each text content after replacement are input into the category prediction model.

[0063] This method of inputting text content can also strengthen the model's attention to position information and improve the model's recognition accuracy. For example, some text content is included in both input boxes and non-input boxes. If you pay too much attention to the text content itself and ignore the position information, it is easy to misidentify the non-input box category as the input box category. The following is an explanation with specific examples.

[0064] Figure 3c A schematic diagram of a sample image in another embodiment is shown. Figure 3c In the example above, the text contents in the two boxes are similar. During the training process, if the text contents of the non-input boxes are randomly replaced with the text contents of the input box, the model can resist the influence of such noise when making judgments, pay less attention to the wrong text contents, and make judgments based more on the position.

[0065] The above is an explanation of the input process. The following is an explanation of the above prediction processing process.

[0066] The above prediction process may include: step a, extracting image features from a sample image using a feature extractor. Step b, determining a feature vector of each text content based on a word vector of each word in each text content using a first encoder. Step c, determining a comprehensive feature representation of each interface element based on image features and feature vectors of each text content using a second encoder. Step d, determining a first category prediction result of each interface element based on each comprehensive feature representation using a classifier.

[0067] First, in step a, the feature extractor can be implemented as a SEResNeXt-50 series network, which is trained based on the imageNet dataset. Compared with the previous network, this series of networks has increased the number of channel groups. The specific implementation principle is: by controlling the size of the image (including depth and width), the important features between each channel are enhanced and the unimportant features are weakened, so that the extracted features are more directional.

[0068] The image features obtained in step a can be converted into a feature map of a predetermined size. For example, it can be converted into a feature map of 7*7*1024. Then, a matrix conversion function (for example, reshape()) can be used to flatten the feature map of the predetermined size into a row, so that a predetermined number of feature maps can be obtained, and the dimensions of each feature map are the same.

[0069] Of course, in practical applications, after obtaining a predetermined number of feature maps, they can also be compressed so that the dimension of the feature map is consistent with the dimension of the text content. For example, for the above 7*7*1024 feature map, after flattening it, a 49*1024 feature map can be obtained, and then a convolution layer with a convolution kernel size of 1*1*1024*768 can be used to compress it into a 49*768 feature map. Among them, 49 represents the number of feature maps, and 768 represents the dimension of the feature map.

[0070] Next, in step b, the first encoder may include a first embedding layer and a first attention layer. Specifically, in the first embedding layer, the embedding vector of each word in each text content may be obtained based on the word vector of each word and the corresponding sequence number. In the first attention layer, different weights are applied to the embedding vectors of each word in the same text content to obtain the feature vector of the corresponding text content.

[0071] In one example, the first encoder is specifically a Transformer encoder, so that after the word vectors of each word in a text content and the corresponding sequence numbers are input into the Transformer encoder, the [CLS] token of the Transformer encoder can be output as a feature vector of the text content.

[0072] It should be noted that, in general, it is more reasonable to consider the text content contained in the interface elements as a whole. Therefore, this solution first inputs each word in each text content into the first encoder to obtain the feature vector of each text content, and then inputs the feature vector of each text content into the second encoder, so that the second encoder can focus on the correlation between the text contents, as well as the correlation between the image features and the text contents, which can improve the accuracy of the feature representation of the interface elements.

[0073] Secondly, in step c, the second encoder may include a second embedding layer and a second attention layer. In the second embedding layer, based on each feature map and its sequence number, and the position information of each feature map in the sample image, the embedding vector of each feature map is obtained, and based on the feature vector of each text content and its sequence number, and the position information of the corresponding interface element, the embedding vector of each text content is obtained. In the second attention layer, different weights are applied to the embedding vectors of each feature map and the embedding vectors of each text content to obtain a comprehensive feature representation of each interface element.

[0074] First, the second encoder may also be a Transformer encoder. In addition, the position information of each feature map in the sample image may be determined based on the coordinate information of the N equally divided regions after the sample image is divided into N equally divided regions. Here, N is the number of feature maps finally obtained in the above step a.

[0075] Taking the number of feature maps (ie, N) as 49 as an example, the sample image is divided into 49 equal parts, that is, the length and width of the sample image are divided into 7 equal parts respectively, so that 49 equally divided regions can be obtained. For each equally divided region, it has corresponding coordinate information. In an example, the coordinate information of each equally divided region is composed of the vertex coordinates of the upper left corner and the vertex coordinates of the lower right corner of the region, which can be specifically expressed as: (((X 左上 ,Y 左上 ),( X 右下 ,Y 右下 )).

[0076] It should be noted that the above-mentioned equally divided regions can be flattened into a row in a predetermined order (for example, from left to right, from top to bottom), and each equally divided region after being flattened has a corresponding sequence number (generally starting from 0). Then, the corresponding relationship between each equally divided region and each feature map can be established according to the sequence number, and for any feature map, the coordinate information of the corresponding equally divided region can be used as its position information in the sample image.

[0077] In addition, the sequence numbering (starting from 0) of the above-mentioned text contents can be determined after the text contents are laid out in a row according to a predetermined sequence. Taking Table 1 as an example, after the text contents are laid out in a row from left to right and from top to bottom, the sequence numbering of the text content: "Login password" can be: "2", the sequence numbering of the text content: "Mobile phone" can be: "8", and so on.

[0078] The position information of the above interface elements can be determined based on the coordinate information of the bounding box detected by OCR technology. The coordinate information of the bounding box here can also be composed of the vertex coordinates of the upper left corner of the box and the vertex coordinates of the lower right corner of the box. The specific representation method can be referred to the above-mentioned equally divided area, which will not be repeated here.

[0079] Of course, in practical applications, the second encoder can also be used to obtain a comprehensive feature representation of the sample image. Specifically, the [CLS] token output of the Transformer encoder can be used as a comprehensive feature representation of the sample image.

[0080] In summary, the second encoder obtains the comprehensive feature representation of each interface element by comprehensively considering the correlation between each text content and the correlation between image features and each text content, which can improve the accuracy of the feature representation of the interface elements and further improve the accuracy of interface element category recognition.

[0081] Finally, in step d, the first category prediction results of each interface element may include two: one is a main category prediction result, and the other is a subcategory prediction result.

[0082] In addition, when the second encoder further determines the comprehensive feature representation of the sample image, the classifier can be used to determine the first direction prediction result based on the comprehensive feature representation of the sample image.

[0083] Step 208: training a category prediction model based on at least the first category prediction result of each interface element and each category label.

[0084] In one example, the first losses corresponding to each interface element can be determined based on the first category prediction results and category labels of each interface element. Then, the first losses are averaged or weighted averaged to obtain the predicted loss. Finally, the first parameter gradient is determined based on the predicted loss, and the parameters of the category prediction model are adjusted based on the first parameter gradient. At this point, the category prediction model training is completed.

[0085] In one example, the first loss of the interface element i may be determined according to the following formula:

[0086] (Formula 1)

[0087] in, is the first loss of interface element i, The main category loss can be determined based on the main category prediction result and the main category label of the interface element i. is a confidence function. Specifically, if interface element i has a subcategory label, for example, interface element i is an input box or a prompt, then it is 1, otherwise it is 0. is the subcategory loss, which can be determined based on the subcategory prediction result and subcategory label of interface element i.

[0088] It can be seen that for any interface element i, if it has both a main category label and a subcategory label, the main category loss can be determined based on the main category prediction result and the main category label, and the subcategory loss can be determined based on the subcategory prediction result and the subcategory label. Then the main category loss and the subcategory loss are fused (for example, summed or weighted summed) to obtain the corresponding first loss.

[0089] For any interface element i, if it only has a main category label, the main category loss can be determined according to the main category prediction result and the main category label, and the main category loss can be used as the corresponding first loss.

[0090] In another example, if the classifier can also be used to determine the first direction prediction result of the sample image, then the second loss can also be determined based on the first direction prediction result and the direction label. Then the above prediction loss and the second loss are fused (for example, summed or weighted summed) to obtain the total loss. Finally, based on the total loss, the first parameter gradient is determined, and the parameters of the category prediction model are adjusted based on the first parameter gradient. At this point, the training process of the category prediction model is completed.

[0091] In one example, the total loss can be determined according to the following formula:

[0092] (Formula 2)

[0093] in, is the total loss, For the second loss, is the first loss of interface element i, and N is the number of interface elements.

[0094] It should be noted that in the embodiment of this specification, by adding the above second loss, the model can accurately identify interface elements that do not contain text content. In short, by adding the second loss in this solution, the purpose of auxiliary training model is achieved.

[0095] In another example, after determining the above-mentioned prediction loss or total loss, the feature gradient of the prediction loss (or total loss) relative to the word vector of each word in each text content can be determined. According to the feature gradient, the target perturbation amount is determined. The corresponding target perturbation amount is superimposed on the word vector of each word, thereby obtaining each perturbation content.

[0096] In one example, determining the target disturbance amount according to the feature gradient may include: obtaining the gradient sign of the feature gradient, and taking the product of the gradient sign and a predetermined value ε as the target disturbance amount; or normalizing the feature gradient, and taking the product of the normalized feature gradient and a predetermined value ε as the target disturbance amount. The predetermined value ε here is a hyperparameter.

[0097] Among them, the gradient sign of the above-mentioned characteristic gradient can be obtained by using a sign function.

[0098] After obtaining each disturbance content, the sample image and each disturbance content may be input into a category prediction model to perform another prediction process.

[0099] It should be understood that through this prediction process again, the second category prediction results of each interface element can be obtained. In addition, the second direction prediction results of the sample image can also be obtained. After that, the third losses can be obtained according to the second category prediction results of each interface element and each category label; and the fourth loss can be obtained according to the second direction prediction results and direction labels of the sample image. After the third losses and the fourth losses are fused, the second parameter gradient can be obtained based on the fused loss.

[0100] It should be noted that the calculation method of the third loss can refer to the above formula 1, and the calculation method of the fourth loss is similar to the calculation method of the above second loss. The determination method of the fusion loss of each third loss and the fourth loss can refer to formula 2, which will not be repeated in this specification.

[0101] After the second parameter gradient is determined, the first parameter gradient and the second parameter gradient may be fused (eg, summed or weighted summed), and then the parameters of the category prediction model are adjusted according to the fused gradient. The training process of the category prediction model is now complete.

[0102] It should be noted that the model training method given in the above example actually adds adversarial training to the normal training method based on gradient descent. The training process can be summarized as follows: after the sample image and each text content are input into the model and the forward loss is calculated, the model parameters are not adjusted directly based on the parameter gradient calculated by the forward loss, but the feature gradient is calculated based on the forward loss, and then the text content is perturbed in the direction of the rising feature gradient, and then the sample image and each perturbed content are input into the model, the forward loss is calculated again, and the parameter gradient is calculated again, and finally the parameter gradients calculated twice are fused, and the model parameters are adjusted based on the fused gradient.

[0103] The following is an explanation of the reasons for adding adversarial training: When filling out a form, some high-frequency words (such as account, user, etc.) will appear, causing the model to converge quickly and overfit on the training set. For this reason, adversarial training is needed to increase the model's ability to resist noise. In addition, forms on non-standard websites often contain typos, which usually affect the accuracy of the model. In this solution, by adding adversarial training, the impact of typos on model accuracy can be reduced, which means that the robustness of the model can be improved.

[0104] Figure 4 A schematic diagram of a method for training a category prediction model in one embodiment is shown. Figure 4 In the example, for the current sample image, two different processing can be performed through two links.

[0105] The first processing includes: inputting the sample image into the feature extractor to extract image features. The image features here can include 49 feature maps after processing (flattening and compression, etc.), represented as V1-V49. After that, each feature map and its sequence number 0-48, as well as the location information of each feature map in the sample image Box V1 -Box V49 , input into the second encoder for encoding processing.

[0106] The second processing includes: obtaining the text content contained in each interface element in the sample image. For each text content, the word vector t1-t4 of each word and its sequence number 1-4 are input into the first encoder to obtain the corresponding feature vector, which is represented as: T1-T6. After that, the feature vector of each text content and its sequence number 1-6, as well as the position information Box of the corresponding interface element can be obtained. T1 -Box T6 The data is input into the second encoder for encoding.

[0107] The second encoder applies different weights to the embedding vectors of each feature map and the embedding vectors of each text content to obtain a comprehensive feature representation of each interface element. In addition, a comprehensive feature representation of the sample image can also be obtained. Finally, based on the comprehensive feature representations, the classifier can be used to obtain the main category prediction result L1 and the subcategory prediction result L2 of each interface element, as well as the first direction prediction result label of the sample image. According to L1 and L2 of each interface element and the category label, and the label and direction label of the sample image, a category prediction model can be trained. The specific training process can be found in the last two examples in step 208, which will not be repeated in this specification.

[0108] In summary, the training method of the category prediction model provided in the embodiment of this specification can train the category prediction model based on the image features of the sample image and the text content of each interface element contained in the sample image. In other words, the category prediction model can be trained based on a variety of information such as vision and text, which can greatly improve the accuracy of the model. In addition, during the prediction process, the present solution first uses the first encoder to obtain the feature vector of each text content, and then inputs it into the second encoder, so that when the second encoder determines the comprehensive feature representation of the interface element, it can focus on the correlation between the text contents, rather than just the correlation between individual words. This can improve the accuracy of the feature representation of the interface element, and thus improve the accuracy of the category prediction results.

[0109] It should be understood that after the category prediction model is trained, the category of the interface element can be identified based on the category prediction model. The specific identification process is described below.

[0110] Figure 5 FIG. 1 is a flow chart of a method for identifying interface element categories according to an embodiment. It is understood that the method can be executed by any device, equipment, platform, or device cluster with computing and processing capabilities. Figure 5 As shown, the identification process includes at least the following steps.

[0111] Step 502: Acquire a target image, where the target image includes a target interface element.

[0112] The target image here may be, for example, a form image. The target interface element may be a prompt, an input box, a button or others.

[0113] Step 504: Acquire target content contained in the target interface element.

[0114] In one example, OCR technology may be used to perform text recognition on a target image to obtain text content and a corresponding bounding box contained in a target interface element.

[0115] Step 506, obtaining a category prediction model.

[0116] The category prediction model can be Figure 2 The method steps shown are trained. As mentioned above, the category prediction model may include: a feature extractor, a first encoder, a second encoder and a classifier.

[0117] Step 508: Input the target image and target content into the category prediction model for prediction processing to obtain the target category of the target interface element.

[0118] The prediction process may specifically include: extracting target image features from the target image using a feature extractor. Determining a feature vector of the target content using a first encoder based on word vectors of each word in the target content. Determining a target feature representation of the target interface element using a second encoder based on the target image features and the feature vector of the target content. Determining a target category of the target interface element based on the target feature representation using a classifier.

[0119] The target category of the target interface element may include a main category and a subcategory. The main category may include any one of the following: prompt, input box, button or others. For an interface element whose main category is an input box or prompt, its corresponding subcategory may include any one of the following: account, password, mobile phone number, name, verification code, etc.

[0120] Corresponding to the above-mentioned training method of the category prediction model, an embodiment of the present specification also provides a training device for the category prediction model, such as Figure 6 As shown, the device may include:

[0121] The acquisition unit 602 is used to acquire a sample image, where the sample image includes a number of interface elements, each of which has a corresponding category label.

[0122] The acquisition unit 602 is further used to acquire each text content included in each interface element.

[0123] The prediction processing unit 604 is used to input the sample image and each text content into the category prediction model for prediction processing. The category prediction model includes a feature extractor, a first encoder, a second encoder and a classifier. The prediction processing includes: using the feature extractor to extract image features from the sample image, using the first encoder to determine the feature vector of each text content based on the word vector of each word in each text content, using the second encoder to determine the comprehensive feature representation of each interface element based on the image feature and the feature vector of each text content, and using the classifier to determine the first category prediction result of each interface element based on each comprehensive feature representation.

[0124] The prediction processing unit 604 is specifically used for:

[0125] Masking a preset proportion of the text content in each text content with a preset phrase to obtain each text content after masking;

[0126] The sample image and the text contents after masking are input into the category prediction model.

[0127] Among them, each interface element includes an input box;

[0128] The prediction processing unit 604 is further specifically configured to:

[0129] Replacing a preset proportion of candidate contents in each text content corresponding to each candidate content not in the input box with the text content in the input box to obtain each text content after replacement processing;

[0130] The sample images and the replaced text contents are input into the category prediction model.

[0131] The training unit 606 is used to train a category prediction model based on at least the first category prediction result of each interface element and each category label.

[0132] The training unit 606 includes:

[0133] The determination module 6062 is used to determine the first parameter gradient according to at least the first category prediction result of each interface element and each category label.

[0134] The perturbation module 6064 is used to perturb each text content to generate each perturbation content.

[0135] The disturbance module 6064 is specifically used for:

[0136] Determine a feature gradient of a prediction loss relative to a word vector for each word, the prediction loss being determined based on at least the first category prediction result and each category label;

[0137] According to the characteristic gradient, the target disturbance amount is determined;

[0138] The target perturbation amount is superimposed on the word vector of each word to obtain the perturbation content.

[0139] The disturbance module 6064 is also specifically used for:

[0140] Obtain the gradient sign of the characteristic gradient, and use the product of the gradient sign and a predetermined value as the target disturbance amount; or,

[0141] The feature gradient is normalized, and the product of the normalized feature gradient and a predetermined value is used as the target perturbation.

[0142] The input module 6066 is used to input the sample image and each disturbance content into the category prediction model for prediction processing to obtain the second category prediction result of each interface element.

[0143] The determination module 6062 is further configured to determine a second parameter gradient at least according to the second category prediction result of each interface element and each category label.

[0144] The adjustment module 6068 is used to fuse the first parameter gradient and the second parameter gradient, and adjust the parameters of the category prediction model based on the fused gradient.

[0145] Optionally, the sample image further has a direction label, which indicates the layout direction of the sample image;

[0146] The prediction processing unit 604 is further configured to use a classifier to determine a first direction prediction result of the sample image.

[0147] The training unit 606 is specifically used for:

[0148] Determine each first loss according to the first category prediction result and each category label of each interface element, and determine the second loss according to the first direction prediction result and the direction label;

[0149] According to each first loss and second loss, a total loss is obtained;

[0150] Based on the total loss, adjust the parameters of the category prediction model.

[0151] Optionally, each of the above interface elements includes an input box, the first category prediction result of the input box includes a main category prediction result and a subcategory prediction result, and the category label corresponding to the input box includes a main category label and a subcategory label;

[0152] The training unit 606 is further specifically configured to:

[0153] Determine the main category loss according to the main category prediction result and the main category label of the input box, and determine the subcategory loss according to the subcategory prediction result and the subcategory label of the input box;

[0154] The main category loss and the subcategory loss are fused to obtain the first loss corresponding to the input box.

[0155] Optionally, the first encoder comprises a first embedding layer and a first attention layer;

[0156] The prediction processing unit 604 is specifically used for:

[0157] In the first embedding layer, the embedding vector of each word is obtained based on the word vector of each word in each text content and the corresponding sequence number;

[0158] In the first attention layer, different weights are given to the embedding vectors of each word in the same text content to obtain the feature vectors of each text content.

[0159] Optionally, the image features include N feature maps, and the second encoder includes a second embedding layer and a second attention layer;

[0160] The prediction processing unit 604 is specifically used for:

[0161] In the second embedding layer, based on each feature map and its sequence number, and the position information of each feature map in the sample image, the embedding vector of each feature map is obtained, and based on the feature vector of each text content and its sequence number, and the position information of the corresponding interface element, the embedding vector of each text content is obtained;

[0162] In the second attention layer, different weights are given to the embedding vectors of each feature map and the embedding vectors of each text content to obtain a comprehensive feature representation of each interface element.

[0163] The position information of each feature map in the sample image is determined based on the coordinate information of the N equally divided areas after the sample image is divided into N equally divided areas.

[0164] The position information of the interface elements is obtained by performing text recognition on the sample images.

[0165] The functions of the functional modules of the device in the above-mentioned embodiment of this specification can be implemented through the steps of the above-mentioned method embodiment. Therefore, the specific working process of the device provided by one embodiment of this specification will not be repeated here.

[0166] A training device for a category prediction model provided in an embodiment of the present specification can simultaneously focus on the image features of an image and the correlation between the text contents of each interface element contained in the image, thereby more accurately identifying the category of the interface element.

[0167] Corresponding to the above-mentioned method for identifying the category of interface elements, an embodiment of the present specification also provides a device for identifying the category of interface elements, such as Figure 7 As shown, the device may include:

[0168] The acquisition unit 702 is used to acquire a target image, where the target image includes a target interface element.

[0169] The acquisition unit 702 is further used to acquire the target content contained in the target interface element.

[0170] The acquisition unit 702 is further used to acquire a category prediction model, wherein the category prediction model is based on Figure 2 The method steps shown are obtained by training.

[0171] The prediction processing unit 704 is used to input the target image and target content into the category prediction model for prediction processing to obtain the target category of the target interface element.

[0172] The functions of the functional modules of the device in the above-mentioned embodiment of this specification can be implemented through the steps of the above-mentioned method embodiment. Therefore, the specific working process of the device provided by one embodiment of this specification will not be repeated here.

[0173] An interface element category recognition device provided in an embodiment of the present specification can accurately recognize the category of the interface element.

[0174] According to another embodiment, there is also provided a computer-readable storage medium having a computer program stored thereon, which, when executed in a computer, causes the computer to execute a combination of Figure 2 , Figure 4 or Figure 5 The method described.

[0175] According to another embodiment of the present invention, a computing device is provided, including a memory and a processor, wherein the memory stores executable code, and when the processor executes the executable code, the Figure 2 , Figure 4 or Figure 5 The method described.

[0176] Each embodiment in this specification is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0177] The steps of the method or algorithm described in conjunction with the disclosure of this specification can be implemented in hardware or by a processor executing software instructions. The software instructions can be composed of corresponding software modules, and the software modules can be stored in RAM memory, flash memory, ROM memory, EPROM memory, EEPROM memory, register, hard disk, mobile hard disk, CD-ROM or any other form of storage medium known in the art. An exemplary storage medium is coupled to the processor so that the processor can read information from the storage medium and write information to the storage medium. Of course, the storage medium can also be a component of the processor. The processor and the storage medium can be located in an ASIC. In addition, the ASIC can be located in a server. Of course, the processor and the storage medium can also be present in the server as discrete components.

[0178] Those skilled in the art will appreciate that in one or more of the above examples, the functions described in the present invention may be implemented in hardware, software, firmware, or any combination thereof. When implemented using software, the functions may be stored in a computer-readable medium or transmitted as one or more instructions or codes on a computer-readable medium. Computer-readable media include computer storage media and communication media, wherein communication media include any media that facilitates the transmission of a computer program from one place to another. The storage medium may be any available medium that a general or special-purpose computer can access.

[0179] The above is a description of a specific embodiment of the specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in an order different from that in the embodiments and still achieve the desired results. In addition, the processes depicted in the drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0180] The specific implementation methods described above further illustrate the purpose, technical solutions and beneficial effects of this specification. It should be understood that the above description is only the specific implementation method of this specification and is not intended to limit the protection scope of this specification. Any modifications, equivalent substitutions, improvements, etc. made on the basis of the technical solutions of this specification should be included in the protection scope of this specification.

Claims

1. A training method for a category prediction model, comprising: Acquire a sample image, wherein the sample image includes a plurality of interface elements, each of which has a corresponding category label; Get the text content contained in each interface element; Inputting the sample image and the text contents into a category prediction model for prediction processing; The category prediction model includes a feature extractor, a first encoder, a second encoder and a classifier; the prediction process includes: using the feature extractor to extract image features from the sample image; using the first encoder to determine the overall feature vector of each text content based on the word vector of each word in the text content; inputting the image features and the overall feature vector of each text content into the second encoder, so that the second encoder determines the comprehensive feature representation of each interface element based on the correlation between the text contents and the correlation between the image features and the text contents; using the classifier to determine the first category prediction result of each interface element based on each comprehensive feature representation; The category prediction model is trained based on at least the first category prediction results of each interface element and each category label.

2. The method according to claim 1, wherein: The step of training the category prediction model at least according to the first category prediction results of each interface element and each category label includes: Determining a first parameter gradient at least according to the first category prediction results of the respective interface elements and the respective category labels; Perturbing each of the text contents to generate each perturbed content; Inputting the sample image and the disturbance contents into the category prediction model for the prediction process to obtain the second category prediction result of each interface element; Determining a second parameter gradient at least according to the second category prediction results of each interface element and each category label; The first parameter gradient and the second parameter gradient are fused, and based on the fused gradient, the parameters of the category prediction model are adjusted.

3. The method according to claim 2, wherein: The perturbing the contents of each text includes: Determine a feature gradient of a prediction loss relative to a word vector of each word, wherein the prediction loss is determined at least based on the first category prediction result and each category label; Determining a target disturbance amount according to the characteristic gradient; The target disturbance amount is respectively superimposed on the word vector of each word, so as to obtain the disturbance content.

4. The method according to claim 3, wherein: Determining a target disturbance amount according to the characteristic gradient includes: Obtaining the gradient sign of the characteristic gradient, and taking the product of the gradient sign and a predetermined value as the target disturbance amount; or, The characteristic gradient is normalized, and the product of the normalized characteristic gradient and the predetermined value is used as the target disturbance amount.

5. The method according to claim 1, wherein: The sample image also has a direction label, and the direction label indicates the layout direction of the sample image; The prediction process further includes determining a first direction prediction result of the sample image using the classifier; The training of the category prediction model comprises: Determine each first loss according to the first category prediction result of each interface element and each category label, and determine the second loss according to the first direction prediction result and the direction label; Obtaining a total loss according to the first losses and the second loss; According to the total loss, the parameters of the category prediction model are adjusted.

6. The method according to claim 5, wherein: Each interface element includes an input box; the first category prediction result of the input box includes a main category prediction result and a subcategory prediction result; the category label corresponding to the input box includes a main category label and a subcategory label; The determining of each first loss according to the first category prediction result of each interface element and each category label includes: Determine a main category loss according to a main category prediction result and a main category label of the input frame, and determine a subcategory loss according to a subcategory prediction result and a subcategory label of the input frame; The main category loss and the subcategory loss are fused to obtain a first loss corresponding to the input box.

7. The method according to claim 1, wherein: The step of inputting the sample image and each text content into a category prediction model includes: Masking a preset proportion of the text contents in the text contents with a preset phrase to obtain the masked text contents; The sample image and each text content after masking are input into the category prediction model.

8. The method according to claim 1, wherein: Each of the interface elements includes an input box; The step of inputting the sample image and each text content into a category prediction model includes: Replacing a preset proportion of candidate contents in the text contents corresponding to the non-input boxes with the text contents in the input boxes, to obtain the replaced text contents; The sample image and each text content after replacement processing are input into the category prediction model.

9. The method according to claim 1, wherein: The first encoder comprises a first embedding layer and a first attention layer; Determining the overall feature vector of each text content includes: In the first embedding layer, based on the word vector of each word in the text content and the corresponding sequence number, an embedding vector of each word is obtained; In the first attention layer, different weights are given to the embedding vectors of each word in the same text content to obtain the overall feature vector of each text content.

10. The method according to claim 1, wherein: The image features include N feature maps; the second encoder includes a second embedding layer and a second attention layer; Determining the comprehensive feature representation of each interface element includes: In the second embedding layer, based on each feature map and its sequence number, and the position information of each feature map in the sample image, an embedding vector of each feature map is obtained, and based on the overall feature vector of each text content and its sequence number, and the position information of the corresponding interface element, an embedding vector of each text content is obtained; In the second attention layer, different weights are applied to the embedding vectors of the feature maps and the embedding vectors of the text contents to obtain a comprehensive feature representation of the interface elements.

11. The method according to claim 10, wherein: The position information of each feature map in the sample image is determined based on the coordinate information of the N equally divided areas after the sample image is divided into N equally divided areas.

12. The method according to claim 10, wherein: The position information of the interface element is obtained by performing text recognition on the sample image.

13. A method for identifying an interface element category, comprising: Acquire a target image, wherein the target image includes a target interface element; Obtaining target content contained in the target interface element; Obtain a category prediction model, wherein the category prediction model Trained according to the method as claimed in claim 1; The target image and the target content are input into the category prediction model for prediction processing to obtain the target category of the target interface element.

14. A training device for a category prediction model, comprising: An acquisition unit, used for acquiring a sample image, wherein the sample image includes a plurality of interface elements, each of which has a corresponding category label; The acquisition unit is further used to acquire each text content contained in each interface element; A prediction processing unit, used for inputting the sample image and the text contents into a category prediction model for prediction processing; The category prediction model includes a feature extractor, a first encoder, a second encoder and a classifier; the prediction process includes: using the feature extractor to extract image features from the sample image; using the first encoder to determine the overall feature vector of each text content based on the word vector of each word in the text content; inputting the image features and the overall feature vector of each text content into the second encoder, so that the second encoder determines the comprehensive feature representation of each interface element based on the correlation between the text contents and the correlation between the image features and the text contents; using the classifier to determine the first category prediction result of each interface element based on each comprehensive feature representation; A training unit is used to train the category prediction model based on at least the first category prediction results of each interface element and each category label.

15. The device according to claim 14, wherein: The training unit comprises: A determination module, configured to determine a first parameter gradient based at least on the first category prediction results of the interface elements and the category labels; A disturbance module, used for disturbing the text contents to generate disturbance contents; An input module, used for inputting the sample image and the disturbance contents into the category prediction model for the prediction process to obtain a second category prediction result of each interface element; The determination module is further configured to determine a second parameter gradient based at least on the second category prediction results of each interface element and each category label; An adjustment module is used to fuse the first parameter gradient and the second parameter gradient, and adjust the parameters of the category prediction model based on the fused gradient.

16. The device according to claim 15, wherein: The disturbance module is specifically used for: Determine a feature gradient of a prediction loss relative to a word vector of each word, wherein the prediction loss is determined at least based on the first category prediction result and each category label; Determining a target disturbance amount according to the characteristic gradient; The target disturbance amount is respectively superimposed on the word vector of each word, so as to obtain the disturbance content.

17. The device according to claim 14, wherein: The sample image also has a direction label, and the direction label indicates the layout direction of the sample image; The prediction processing unit is further used to determine the first direction prediction result of the sample image using the classifier; The training unit is specifically used for: Determine each first loss according to the first category prediction result of each interface element and each category label, and determine the second loss according to the first direction prediction result and the direction label; Obtaining a total loss according to the first losses and the second loss; According to the total loss, the parameters of the category prediction model are adjusted.

18. The device according to claim 17, wherein: Each interface element includes an input box; the first category prediction result of the input box includes a main category prediction result and a subcategory prediction result; the category label corresponding to the input box includes a main category label and a subcategory label; The training unit is also specifically used for: Determine a main category loss according to a main category prediction result and a main category label of the input frame, and determine a subcategory loss according to a subcategory prediction result and a subcategory label of the input frame; The main category loss and the subcategory loss are fused to obtain a first loss corresponding to the input box.

19. The device according to claim 14, wherein: The prediction processing unit is specifically used for: Masking a preset proportion of the text contents in the text contents with a preset phrase to obtain the masked text contents; The sample image and each text content after masking are input into the category prediction model.

20. The device according to claim 14, wherein: Each of the interface elements includes an input box; The prediction processing unit is further specifically used for: Replacing a preset proportion of candidate contents in the text contents corresponding to the non-input boxes with the text contents in the input boxes, to obtain the replaced text contents; The sample image and each text content after replacement processing are input into the category prediction model.

21. The device according to claim 14, wherein: The first encoder comprises a first embedding layer and a first attention layer; The prediction processing unit is specifically used for: In the first embedding layer, based on the word vector of each word in the text content and the corresponding sequence number, an embedding vector of each word is obtained; In the first attention layer, different weights are given to the embedding vectors of each word in the same text content to obtain the overall feature vector of each text content.

22. The device according to claim 14, wherein: The image features include N feature maps; the second encoder includes a second embedding layer and a second attention layer; The prediction processing unit is specifically used for: In the second embedding layer, based on each feature map and its sequence number, and the position information of each feature map in the sample image, an embedding vector of each feature map is obtained, and based on the overall feature vector of each text content and its sequence number, and the position information of the corresponding interface element, an embedding vector of each text content is obtained; In the second attention layer, different weights are applied to the embedding vectors of the feature maps and the embedding vectors of the text contents to obtain a comprehensive feature representation of the interface elements.

23. A device for identifying the category of an interface element, comprising: An acquisition unit, used to acquire a target image, wherein the target image includes a target interface element; The acquisition unit is further used to acquire the target content contained in the target interface element; The acquisition unit is further used to acquire a category prediction model, wherein the category prediction model is trained according to the method according to claim 1; The prediction processing unit is used to input the target image and the target content into the category prediction model for prediction processing to obtain the target category of the target interface element.

24. A computer-readable storage medium having a computer program stored thereon, wherein: When the computer program is executed in a computer, the computer is caused to execute the method according to any one of claims 1 to 13.

25. A computing device comprising a memory and a processor, wherein: The memory stores executable codes, and when the processor executes the executable codes, the method according to any one of claims 1 to 13 is implemented.

Images