Method and device for recognizing page images

Abstract

The present invention discloses a method and apparatus for recognizing page images, relating to the field of computer technology. A specific implementation of the method includes: obtaining text attributes of page text and determining image features corresponding to the text attributes; extracting local features of the page image; performing feature learning based on text attributes on the local features to obtain predicted features; and determining whether the page image matches the page text based on the similarity between the local features or the predicted features and the image features. This implementation enables automatic analysis of page images, thereby accurately and efficiently determining whether the page image matches the page text.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to a method and device for recognizing page images. Background Art

[0002] With the development of the Internet and big data, new technologies are emerging one after another, changing our lives every day. At the same time, many new applications have evolved based on artificial intelligence, and various pages have gradually become more personalized, refined and diversified. The images on the page can serve as an important reference for users to understand the page. How to display the most valuable images to users is a meaningful task. It is in this context that identifying whether the images on the page match the page content and filtering out irrelevant images can present users with a brand new visual experience.

[0003] The existing technology is for images on the page, and operators perform manual review to check the image quality.

[0004] In the process of implementing the present invention, the inventors discovered that the prior art has at least the following problems:

[0005] Audit standards vary from person to person and are not consistent; human energy is limited and efficiency is low; and analysis costs are high due to labor costs. Summary of the Invention

[0006] In view of this, an embodiment of the present invention provides a method and device for identifying page images, which uses a deep learning algorithm to automatically identify page images and predict whether the page images match text attributes based on a unified standard. This method significantly improves both recognition accuracy and prediction speed, freeing up manual resources.

[0007] To achieve the above objective, according to one aspect of an embodiment of the present invention, a method for recognizing a page image is provided.

[0008] A method for recognizing a page image according to an embodiment of the present invention includes:

[0009] Obtaining text attributes of page text and determining image features corresponding to the text attributes;

[0010] Extract local features of page images;

[0011] Performing feature learning based on the text attributes on the local features to obtain prediction features;

[0012] Whether the page image is consistent with the page text is determined according to the similarity between the local feature or the predicted feature and the image feature.

[0013] Optionally, extracting local features of the page image includes:

[0014] Obtaining a pixel matrix of the page image;

[0015] Reading global features from the pixel matrix using a multi-label image classification network architecture;

[0016] The multi-label image classification network architecture is used to pool the key points of the global features to obtain local features.

[0017] Optionally, performing feature learning based on the text attributes on the local features to obtain prediction features includes:

[0018] Using the multi-label image classification network architecture, the local features are subjected to feature learning based on text attributes to obtain learning features;

[0019] The key points of the learning features are pooled using the multi-label image classification network architecture to obtain predicted features.

[0020] Optionally, determining whether the page image and the page text are consistent with each other based on similarity between the local feature or the predicted feature and the image feature includes:

[0021] Determining whether the local features and the predicted features are similar to the image features using the multi-label image classification network architecture;

[0022] If the local feature or the predicted feature is similar to the image feature, determining that the page image is consistent with the page text;

[0023] If the local feature or the predicted feature is not similar to the image feature, it is determined that the page image does not match the page text.

[0024] Optionally, the training process of the multi-label image classification network architecture includes:

[0025] Initializing the parameters of the multi-label image classification network architecture using the parameters of the existing image classification basic network structure;

[0026] Extracting a base image marked with a category label, and performing translation, rotation, flipping, scaling, color conversion, or noise perturbation on the base image to generate an enhanced image;

[0027] Adding the basic image and its corresponding category label and the enhanced image and its corresponding category label to a training dataset, and dividing the training dataset into a training set, a validation set, and a test set according to a preset ratio;

[0028] The training set, the validation set, and the test set are read into a memory using a reading thread, and the training set, the validation set, and the test set are read from the memory using a computing thread to train and test the multi-label image classification network architecture.

[0029] To achieve the above objective, according to another aspect of an embodiment of the present invention, a device for recognizing a page image is provided.

[0030] An apparatus for recognizing a page image according to an embodiment of the present invention includes:

[0031] An acquisition module, configured to acquire text attributes of page text and determine image features corresponding to the text attributes;

[0032] Extraction module, used to extract local features of page images;

[0033] A learning module, configured to perform feature learning on the local features based on the text attributes to obtain prediction features;

[0034] A determination module is configured to determine whether the page image is consistent with the page text based on similarity between the local feature or the predicted feature and the image feature.

[0035] Optionally, the extraction module is further configured to:

[0036] Obtaining a pixel matrix of the page image;

[0037] Reading global features from the pixel matrix using a multi-label image classification network architecture;

[0038] The multi-label image classification network architecture is used to pool the key points of the global features to obtain local features.

[0039] Optionally, the learning module is further configured to:

[0040] Using the multi-label image classification network architecture, the local features are subjected to feature learning based on text attributes to obtain learning features;

[0041] The key points of the learning features are pooled using the multi-label image classification network architecture to obtain predicted features.

[0042] Optionally, the determining module is further configured to:

[0043] Determining whether the local features and the predicted features are similar to the image features using the multi-label image classification network architecture;

[0044] If the local feature or the predicted feature is similar to the image feature, determining that the page image is consistent with the page text;

[0045] If the local feature or the predicted feature is not similar to the image feature, it is determined that the page image does not match the page text.

[0046] Optionally, a training module is also included for:

[0047] Initializing the parameters of the multi-label image classification network architecture using the parameters of the existing image classification basic network structure;

[0048] Extracting a base image marked with a category label, and performing translation, rotation, flipping, scaling, color conversion, or noise perturbation on the base image to generate an enhanced image;

[0049] Adding the basic image and its corresponding category label and the enhanced image and its corresponding category label to a training dataset, and dividing the training dataset into a training set, a validation set, and a test set according to a preset ratio;

[0050] The training set, the validation set, and the test set are read into a memory using a reading thread, and the training set, the validation set, and the test set are read from the memory using a computing thread to train and test the multi-label image classification network architecture.

[0051] To achieve the above objective, according to another aspect of an embodiment of the present invention, an electronic device for recognizing a page image is provided.

[0052] An electronic device for recognizing page images according to an embodiment of the present invention includes: one or more processors; a storage device for storing one or more programs, and when the one or more programs are executed by the one or more processors, the one or more processors implement a method for recognizing page images according to an embodiment of the present invention.

[0053] To achieve the above objective, according to another aspect of an embodiment of the present invention, a computer-readable storage medium is provided.

[0054] A computer-readable storage medium according to an embodiment of the present invention stores a computer program, which, when executed by a processor, implements a method for recognizing a page image according to an embodiment of the present invention.

[0055] One embodiment of the above invention has the following advantages or beneficial effects: because it adopts the technical means of obtaining the text attributes of the page and determining the image features corresponding to the text attributes; extracting the local features of the page image; performing feature learning based on the text attributes on the local features to obtain predicted features; and determining whether the page image and the page text are consistent based on the similarity between the local features or the predicted features and the image features, it overcomes the technical problems of the existing technology in judging whether the page image and the page text are consistent, that is, the review standards vary from person to person and the standards are inconsistent; the energy of people is limited and the efficiency is low; and the analysis cost is high, thereby achieving the technical effect of being able to automatically analyze the page image, obtain local features and predicted features from the page image, and thus accurately and efficiently determine whether the page image and the page text are consistent.

[0056] The further effects of the above-mentioned non-conventional optional manner will be described below in conjunction with specific embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0057] The accompanying drawings are provided for a better understanding of the present invention and are not intended to limit the present invention.

[0058] Figure 1 is a schematic diagram of main steps of a method for recognizing a page image according to an embodiment of the present invention;

[0059] Figure 2 Schematic diagram of the implementation process of a multi-label image classification network architecture according to a reference embodiment of the present invention;

[0060] Figure 3 is a schematic diagram of an application of a method for recognizing a page image according to an embodiment of the present invention;

[0061] Figure 4 is a schematic diagram of main modules of an apparatus for recognizing a page image according to an embodiment of the present invention;

[0062] Figure 5 is an exemplary system architecture diagram in which embodiments of the present invention may be applied;

[0063] Figure 6 It is a schematic diagram of the structure of a computer system of a terminal device or a server suitable for implementing an embodiment of the present invention. DETAILED DESCRIPTION

[0064] The following description of exemplary embodiments of the present invention is made in conjunction with the accompanying drawings, in which various details of the embodiments of the present invention are included to facilitate understanding. These details should be considered as merely exemplary. Therefore, it should be appreciated by those skilled in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present invention. Similarly, for the sake of clarity and conciseness, descriptions of well-known functions and structures are omitted in the following description.

[0065] It should be pointed out that, in the absence of conflict, the embodiments of the present invention and the technical features therein may be combined with each other.

[0066] Figure 1 FIG. 4 is a schematic diagram of main steps of a method for recognizing a page image according to an embodiment of the present invention.

[0067] like Figure 1 As shown, the method for recognizing a page image according to an embodiment of the present invention mainly includes the following steps:

[0068] Step S101: Acquire text attributes of page text and determine image features corresponding to the text attributes.

[0069] Usually a page may include both text and images. Whether the image corresponds to the text, that is, whether the page image matches the page text, for example, if a page is an introduction to football, if the page image is about football, then the page image and page text do not match; if the page image is about diving, then the page image and page text do not match. During analysis, we can first determine which image features the text attributes of the current page correspond to, and use the image features to analyze the page image.

[0070] Step S102: extracting local features of the page image.

[0071] For a page image, its content is certain, that is, its features are also certain, and local features included therein can be extracted. Through the local features, it can be determined whether the page image includes image features corresponding to text attributes.

[0072] In an embodiment of the present invention, step S102 can be implemented in the following manner: obtaining a pixel matrix of the page image; reading global features from the pixel matrix using a multi-label image classification network architecture; and pooling the key points of the global features using the multi-label image classification network architecture to obtain local features.

[0073] Global features refer to basic features such as graphics, materials, colors, outlines, or positions within a page image. Pooling the key points of these basic features can further derive local features. Pooling involves compressing global features, such as maximum pooling and average pooling.

[0074] Furthermore, the multi-label image classification network architecture is an improved neural network structure based on the multi-label clothing and color image classification (FashionNet), primarily used for image classification. It incorporates a key attribute localization layer on top of classic convolution. Steps S102 through S104 of the present embodiment can all be implemented using the multi-label image classification network architecture.

[0075] Step S103: Perform feature learning based on text attributes on the local features to obtain prediction features.

[0076] Feature learning based on text attributes refers to finding the prediction features that are closest to the text attributes from local features, that is, finding the features included in the page image that are most consistent with the text attributes.

[0077] In an embodiment of the present invention, step S103 can be implemented in the following manner: using a multi-label image classification network architecture to perform feature learning based on text attributes on local features to obtain learning features; using a multi-label image classification network architecture to pool the key points of the learning features to obtain prediction features.

[0078] For feature learning, a multi-label image classification network architecture can also be used, and in this process, the key points of the learning features are also pooled to obtain the predicted features.

[0079] Step S104: determining whether the page image and the page text are consistent based on the similarity between the local features or the predicted features and the image features.

[0080] After obtaining the local features and predicted features, we can further analyze the similarity with the image features to determine whether the page image is consistent with the page text.

[0081] In an embodiment of the present invention, step S104 can be implemented in the following manner: using a multi-label image classification network architecture to determine whether the local features and predicted features are similar to the image features; if the local features or predicted features are similar to the image features, it is determined that the page image is consistent with the page text; if the local features or predicted features are not similar to the image features, it is determined that the page image is inconsistent with the page text.

[0082] Whether local features are similar to image features, and whether predicted features are similar to image features can also be judged using the multi-label image classification network architecture. The key attribute positioning layer added on the basis of classical convolution can judge the visibility of local features and predicted features, that is, whether the local features and predicted features are the aforementioned image features. If the local features are similar to the image features or the predicted features are similar to the image features, it means that the local features or predicted features are visible, and the page image includes image features, that is, the image features are consistent with the text attributes, otherwise the opposite is true.

[0083] In an embodiment of the present invention, the training process of the multi-label image classification network architecture includes: initializing the parameters of the multi-label image classification network architecture using the parameters of the existing image classification basic network structure; extracting a basic image marked with a category label, and translating, rotating, flipping, scaling, color changing or noise perturbation the basic image to generate an enhanced image; adding the basic image and its corresponding category label and the enhanced image and its corresponding category label to a training data set, and dividing the training data set into a training set, a validation set and a test set according to a preset ratio; using a reading thread to read the training set, validation set and test set into memory, and using a computing thread to read the training set, validation set and test set from memory to train and test the multi-label image classification network architecture.

[0084] The parameter types of the basic network structure of image classification and the multi-label image classification network architecture are similar. In order to improve training efficiency and reduce the amount of training data, the multi-label image classification network architecture can use the parameters of the existing basic network structure of image classification as initial parameters. The training data in the training data set mainly includes two categories. One category is the basic image with the category label marked, and the other category is the enhanced image generated by performing operations such as translation, rotation, flipping, scaling, color transformation or noise perturbation on the basic image. The category label of the enhanced image is the same as the category label of its corresponding basic image. For training data, the training data in the training data set can be divided according to a preset ratio to obtain a training set, a validation set and a test set. The training set is used to train the multi-label image classification network architecture, the validation set is used to verify the accuracy of the multi-label image classification network architecture, and the test set is used to simulate real data to test the multi-label image classification network architecture. The preset ratio can be set according to actual needs. As a preferred embodiment, the ratio of the training set, validation set and test set can be 8:1:1.

[0085] Furthermore, since neural networks must first read in data before they can perform calculations, for example, it takes 0.1 seconds to read in data and 0.9 seconds to calculate it. This means that for every second that passes, the GPU has 0.1 seconds to do nothing, which greatly reduces computational efficiency. To address this issue, when training a multi-label image classification network architecture, the read data and the calculated data are placed in two separate threads. The read thread continuously reads the training data from the training dataset into a memory queue, while the calculation thread is responsible for the calculation. When the calculation requires data, the calculation thread reads it directly from the memory queue. Furthermore, to facilitate the reading of training data by the multi-label image classification network architecture, the training data format can be unified into tfrecord, a binary file that stores image data and category labels in a unified manner.

[0086] It can be seen from the method for recognizing page images according to an embodiment of the present invention that the method adopts the following technical means: obtaining the text attributes of the page and determining the image features corresponding to the text attributes; extracting the local features of the page image; performing feature learning based on the text attributes on the local features to obtain the predicted features; and determining whether the page image is consistent with the page text based on the similarity between the local features or the predicted features and the image features. Therefore, the method overcomes the technical problems of the prior art in judging whether the page image and the page text are consistent, that is, the review standards vary from person to person and the standards are inconsistent; the energy of people is limited and the efficiency is low; and the analysis cost is high, thereby achieving the technical effect of being able to automatically analyze the page image, obtain the local features and the predicted features from the page image, and thus accurately and efficiently determine whether the page image and the page text are consistent.

[0087] Figure 2 It is a schematic diagram of the implementation process of a method for multi-label image classification network architecture according to a reference embodiment of the present invention.

[0088] like Figure 2 As shown in FIG, the multi-label image classification network architecture (improved FashionNet) of the embodiment of the present invention adds a key attribute location layer on the basis of the basic convolutional network. After extracting the global and local features of the page image, the key attributes of the page image are located, as follows:

[0089] The global feature layer is mainly used to extract global features such as the outline and position of the entire page image;

[0090] The local feature layer is mainly used to pool key points to obtain local features of the page image;

[0091] The pooling sharing layer learns features through text attributes and then uses the estimated key point positions to pool the learned features to obtain predicted features. This layer is performed in an iterative manner to achieve parameter and algorithm sharing, and the same calculation is performed on all decomposed local features.

[0092] The image key point positioning layer is used to predict the location of key points and their visibility. Visibility refers to whether the key points exist (that is, the page image includes image features).

[0093] In order to further illustrate the technical idea of ​​the present invention, the technical solution of the present invention is now described in combination with specific application scenarios.

[0094] Taking the comment pictures of a certain page as an example, we filter out the comment pictures that are relevant to the content of the page and delete the irrelevant comment pictures. Figure 3As shown, in this scenario, the method for recognizing page images in an embodiment of the present invention can be divided into six parts, namely: data preparation, data enhancement, basic network structure for image classification, FashionNet network improvement, parameter debugging, and online business application. Among them, data preparation and data enhancement are to prepare training data for FashionNet network improvement, the basic network structure for image classification provides initialization parameters for FashionNet network improvement, parameter debugging is to train FashionNet network improvement, and online business application is to use FashionNet network improvement to identify whether the comment image is related to the page content. Specifically,

[0095] 1. Data Preparation

[0096] 1. Hive image corresponding label extraction

[0097] The basic data in the training dataset can be the uniform resource locator (URL) and category label of the basic image extracted from the data warehouse (hive) table of some big data platform;

[0098] 2. Dataset Division

[0099] The training dataset is divided into training set, validation set, and test set in a ratio of 8:1:1. The training set is used to train the improved FashionNet, the validation set is used to verify the accuracy of the improved FashionNet, and the test set is used to simulate real online data offline.

[0100] 3. Data reading distributed design

[0101] The reading of data and the calculation are placed in two threads respectively. The reading thread continuously reads images from the file system into a memory queue, while the calculation thread is responsible for the calculation. When the calculation needs data, it can directly get it from the memory queue.

[0102] 4. Data format conversion

[0103] The format of training data needs to be converted to tfrecord to facilitate reading by the improved FashionNet.

[0104] 2. Data Augmentation

[0105] Deep learning requires a large number of training samples. Generally speaking, the more data there is, the better the trained model will be. In image tasks, simple translation, scaling, and color transformation of the input image will not affect the image category. Data augmentation refers to the use of translation, scaling, color transformation, etc. to artificially increase the number of training set samples, thereby obtaining more sufficient enhanced data for training the improved FashionNet.

[0106] 1. Image translation, rotation and flip

[0107] Translate, rotate, horizontally, or flip the base image up and down within a certain scale range;

[0108] 2. Random cropping and scaling

[0109] Cut out a piece from the original base image;

[0110] 3. Brightness and color changes

[0111] Perform some transformations on the RGB color space of the base image;

[0112] 4. Noise disturbance

[0113] Add some artificially generated noise to the base image.

[0114] 3. Basic network structure of image classification

[0115] 1. First convolution layer

[0116] Build the first layer of convolution, using 1*1, 3*3 convolution kernels, established with conv2d, and fill the borders with 0. The convolution kernel mainly detects a certain area of ​​the image through a certain kernel block, and then changes the brightness of the pixel according to the brightness difference between the detected pixel and the surrounding pixels. Among them, the role of the 1*1 convolution kernel is to reduce the brightness of the central pixel. If a pixel is dimmer than the pixels around it, it will be further reduced. The role of the 3*3 convolution kernel is to calculate the brightness difference between the central pixel and the surrounding adjacent pixels. There are 5 main parameters of the convolution layer core, which are explained as follows:

[0117] Input: refers to the input image to be convolved, which is a Tensor with a shape of [batch, in_height, in_width, in_channels]. The specific meaning is [the number of images in a batch during training, the image height, the image width, and the number of image channels]. This is a thinking Tensor. In this embodiment of the present invention, the type is set to float32;

[0118] Filter: The convolution kernel in CNN is a Tensor with a shape of [filter_height, filter_width, in_channels, out_channels]. The specific meaning is [height of the convolution kernel, width of the convolution kernel, number of image channels, number of convolution kernels]. The type is the same as input. The third dimension in_channels is the third dimension of the parameter input.

[0119] Strides: The step size in each dimension of the image during convolution. This is a one-dimensional vector. The first and fourth dimensions default to 1, and the third and fourth dimensions are the step sizes for parallel and vertical sliding, respectively.

[0120] Padding: string type, can only be one of 'SAME' and 'VALID'. This value determines different convolution methods.

[0121] Use_cudnn_on_gpu: whether to use cudnn acceleration, the default is True;

[0122] 2. First layer pooling

[0123] In the embodiment of the present invention, the first layer of pooling adopts maximum pooling, followed by a local response normalization layer. The pooling function has four parameters, namely:

[0124] Value: the input to be pooled. The pooling layer is connected after the convolution layer. The input is feature_map and the shape is [batch, height, width, channels].

[0125] Ksizes: The size of the pooling window. Take a thinking vector. In this embodiment of the present invention, it is [1, height, width, 1]. Because we do not want to pool on batches and channels, these two dimensions are set to 1;

[0126] Strides: Similar to convolution, the stride of the window sliding in each dimension. In this embodiment of the present invention, it is [1, stride, stride, 1];

[0127] Padding: Similar to convolution, it can take 'VALID' or 'SAME', and returns a Tensor with the same type and shape as [batch, height, width, channels].

[0128] 3. Second convolution layer

[0129] The convolution kernel of the second convolution layer is set to 5*5, and the activation function is selected as relu;

[0130] 4. Second layer pooling

[0131] The second layer of pooling uses average pooling, followed by a local response normalization layer. An important function of pooling is to help the input data represent approximate invariance. Translation invariance means that when the input data is slightly translated, the output result after pooling does not change.

[0132] 5. Two fully connected layers

[0133] It is followed by two fully connected layers, the first layer is relu activation and the second layer is softmax;

[0134] 4. FashionNet Network Improvements

[0135] The embodiment of the present invention adds the idea of ​​improving FashionNet on the basis of the basic convolutional network, extracts the global features and local features of the product image, and locates the key attributes of the image.

[0136] 1. Global feature layer

[0137] This layer mainly extracts global features such as the outline and position of the entire page image;

[0138] 2. Local feature layer

[0139] This layer mainly pools the key points to obtain the local features of the page image;

[0140] 3. Pooling Shared Layer

[0141] Feature learning is done through text attributes, and the estimated key point positions are used to pool the learned features to obtain predicted features. This layer is performed in an iterative manner to achieve parameter and algorithm sharing, and the same calculation is performed on all decomposed local features;

[0142] 4. Image key point positioning layer

[0143] Predict the location of key points and their visibility. Visibility refers to whether the key points exist (i.e., the image features are included in the page image);

[0144] 5. FashionNet loss function

[0145] The improved FashionNet network mainly uses three loss functions in an iterative manner, namely:

[0146] L2 loss function for key attribute localization;

[0147] Softmax loss function for product label prediction;

[0148] Cross entropy loss function for image attribute prediction;

[0149] 6. The image becomes clearer

[0150] Super-resolution refers to the process of increasing the resolution of an image, thereby making it clearer. If any formatting errors are present in the output image of each layer, super-resolution is used to reconstruct the image. This super-resolution image captures some of the image's information and is much clearer than simply rescaling the image. For example, if a three-channel image contains a single-channel grayscale image, this image is deleted. After deleting the damaged image, the image is scaled to a uniform size and regenerated using super-resolution.

[0151] 5. Parameter debugging

[0152] Learning_rate_decay_type

[0153] Learning rate: In the configuration file of the embodiment of the present invention, it is possible to set whether the learning rate automatically decreases, for example, to use a fixed learning rate or a learning rate that decays proportionally with the number of iterations;

[0154] Optimizer

[0155] Indicates the selected optimizer. The optimizers set in the embodiment of the present invention include stochastic gradient descent (SGD), batch gradient descent (BGD), and adaptive Adam (a small step size for updating high-frequency features and a large step size for updating low-frequency features);

[0156] Weight_decay

[0157] The embodiment of the present invention refers to the selected weight_decay value, which is the quadratic regularization hyperparameter of all parameters in the improved FashionNet;

[0158] Tensorboard dynamic debugging

[0159] When training the improved FashionNet, use tensorboard to visualize the training process. In tensorboard, you can see the loss curve. Observing the loss curve helps debug parameters. When the loss curve is relatively flat and converges slowly, you can consider increasing the learning rate to speed up convergence. If the loss curve fluctuates greatly and cannot converge, it may be because the learning rate is too large. Try to reduce the learning rate appropriately. The improved FashionNet can also train the parameters of different network layers. Through visual comparison, it can be concluded whether it is better to train only a certain layer or train all layers.

[0160] 6. Online Business Applications

[0161] 1. Review image category identification

[0162] For each comment image, identify whether it matches the text attributes of the page. If not, the comment image can be prioritized or filtered.

[0163] 2. Multi-party joint debugging

[0164] After the improved FashionNet is trained, it can be debugged by the algorithm, server, and client to ensure the correctness of business logic and stable program performance.

[0165] 3. Test Evaluation

[0166] Product testers will conduct manual evaluation and online AB test (AB test) data analysis on the improved FashionNet to be launched. The new improved FashionNet can only replace the old improved FashionNet if it meets the online requirements. The online requirements, that is, the offline evaluation indicators are image prediction accuracy or F2 score, where:

[0167] Image prediction accuracy represents the number of classified samples divided by the total number of samples. The higher the accuracy, the better the classification effect.

[0168] The F2score value range is from 0 to 1. The larger the value, the better the model effect.

[0169] TP is a true positive (predicted to be positive but actually positive), FP is a false positive (predicted to be positive but actually negative), FN is a false negative (predicted to be negative but actually positive), TN is a true negative (predicted to be negative but actually negative), α 2 It is a square coefficient with a value range of 0 to 1 and can be determined based on actual conditions.

[0170] 4. Online traffic cut

[0171] The improved fashionNet of the embodiment of the present invention has passed the test evaluation, has been put online and has reduced traffic, and has increased the reference value of reviews for users.

[0172] Figure 4 FIG. 4 is a schematic diagram of main modules of an apparatus for recognizing a page image according to an embodiment of the present invention.

[0173] like Figure 4 As shown, the apparatus 400 for recognizing a page image according to an embodiment of the present invention includes: an acquisition module 401 , an extraction module 402 , a learning module 403 and a determination module 404 .

[0174] in,

[0175] An acquisition module 401 is configured to acquire text attributes of a page text and determine image features corresponding to the text attributes;

[0176] Extraction module 402, for extracting local features of the page image;

[0177] A learning module 403 is configured to perform feature learning on the local features based on the text attributes to obtain prediction features;

[0178] The determination module 404 is configured to determine whether the page image and the page text are consistent with each other based on the similarity between the local feature or the predicted feature and the image feature.

[0179] In this embodiment of the present invention, the extraction module 402 may also be used to:

[0180] Obtaining a pixel matrix of the page image;

[0181] Reading global features from the pixel matrix using a multi-label image classification network architecture;

[0182] The multi-label image classification network architecture is used to pool the key points of the global features to obtain local features.

[0183] In this embodiment of the present invention, the learning module 403 is further configured to:

[0184] Using the multi-label image classification network architecture, the local features are subjected to feature learning based on text attributes to obtain learning features;

[0185] The key points of the learning features are pooled using the multi-label image classification network architecture to obtain predicted features.

[0186] In this embodiment of the present invention, the determining module 404 is further configured to:

[0187] Determining whether the local features and the predicted features are similar to the image features using the multi-label image classification network architecture;

[0188] If the local feature or the predicted feature is similar to the image feature, determining that the page image is consistent with the page text;

[0189] If the local feature or the predicted feature is not similar to the image feature, it is determined that the page image does not match the page text.

[0190] In this embodiment of the present invention, the apparatus 400 further includes a training module (not shown in the figure) configured to:

[0191] Initializing the parameters of the multi-label image classification network architecture using the parameters of the existing image classification basic network structure;

[0192] Extracting a base image marked with a category label, and performing translation, rotation, flipping, scaling, color conversion, or noise perturbation on the base image to generate an enhanced image;

[0193] Adding the basic image and its corresponding category label and the enhanced image and its corresponding category label to a training dataset, and dividing the training dataset into a training set, a validation set, and a test set according to a preset ratio;

[0194] The training set, the validation set, and the test set are read into a memory using a reading thread, and the training set, the validation set, and the test set are read from the memory using a computing thread to train and test the multi-label image classification network architecture.

[0195] It can be seen from the device for recognizing page images according to an embodiment of the present invention that the device adopts the technical means of obtaining the text attributes of the page and determining the image features corresponding to the text attributes; extracting the local features of the page image; performing feature learning based on the text attributes on the local features to obtain the predicted features; and determining whether the page image and the page text are consistent based on the similarity between the local features or the predicted features and the image features. Therefore, the technical problems of the prior art in judging whether the page image and the page text are consistent are overcome, such as the review standards vary from person to person and the standards are inconsistent; the energy of people is limited and the efficiency is low; and the analysis cost is high. The technical effect of automatically analyzing the page image and obtaining the local features and the predicted features from the page image is achieved, so as to accurately and efficiently determine whether the page image and the page text are consistent.

[0196] Figure 5 An exemplary system architecture 500 is shown to which the method or apparatus for recognizing a page image according to an embodiment of the present invention may be applied.

[0197] like Figure 5 As shown, system architecture 500 may include terminal devices 501, 502, 503, a network 504, and a server 505. Network 504 is used to provide a medium for communication links between terminal devices 501, 502, 503 and server 505. Network 504 may include various connection types, such as wired or wireless communication links or fiber optic cables.

[0198] Users can use terminal devices 501, 502, and 503 to interact with server 505 via network 504 to receive or send messages, etc. Various communication client applications can be installed on terminal devices 501, 502, and 503, such as shopping applications, web browser applications, search applications, instant messaging tools, email clients, social platform software, etc.

[0199] The terminal devices 501 , 502 , and 503 may be various electronic devices having a display screen and supporting web browsing, including but not limited to smart phones, tablet computers, laptop computers, and desktop computers.

[0200] Server 505 may be a server that provides various services, such as a backend management server that supports shopping websites browsed by users using terminal devices 501, 502, and 503. The backend management server may analyze and process received data such as product information query requests, and feed back the processing results (e.g., target push information, product information) to the terminal device.

[0201] It should be noted that the method for recognizing a page image provided by the embodiment of the present invention is generally executed by the server 505 , and accordingly, the device for recognizing a page image is generally provided in the server 505 .

[0202] It should be understood that Figure 5 The number of terminal devices, networks and servers in the embodiment is merely illustrative. Any number of terminal devices, networks and servers may be provided as required.

[0203] Reference below Figure 6 , which shows a schematic structural diagram of a computer system 600 of a terminal device suitable for implementing an embodiment of the present invention. Figure 6 The terminal device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present invention.

[0204] like Figure 6 As shown, the computer system 600 includes a central processing unit (CPU) 601, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 602 or a program loaded from a storage unit 608 into a random access memory (RAM) 603. Various programs and data required for the operation of the system 600 are also stored in the RAM 603. The CPU 601, ROM 602, and RAM 603 are connected to each other via a bus 604. An input / output (I / O) interface 605 is also connected to the bus 604.

[0205] The following components are connected to the I / O interface 605: an input section 606 including a keyboard, a mouse, and the like; an output section 607 including devices such as a cathode ray tube (CRT), a liquid crystal display (LCD), and a speaker; a storage section 608 including a hard disk; and a communication section 609 including a network interface card such as a LAN card or a modem. The communication section 609 performs communication processing via a network such as the Internet. A drive 610 is also connected to the I / O interface 605 as needed. A removable medium 611, such as a magnetic disk, an optical disk, a magneto-optical disk, or a semiconductor memory, is installed in the drive 610 as needed, so that computer programs read therefrom can be installed into the storage section 608 as needed.

[0206] In particular, according to the embodiments disclosed in the present invention, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, the embodiments disclosed in the present invention include a computer program product comprising a computer program carried on a computer-readable medium, the computer program comprising program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network via the communication section 609, and / or installed from a removable medium 611. When the computer program is executed by the central processing unit (CPU) 601, the above-mentioned functions defined in the system of the present invention are performed.

[0207] It should be noted that the computer-readable medium described in the present invention can be a computer-readable signal medium or a computer-readable storage medium, or any combination thereof. A computer-readable storage medium can be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of computer-readable storage media can include, but are not limited to, an electrical connection having one or more conductors, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In the present invention, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. In the present invention, a computer-readable signal medium can include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. This propagated data signal can take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium may also be any computer-readable medium other than a computer-readable storage medium that can transmit, propagate, or transport a program for use by or in conjunction with an instruction execution system, apparatus, or device. Program code embodied on a computer-readable medium may be transmitted using any suitable medium, including but not limited to wireless, wireline, optical fiber cable, RF, or any suitable combination thereof.

[0208] The flowcharts and block diagrams in the accompanying drawings illustrate the possible implementation architecture, functions and operations of the systems, methods and computer program products according to various embodiments of the present invention. In this regard, each box in the flowchart or block diagram can represent a module, program segment, or a part of code, and the above-mentioned module, program segment, or a part of code contains one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in an order different from that marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flowchart, and the combination of boxes in the block diagram or flowchart, can be implemented with a dedicated hardware-based system that performs the specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0209] The modules described in the embodiments of the present invention may be implemented in software or hardware. The modules described may also be provided in a processor. For example, they may be described as comprising an acquisition module, an extraction module, a learning module, and a determination module. The names of these modules do not, in some cases, limit the modules themselves. For example, the extraction module may also be described as a "module for extracting local features of a page image."

[0210] As another aspect, the present invention further provides a computer-readable medium, which may be included in the device described in the above embodiment; or may exist independently and not be incorporated into the device. The computer-readable medium carries one or more programs, and when the one or more programs are executed by the device, the device includes the following steps: Step S101: obtaining text attributes of page text and determining image features corresponding to the text attributes; Step S102: extracting local features of the page image; Step S103: performing feature learning based on text attributes on the local features to obtain predicted features; Step S104: determining whether the page image and page text match based on the similarity between the local features or the predicted features and the image features.

[0211] According to the technical solution of the embodiment of the present invention, because it adopts the technical means of obtaining the text attributes of the page and determining the image features corresponding to the text attributes; extracting the local features of the page image; performing feature learning based on the text attributes on the local features to obtain the predicted features; and determining whether the page image and the page text are consistent based on the similarity between the local features or the predicted features and the image features, it overcomes the technical problems of the prior art in judging whether the page image and the page text are consistent, that is, the review standards vary from person to person and the standards are inconsistent; the energy of people is limited and the efficiency is low; and the analysis cost is high, thereby achieving the technical effect of being able to automatically analyze the page image, obtain the local features and the predicted features from the page image, and thus accurately and efficiently determine whether the page image and the page text are consistent.

[0212] The above specific embodiments do not limit the scope of protection of the present invention. Those skilled in the art will appreciate that various modifications, combinations, sub-combinations, and substitutions may occur depending on design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention are intended to be included within the scope of protection of the present invention.

Claims

1. A method for identifying a page image, characterized in that: include: Obtaining text attributes of page text and determining image features corresponding to the text attributes; Obtaining a pixel matrix of the page image; Reading global features from the pixel matrix using a multi-label image classification network architecture; Using the multi-label image classification network architecture, the key points of the global features are pooled to obtain local features; Using the multi-label image classification network architecture, the local features are subjected to feature learning based on text attributes to obtain learning features; The key points of the learning features are pooled using the multi-label image classification network architecture to obtain prediction features; wherein feature learning based on text attributes refers to finding the learning features closest to the text attributes from local features; Determining whether the local features and the predicted features are similar to the image features using the multi-label image classification network architecture; If the local feature or the predicted feature is similar to the image feature, determining that the page image is consistent with the page text; If the local feature and the predicted feature are not similar to the image feature, it is determined that the page image does not match the page text.

2. The method according to claim 1, characterized in that The training process of the multi-label image classification network architecture includes: Initializing the parameters of the multi-label image classification network architecture using the parameters of the existing image classification basic network structure; Extracting a base image marked with a category label, and performing translation, rotation, flipping, scaling, color conversion, or noise perturbation on the base image to generate an enhanced image; Adding the basic image and its corresponding category label and the enhanced image and its corresponding category label to a training dataset, and dividing the training dataset into a training set, a validation set, and a test set according to a preset ratio; The training set, the validation set, and the test set are read into a memory using a reading thread, and the training set, the validation set, and the test set are read from the memory using a computing thread to train and test the multi-label image classification network architecture.

3. A device for recognizing page images, characterized in that: include: An acquisition module, configured to acquire text attributes of page text and determine image features corresponding to the text attributes; An extraction module, used for obtaining a pixel matrix of the page image; Reading global features from the pixel matrix using a multi-label image classification network architecture; Using the multi-label image classification network architecture, the key points of the global features are pooled to obtain local features; A learning module, configured to perform feature learning based on text attributes on the local features using the multi-label image classification network architecture to obtain learning features; The key points of the learning features are pooled using the multi-label image classification network architecture to obtain prediction features; wherein feature learning based on text attributes refers to finding the learning features closest to the text attributes from local features; A determination module is used to use the multi-label image classification network architecture to determine whether the local features and the predicted features are similar to the image features; if the local features or the predicted features are similar to the image features, it is determined that the page image is consistent with the page text; if the local features and the predicted features are not similar to the image features, it is determined that the page image is inconsistent with the page text.

4. The device according to claim 3, characterized in that Also includes training modules for: Initializing the parameters of the multi-label image classification network architecture using the parameters of the existing image classification basic network structure; Extracting a base image marked with a category label, and performing translation, rotation, flipping, scaling, color conversion, or noise perturbation on the base image to generate an enhanced image; Adding the basic image and its corresponding category label and the enhanced image and its corresponding category label to a training dataset, and dividing the training dataset into a training set, a validation set, and a test set according to a preset ratio; The training set, the validation set, and the test set are read into a memory using a reading thread, and the training set, the validation set, and the test set are read from the memory using a computing thread to train and test the multi-label image classification network architecture.

5. An electronic device for recognizing a page image, characterized in that: include: one or more processors; a storage device for storing one or more programs, When the one or more programs are executed by the one or more processors, the one or more processors implement the method according to claim 1 or 2.

6. A computer-readable medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the method according to claim 1 or 2 is implemented.

Images