Website type identification method and device, electronic equipment and storage medium

Abstract

The invention relates to the technical field of artificial intelligence, and provides a website type identification method and device, electronic equipment and a storage medium, and the method comprises the steps: obtaining a website address of a to-be-identified website, and obtaining all to-be-classified images in the to-be-identified website based on the website address of the to-be-identified website; based on a feature extraction model, performing feature extraction on each to-be-classified image to obtain an image feature of each to-be-classified image; determining the category of each to-be-classified image based on each text feature in a text feature library and the image feature of each to-be-classified image; and determining the type of the to-be-identified website based on the type of each to-be-classified image. According to the method, the image features and the text features are combined, retrieval type classification judgment of multi-modal features based on image and text description is achieved, and the accuracy of website type recognition can be effectively improved.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a website type identification method, device, electronic device and storage medium. Background Art

[0002] With the rapid development of network information technology, the Internet has become an indispensable part of people's lives. Internet data contains rich information resources and services, which greatly facilitates our work, study and life. With the rapid growth of the number of websites, accurate and efficient type identification of websites has become a major challenge in the field of information management and data analysis. Existing website type identification methods mainly rely on two approaches: manual annotation and automatic classification based on website text content information using artificial intelligence technology.

[0003] Although manual labeling can guarantee the accuracy of recognition to a certain extent, this method is time-consuming, labor-intensive and inefficient, especially when faced with massive amounts of website data, its feasibility is greatly reduced. In addition, manual labeling is also affected by the subjective cognition and experience level of the labeler, which may lead to inconsistency and limitations in the recognition results.

[0004] In order to overcome the limitations of manual labeling, a website type recognition method based on artificial intelligence technology has emerged. This method classifies websites by analyzing the text content information of the website, such as web page titles, descriptions, and text content, using machine learning or deep learning algorithms. However, due to the diversity and complexity of website content and the differences in expression between different websites, the existing website type recognition methods based on text content still face the problem of inaccurate recognition. Therefore, how to improve the accuracy and stability of website type recognition while ensuring recognition efficiency has become a technical problem that needs to be solved urgently. Summary of the invention

[0005] The present invention provides a website type identification method, device, electronic device and storage medium, which are used to solve the defects of low website type identification efficiency and accuracy in related technologies.

[0006] The present invention provides a website type identification method, comprising: Obtaining the URL of a website to be identified, and based on the URL of the website to be identified, obtaining all images to be classified in the website to be identified; Based on a feature extraction model, feature extraction is performed on each image to be classified to obtain image features of each image to be classified, wherein the feature extraction model is obtained by comparative learning and training based on a sample image and a sample text description corresponding to the sample image; Determining the category of each image to be classified based on each text feature in a text feature library and the image features of each image to be classified, wherein the text feature library is constructed based on the sample text description; Based on the categories of the images to be classified, the type of the website to be identified is determined.

[0007] According to a website type identification method provided by the present invention, the sample text description is a hierarchical text description characterizing the category to which the sample image belongs, and the sample text description includes a plurality of text class names for characterizing the category to which the sample image belongs.

[0008] According to a website type identification method provided by the present invention, the step of constructing the text feature library includes: Based on the granularity requirement of website type identification, determining a target text class name from the sample text description; Based on the feature extraction model, feature extraction is performed on the target text class name to obtain text features of the target text class name; The text feature library is constructed based on the target text class name and the text features of the target text class name.

[0009] According to a website type identification method provided by the present invention, the method of determining the category of each image to be classified based on each text feature in a text feature library and the image features of each image to be classified includes: Based on each text feature in the text feature library and an image feature of any image to be classified, calculating the similarity between the image feature of any image to be classified and each text feature; The similarities between the image features of any image to be classified and the text features are compared, and the text class name corresponding to the text feature with the greatest similarity is used as the class of any image to be classified.

[0010] According to a website type identification method provided by the present invention, determining the type of the website to be identified based on the category of each image to be classified includes: Based on the categories of the images to be classified, determining the number of images in each category; Based on the number of images in each category, the type of the website to be identified is determined.

[0011] According to a website type identification method provided by the present invention, the training step of the feature extraction model includes: Acquire an initial model and a sample data set, wherein the sample data set includes a plurality of sample images and at least one sample text description corresponding to each sample image; Based on each sample image and each sample text description in the sample data set, construct a positive sample pair and a negative sample pair; Determining a contrast loss based on the positive sample pair and the negative sample pair; Based on the contrast loss, parameters of the initial model are iterated to obtain the feature extraction model.

[0012] According to a website type identification method provided by the present invention, constructing positive sample pairs and negative sample pairs based on each sample image and each sample text description in the sample data set includes: Based on the image encoding module of the initial model, feature extraction is performed on each sample image to obtain embedded features of each sample image; Based on the text encoding module of the initial model, feature extraction is performed on each sample text description to obtain embedded features of each sample text description; Based on the embedded features of each sample image and the embedded features of each sample text description, positive sample pairs and negative sample pairs are constructed.

[0013] The present invention also provides a website type identification device, comprising: An image acquisition unit, used to acquire the URL of a website to be identified, and based on the URL of the website to be identified, acquire all images to be classified in the website to be identified; A feature extraction unit, configured to extract features of each image to be classified based on a feature extraction model to obtain image features of each image to be classified, wherein the feature extraction model is obtained by comparative learning and training based on a sample image and a sample text description corresponding to the sample image; An image classification unit, used for determining the category of each image to be classified based on each text feature in a text feature library and image features of each image to be classified, wherein the text feature library is constructed based on the sample text description; The type identification unit is used to determine the type of the website to be identified based on the category of each image to be classified.

[0014] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and running on the processor, wherein when the processor executes the computer program, any of the above-mentioned website type identification methods is implemented.

[0015] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the website type identification method described in any one of the above is implemented.

[0016] The present invention also provides a computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the website type identification method described above is implemented.

[0017] The website type identification method, device, electronic device and storage medium provided by the present invention obtain all images to be classified in the website based on the URL of the website to be identified, and extract features for each image to be classified based on the feature extraction model, thereby matching the extracted image features of each image to be classified with each text feature in the text feature library, thereby determining the category of each image to be classified. The combined use of such multimodal features can more accurately classify images, and then accurately determine the type of the website to be identified based on the identified category of each image to be classified. The present invention implements retrieval-based classification judgment based on multimodal features of image and text description, which can not only improve the accuracy of website type identification, but also flexibly adapt to various different usage scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] In order to more clearly illustrate the technical solutions in the present invention or related technologies, the following briefly introduces the drawings required for use in the embodiments or related technical descriptions. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0019] Figure 1 It is a flow chart of the website type identification method provided by the present invention; Figure 2 It is a training schematic diagram of the feature extraction model provided by the present invention; Figure 3 It is a flow chart of a website type identification method based on retrieval-based image classification provided by the present invention; Figure 4 It is a schematic diagram of the process of image analysis provided by the present invention; Figure 5 It is a flowchart of the model training provided by the present invention; Figure 6 It is a flowchart of text feature library registration provided by the present invention; Figure 7 It is a schematic diagram of the process of image classification provided by the present invention; Figure 8 It is a structural schematic diagram of a website type identification device provided by the present invention; Fig. 9 It is a structural schematic diagram of the electronic device provided by the present invention. DETAILED DESCRIPTION

[0020] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0021] With the rapid growth of the number of websites, the classification and management of websites has brought great challenges. If the types of websites can be classified reasonably and quickly, it will not only make it easier for users to find the information they need and improve search efficiency, but also provide advertisers with more accurate delivery channels based on the content and theme of the website, thereby improving the effectiveness of advertising. In addition, it can also facilitate website administrators to manage and maintain website content, improve the quality of the website and user experience. Therefore, accurate classification and identification of website types has important research significance.

[0022] At present, there are two main ways to classify common websites, either by manually marking the type, or by using artificial intelligence methods to classify based on the text content information of the website. However, these methods have certain limitations in efficiency and accuracy. Usually, there are many forms of expression of website information, such as text, images, videos, voice, etc. Among them, images are the most intuitive medium, and using image information to analyze website content is an important means. Therefore, the main research of this invention is to analyze and judge the image content category of the website through image recognition technology, and then comprehensively judge the corresponding website type.

[0023] Using algorithms to determine the category of an image is usually treated as a simple classification problem. Specifically, according to a pre-defined list of types, training images corresponding to each type can be collected, and then a classifier with a specified number of categories can be trained using deep learning. This seems to be a relatively easy problem at first glance, but various problems will be encountered in actual applications. First, there is no unified definition of what types of websites there are, such as whether they are classified according to portal websites, industry websites, entertainment websites, e-commerce websites, etc., or whether they are classified according to portal websites, display websites, marketing websites, transaction websites, service websites, etc., or other classification methods. There is no unified standard. Secondly, in addition to the difficulty in defining classification categories, there may be overlaps between image categories on different websites, resulting in low inter-class separability. This means that even if the images on a website belong to different categories, they may be very similar in visual features, or multiple labels may be required to describe their category information, which increases the challenge of classification. In addition, even scenes of the same semantic class may have huge differences in style, shape, size, and distribution, which makes it difficult to correctly classify images. For example, scenes of city streets may contain different architectural styles, traffic conditions, and crowd density, all of which increase the difficulty of classification.

[0024] When identifying the category of website images, most of the existing technologies treat the image category recognition problem as an ordinary image classification problem. Commonly used deep learning networks can be used as image classification networks, such as ResNet (Residual Network), swin-transformer network, etc., but the classification of website images is more complicated than general classification problems, and the differences between samples of image categories are more complex. Whether good results can be obtained by directly migrating general image classification methods to the field of website image recognition remains to be explored. In addition, some researchers have adopted target detection methods, but this method often detects specific image elements, such as some special tools, website logos, etc., and it is difficult to uniformly cover all image categories, especially some scene images.

[0025] In principle, the recognition of website images is a typical image classification problem, and the specific number of categories can be predefined by the user. The current solutions are basically to label samples, use deep learning technology to train an artificial neural network, and then use the network to classify images. Although the image classification method is simple and fast to implement, it is difficult to define the specific type of website. Most of the images on the website may be multi-label and can be divided into multiple categories. In addition, even images of the same category may require a fine-grained classification due to the large amount of content, rather than simply using a category description. Although some research works have introduced attention mechanisms in the network to improve classification performance, direct classification is still not a good solution and cannot be flexibly adapted to various usage scenarios.

[0026] In general, although image classification technology provides a method for classifying images and then judging the corresponding website type by image category, the effectiveness of its recognition and judgment is difficult to guarantee.

[0027] In view of the above problems, the present invention proposes a website type identification method based on retrieval-based image classification, which jointly completes the image classification task by utilizing the multimodal information of images and texts, and then judges the type of the corresponding website according to the category of the image. This not only improves the accuracy of website type identification, but also can flexibly adapt to different usage scenarios, thereby overcoming the above defects.

[0028] The implementation process of the method is as follows: first, use the image and the corresponding text description of the specified category to form an image-text pair, extract the features of the image and the text description respectively, and use contrastive learning to train a multimodal neural network model so that the image and the text description of its corresponding category are closest; then, in the specific classification process, prepare the text features extracted from the text description of the corresponding category, and when an image needs to be classified, extract the corresponding image features through the neural network model, and then compare the distance with all the prepared text features respectively, and select the text class name corresponding to the text feature with the closest distance as the classification output of the image; finally, according to the classification output result of the image, the type of the corresponding website can be determined. The technical solution of the present invention will be introduced in detail below.

[0029] Figure 1 is a flow chart of the website type identification method provided by the present invention, such as Figure 1 As shown, the method includes: Step 110, obtaining the URL of the website to be identified, and based on the URL of the website to be identified, obtaining all the images to be classified in the website to be identified.

[0030] Specifically, the URL of the website to be identified refers to the URL link that needs to be identified as the type of the website, which can be input by the user, obtained from a database or list, or collected by other means, and the type of the corresponding website has not yet been determined. For example, the URL of the website to be identified can be a directly obtained URL (Uniform Resource Locator) link, or a hyperlink contained in a website, or various links contained in information sent by a text message or an APP, and the embodiment of the present invention does not specifically limit this.

[0031] After obtaining the URL of the website to be identified, all the image information contained in the website can be crawled from the URL. For example, you can send an HTTP request (such as a GET request) to access the website pointed to by the URL, and then parse the HTML or other formats of the website's page content to extract all the image links. These image links point to the images to be classified, which are stored on the website server and can be downloaded locally through a network request or processed directly in memory.

[0032] It is understandable that all the images to be classified in the website to be identified refer to all the image files available for download and processing on the website to be identified. These images may contain key information about the content, theme or function of the website, and therefore can be used as one of the bases for determining the type of the website. In the embodiment of the present invention, these images will be subjected to feature extraction and classification processing to assist in determining the type of the website to be identified.

[0033] Step 120, based on the feature extraction model, feature extraction is performed on each image to be classified to obtain image features of each image to be classified, wherein the feature extraction model is obtained by comparative learning and training based on sample images and sample text descriptions corresponding to the sample images.

[0034] Specifically, after obtaining all the images to be classified in the website, these images can be analyzed one by one to determine the category of each image. Specifically, each image to be classified can first be input into a feature extraction model, and the model is used to extract features of each image to be classified to obtain image features of each image to be classified. Here, the image features of the image to be classified refer to information extracted from the image by the feature extraction model that can represent the content or attributes of the image. These features can be numerical, such as color histograms, texture feature vectors, etc., or more complex representations, such as convolution feature maps in deep learning models, which are not specifically limited in the embodiments of the present invention. In an embodiment of the present invention, image features are used to match text features in a text feature library to determine the category of the image.

[0035] It is understandable that the feature extraction model is a machine learning model whose main function is to extract useful features or representations from input data. In an embodiment of the present invention, the feature extraction model is used to process image data and can extract discriminative image features from the image to be classified. Before executing step 120, the feature extraction model can be pre-trained, which can be implemented by the following steps: first, a large number of sample images and corresponding sample text descriptions are collected and annotated, and a suitable machine learning model is selected as the initial model; then, the sample images and corresponding sample text descriptions are used for comparative learning training, which usually involves inputting the image and text description into the model, extracting image features and text features, and matching the image features and text features to minimize the distance in the feature space or maximize the similarity; according to the loss of comparative learning, the parameters of the model are iteratively updated to optimize the model performance until the predetermined training target (such as accuracy, loss function value, etc.) is reached, and the feature extraction model can be trained.

[0036] The sample images mentioned above refer to image data used to train the feature extraction model, and the sample text descriptions refer to text information corresponding to the sample images, which provide additional information about the image content. Specifically, the sample text descriptions can be hierarchical keyword descriptions from rough to detailed, from major categories to minor categories, such as "entertainment games", "entertainment videos", "entertainment music", "information news", "information education", etc. These descriptions are used to match image features in the comparative learning process to train a feature extraction model that can accurately extract image features.

[0037] It should be understood that contrastive learning is a machine learning technique that trains a model by learning the similarities (or dissimilarities) between data. In an embodiment of the present invention, contrastive learning is used to train a feature extraction model so that it can extract image features that can accurately reflect the relationship between the image and the text description. Specifically, contrastive learning compares the similarity (or distance) between image features and features described in sample text, and adjusts model parameters to maximize the similarity (or minimize the distance), thereby training a feature extraction model that can accurately extract image features.

[0038] Step 130, determining the category of each image to be classified based on each text feature in a text feature library and the image features of each image to be classified, wherein the text feature library is constructed based on the sample text description.

[0039] Specifically, for each image to be classified, after extracting its image features, the image features can be matched with each text feature in the text feature library to determine the category of the image to be classified. Here, the text feature library is a key component that stores text class names and text features corresponding to the text class names. The text feature library can be regarded as a dictionary or mapping table, in which each entry contains a text feature (as a key) and the corresponding text class name (as a value).

[0040] It is understandable that the above text category name can be a complete sample text description, or it can be extracted from the sample text description according to specific task requirements. For example, if a simple large-category classification is to be implemented, "entertainment" and "information" in the sample text description can be selected as text category names, and a text feature library can be constructed based on this. For another example, if a fine-grained classification is to be implemented for entertainment websites, "games", "videos", "music" and the like can be selected from the sample text description as text category names to construct a text feature library.

[0041] After the text feature library is constructed, when performing image classification, the image features of the image to be classified can be similar to each text feature in the text feature library. This can be achieved by calculating the cosine similarity, Euclidean distance and other metrics between two feature vectors; then, the text class name corresponding to the text feature with the greatest similarity can be selected as the category of the image to be classified. Here, each text feature in the text feature library refers to the numerical or vector data extracted from the text class name to represent the text content. These features can be extracted through feature extraction models. They can capture the key information in the text and are distinguishable for different texts.

[0042] Step 140: Determine the type of the website to be identified based on the category of each image to be classified.

[0043] Specifically, after the category of each image to be classified is obtained, the category information of all images to be classified can be integrated to determine the overall type of the corresponding website to be identified.

[0044] For example, according to the categories of all the images to be classified, the number of images corresponding to each category can be determined, so that the type of the website to be identified can be determined according to the category with the largest number of images. For example, if among all the images contained in the website to be identified, the number of images belonging to the category of "game" is the largest, it can be determined that the website is a game type website.

[0045] The method provided by the embodiment of the present invention obtains all the images to be classified in the website based on the URL of the website to be identified, and extracts features for each image to be classified based on the feature extraction model, thereby matching the extracted image features of each image to be classified with each text feature in the text feature library, thereby determining the category of each image to be classified. The combined use of such multimodal features can more accurately classify images, and then accurately determine the type of the website to be identified based on the identified category of each image to be classified. The present invention implements retrieval-based classification judgment based on multimodal features of image and text description, which can not only improve the accuracy of website type identification, but also flexibly adapt to various different usage scenarios.

[0046] Based on any of the above embodiments, the sample text description is a hierarchical text description that characterizes the category to which the sample image belongs, and the sample text description includes a plurality of text class names that are used to characterize the category to which the sample image belongs.

[0047] It should be noted that, in the currently commonly used image classification, each training image is usually given a category label, while the retrieval-based image classification requires one or more specific text descriptions for each image. Conventional retrieval tasks usually use a complete text description, but unlike conventional retrieval tasks, the image classification in the embodiment of the present invention uses a keyword description in a fixed format.

[0048] Specifically, the sample text description is not a simple category label, but a hierarchical text description. Each sample text description may include one or more text class names, which are used together to characterize the category to which the sample image belongs. Specifically, the sample text description may be a hierarchical keyword description from rough to detailed, from large categories to small categories, where each keyword is a text class name. For website image classification, the sample text description may be in the following form: Entertainment Games Entertainment Videos Entertainment Music Information News Information Education In the above examples, each example is a hierarchical text description. For example, "Entertainment Games" is a broad category, and "Games" is a specific subdivision of the "Entertainment" category. Such a hierarchical description helps to understand and classify images more accurately.

[0049] The above are only examples of annotation. In actual applications, they can be further refined according to specific needs. For example, the "game" category can be further subdivided into leisure puzzles, role-playing, etc. The sample text description uses a fixed-format hierarchical text description to describe the category corresponding to the image from coarse to fine, from large to small. The advantages of this annotation method are: (1) When classifying images, simple large-scale classification can be implemented. For example, the keywords "entertainment" and "information" can be retrieved from the sample description text to implement binary classification, and the websites can be classified in a simple form. (2) It can also support fine-grained classification of images. For example, when you need to identify which type of entertainment website an image belongs to, you can retrieve relevant categories based on the description text of the specified category (i.e., entertainment); (3) Solve the problem of partial multi-labeling, because an image can have more than one text description. For example, some website images are composed of several images spliced ​​together, which may meet several descriptions at the same time.

[0050] In addition, by adopting a fixed-format hierarchical text description, when training the model, since the sample text description corresponding to each sample image is similar to a fine-grained classification, the internal differences of the image categories corresponding to each text description are relatively small, making model training easier. In specific applications, since a hierarchical description is used, simple large-category classification or fine-grained classification can be achieved according to specific needs, and multi-label classification can also be achieved, which is more flexible to use.

[0051] Based on any of the above embodiments, Figure 2 It is a training diagram of the feature extraction model provided by the present invention, such as Figure 2 As shown, the training steps of the feature extraction model include: Step 210: Acquire an initial model and a sample data set, where the sample data set includes a plurality of sample images and at least one sample text description corresponding to each sample image.

[0052] Specifically, the initial model refers to the initial state of the model before the training process begins. In an embodiment of the present invention, the initial model can be an untrained neural network model, or it can be a pre-trained model that has been trained on a data set, and the embodiment of the present invention does not specifically limit this. For example, the initial model can adopt the dual encoder model of BEIT-3, and the initial weights can adopt the pre-trained model weights of BEIT-3 based on the retrieval task of coco2014. Subsequently, the initial model can be trained and fine-tuned by collecting and preparing a sample data set to obtain the final feature extraction model. Here, the sample data set can be obtained using an existing annotated data set or by manual annotation.

[0053] Step 220: construct positive sample pairs and negative sample pairs based on each sample image and each sample text description in the sample data set.

[0054] Specifically, after collecting the sample data set, positive sample pairs and negative sample pairs can be constructed based on all sample images and sample text descriptions in the data set. These sample pairs are used to guide the model to learn how to distinguish relevant and irrelevant information. Among them, the positive sample pair is a pair consisting of the embedding features of the sample image (i.e., embedded features) and the embedding features of the sample text description that matches the sample image, and the negative sample pair is a pair consisting of the embedding features of the sample image and the embedding features of the sample text description that does not match the sample image.

[0055] It is understandable that the initial model can adopt a dual encoder model, which uses two independent encoders, one for processing image data and the other for processing text data. In order to construct positive sample pairs and negative sample pairs, the sample images and sample text descriptions in the sample data set can be input into the corresponding encoders respectively to obtain their embedding features, so as to construct positive sample pairs and negative sample pairs according to the embedding features of these images and text descriptions.

[0056] Step 230 : determining a contrast loss based on the positive sample pair and the negative sample pair.

[0057] Specifically, contrastive loss is a loss function used for metric learning, which aims to optimize model parameters so that similar samples are closer in feature space and dissimilar samples are farther away. In an embodiment of the present invention, contrastive loss is used to guide the model to learn how to extract image features more accurately. By applying contrastive loss to fine-tune the model, the matching image embedding features and text description embedding features can be made closer.

[0058] After determining the positive sample pairs and the negative sample pairs, the feature distance (or similarity) between the positive sample pairs and the feature distance (or similarity) between the negative sample pairs can be calculated. The distance can be Euclidean distance, cosine distance, etc. In order to distinguish between positive sample pairs and negative sample pairs, the model usually sets a threshold. According to the feature distance between the positive sample pairs, the feature distance between the negative sample pairs and the threshold, the contrast loss can be calculated.

[0059] Step 240: Based on the contrast loss, perform parameter iteration on the initial model to obtain the feature extraction model.

[0060] Specifically, after calculating the contrastive loss, the model uses the back-propagation algorithm to calculate the gradient of the loss function with respect to the model parameters. Here, the gradient is a vector that indicates the direction in which the loss function value decreases fastest under the current parameter value. By calculating the gradient, the model can understand how to adjust the parameters to reduce the loss. With the gradient information, the model can use optimization algorithms (such as stochastic gradient descent SGD, Adam, etc.) to update the parameters.

[0061] The above steps will be repeated until a certain stopping condition is met. The stopping condition can be reaching a preset number of iterations, the loss function value drops below a certain threshold, or the performance of the model on the validation set no longer improves. During the iteration process, the model will gradually learn how to extract useful features from the sample data, thus obtaining a feature extraction model.

[0062] Based on any of the above embodiments, step 220 specifically includes: Step 221: Based on the image encoding module of the initial model, feature extraction is performed on each sample image to obtain embedded features of each sample image.

[0063] Specifically, the initial model may adopt a dual encoder model, which includes two independent encoders, one for processing image data and the other for processing text data. The encoder for processing image data is an image encoding module, which can convert image data into a numerical representation in a high-dimensional space.

[0064] Through the image coding module, the sample image can be feature extracted to obtain the embedded features of the sample image. Here, the embedded features of the sample image refer to the numerical vector obtained after the image coding module encodes the image. This vector is usually represented in a high-dimensional space, and each dimension represents a certain feature in the image.

[0065] Step 222: Based on the text encoding module of the initial model, feature extraction is performed on each sample text description to obtain embedded features of each sample text description.

[0066] Specifically, the text encoding module refers to an encoder in the model for processing text data, which can convert text data into a numerical representation. Through the text encoding module, feature extraction can be performed on the sample text description to obtain the embedded features of the sample text description. Here, the embedded features of the sample text description refer to the numerical vector obtained after the text encoding module encodes the text description.

[0067] Step 223: construct positive sample pairs and negative sample pairs based on the embedded features of each sample image and the embedded features of each sample text description.

[0068] Specifically, during the model training process, for each sample image, the embedded features of the sample image and the embedded features of its corresponding sample text description can be combined into a positive sample pair, and at the same time, the embedded features of other irrelevant sample text descriptions can be randomly selected to be combined with them to form a negative sample pair.

[0069] By constructing a large number of positive and negative sample pairs to train the model, the model can learn how to distinguish between matching and mismatching feature pairs of images and text descriptions, thereby achieving accurate classification and retrieval of images and text descriptions.

[0070] Based on any of the above embodiments, the steps of constructing the text feature library include: Based on the granularity requirement of website type identification, determining a target text class name from the sample text description; Based on the feature extraction model, feature extraction is performed on the target text class name to obtain text features of the target text class name; The text feature library is constructed based on the target text class name and the text features of the target text class name.

[0071] Specifically, the granularity requirement of website type identification refers to the requirement for the accuracy of website classification (that is, image classification) when identifying the website type, which determines the standard and level of detail used when building the text feature library and performing classification. Specifically, the granularity requirement can be divided into coarse-grained and fine-grained. Under the coarse-grained requirement, we only care about which category the image belongs to, not which specific type it belongs to under the category. For example, images can be divided into two categories: "entertainment" and "information". This classification method is simple and fast, and is suitable for scenarios where websites need to be simply classified. Under the fine-grained requirement, it is necessary to further subdivide the specific type of the image under a certain category, such as further identifying entertainment websites and judging whether they are games, videos, or music websites. This classification method is more accurate and suitable for scenarios where a deep understanding of the specific type of the website is required.

[0072] According to the granularity requirements of website type identification, the target text class name can be retrieved from the sample text description. Specifically, each sample text description can be parsed to identify the keywords or phrases therein, and according to the granularity requirements, the appropriate target text class name can be selected from the parsed keywords or phrases. For example, under coarse-grained requirements, "entertainment" and "information" can be selected as target text class names; under fine-grained requirements, more specific types can be selected, such as "games", "videos", "music", etc. under the "entertainment" type.

[0073] It can be understood that the target text class name refers to the keywords or phrases determined from the sample text description according to the granularity requirements and used to build the text feature library and perform classification. These target text class names represent the classification information of the image and are an important basis for judging the type of the website.

[0074] After selecting the target text class name, these text categories can be input into the feature extraction model in turn to extract the text features of each target text class name one by one. Subsequently, the text feature can be used as the key and the corresponding target text class name as the value to construct a text feature library.

[0075] Based on any of the above embodiments, step 130 specifically includes: Step 131, based on each text feature in the text feature library and the image feature of any image to be classified, calculating the similarity between the image feature of any image to be classified and each text feature; Step 132 , comparing the similarities between the image features of any image to be classified and the text features, and taking the text class name corresponding to the text feature with the greatest similarity as the class of any image to be classified.

[0076] Specifically, after the text feature library is constructed, when performing image classification, the text feature closest to the image feature can be retrieved from the text feature library based on the image feature of the image to be classified, so that the text class name corresponding to the text feature with the greatest similarity (i.e., the closest) can be used as the category of the image to be classified.

[0077] Here, for a certain image to be classified, in order to retrieve the text feature closest to the feature of the image from the text feature library, it can be achieved by calculating the similarity between the feature of the image and each text feature in the text feature library. Specifically, a similarity algorithm can be selected, such as cosine similarity, Euclidean distance, etc., and the selected algorithm can be used to calculate the similarity between the image feature of the image to be classified and each text feature in the text feature library, and each calculated similarity value is stored for subsequent comparison.

[0078] All the calculated similarity values ​​are compared and sorted. According to the sorting results, the value with the greatest similarity and its corresponding text feature can be found. According to the text feature with the greatest similarity, the corresponding text class name is found in the text feature library, so that the text class name can be output as the category of the image to be classified.

[0079] Based on any of the above embodiments, step 140 specifically includes: Step 141, based on the categories of the images to be classified, determining the number of images in each category; Step 142: Determine the type of the website to be identified based on the number of images in each category.

[0080] Specifically, after determining the category of each image to be classified, the categories of all images to be classified can be traversed, and the number of images corresponding to each category can be counted. Subsequently, the images can be sorted according to the number of images corresponding to each category, and the category with the largest number of images is used as the type corresponding to the website to be identified.

[0081] Based on any of the above embodiments, Figure 3 FIG. 1 is a flow chart of a website type identification method based on retrieval-based image classification provided by the present invention. Figure 3 As shown in the figure, the process of using artificial intelligence technology to analyze website types is divided into the following four steps: Step S1, obtaining website information, which may be a URL link obtained directly, a hyperlink from a website, or various links contained in information sent by a text message or an APP; Step S2, crawling all images contained in the website according to the acquired website information; Step S3, analyzing all captured images one by one to determine the category of each image; Step S4, combining the category information of all the images in step S3 to determine the type of the entire website.

[0082] Figure 4 It is a schematic diagram of the process of image analysis provided by the present invention, such as Figure 4 As shown, the embodiment of the present invention adopts a new retrieval-based image classification method to implement the image analysis in step S3. The specific image analysis process is as follows: Step 1: Data preparation; Specifically, sample images and sample text descriptions corresponding to the sample images are collected. The sample text descriptions are hierarchical keyword descriptions from rough to detailed and from major categories to minor categories.

[0083] Step 2: Model training; Figure 5 It is a flow chart of the model training provided by the present invention, such as Figure 5 As shown, the model can adopt the dual encoder model of BEIT-3. The initial weights adopt the pre-trained model weights of BEIT-3 based on the retrieval task of coco2014. Then, by using the collected sample image and sample text description data pairs, the image embedding features and the text description embedding features are obtained respectively through the dual encoder. The model results are fine-tuned by contrastive learning, so that the matching image embedding and text description embedding features are closer, and finally a fine-tuned neural network model is obtained.

[0084] It is understandable that in the dual encoder architecture, two independent encoding modules, namely the image encoding module and the text encoding module, are used to process image and text data respectively so as to map them to the same representation space, thereby facilitating subsequent image-text matching, cross-modal retrieval and other tasks. The image encoding module may include a multi-head attention model and an image feedforward network, wherein the multi-head attention model can help the image encoding module capture the correlation between different regions in the image, and the image feedforward network can further process and transform the output of the multi-head attention model to extract higher-level feature representations. The text encoding module may also include a multi-head attention model and a text feedforward network, wherein the multi-head attention model is used to help the text encoding module understand the semantic relationship between different words or phrases in the text, and the text feedforward network is also used to further process and transform the text output of the multi-head attention model.

[0085] Step 3: Register text feature library; Figure 6 It is a flowchart of the text feature library registration provided by the present invention, such as Figure 6 As shown in the figure, according to the specific task requirements, the text information of the category to be identified (i.e., the text class name) is selected, and the text embedding features of the desired text class name are extracted one by one. Based on the text class name and the extracted text embedding features, a text feature library is formed.

[0086] Step 4: Image classification; Figure 7 It is a schematic diagram of the process of image classification provided by the present invention, such as Figure 7 As shown in the figure, all the images to be classified obtained from the website are input into the neural network model in turn, the embedding features of each image are extracted, and the similarity between the embedding features of each image and all the text embedding features registered in the text feature library is calculated. The text class name corresponding to the text embedding feature with the largest similarity is used as the text label of the final image classification output.

[0087] Based on the above steps 1 to 4, the corresponding text feature library is retrieved through the embedding features of the image, thereby achieving the purpose of distinguishing website types through retrieval-based image classification.

[0088] The website type identification device provided by the present invention is described below. The website type identification device described below and the website type identification method described above can be referred to each other.

[0089] Based on any of the above embodiments, Figure 8is a schematic diagram of the structure of the website type identification device provided by the present invention, such as Figure 8 As shown, the device comprises: The image acquisition unit 810 is used to acquire the URL of the website to be identified, and based on the URL of the website to be identified, acquire all the images to be classified in the website to be identified; A feature extraction unit 820 is used to extract features of each image to be classified based on a feature extraction model to obtain image features of each image to be classified, wherein the feature extraction model is obtained by comparative learning and training based on a sample image and a sample text description corresponding to the sample image; An image classification unit 830, configured to determine the category of each image to be classified based on each text feature in a text feature library and image features of each image to be classified, wherein the text feature library is constructed based on the sample text description; The type identification unit 840 is used to determine the type of the website to be identified based on the category of each image to be classified.

[0090] The device provided by the embodiment of the present invention obtains all the images to be classified in the website based on the URL of the website to be identified, and extracts features for each image to be classified based on the feature extraction model, thereby matching the extracted image features of each image to be classified with each text feature in the text feature library, thereby determining the category of each image to be classified. The combined use of such multimodal features can more accurately classify images, and then accurately determine the type of the website to be identified based on the identified category of each image to be classified. The present invention implements retrieval-based classification judgment based on multimodal features of image and text description, which can not only improve the accuracy of website type identification, but also flexibly adapt to various different usage scenarios.

[0091] Based on any of the above embodiments, the sample text description is a hierarchical text description that characterizes the category to which the sample image belongs, and the sample text description includes a plurality of text class names that are used to characterize the category to which the sample image belongs.

[0092] Based on any of the above embodiments, the device further includes a feature library construction unit, and the feature library construction unit is used to: Based on the granularity requirement of website type identification, determining a target text class name from the sample text description; Based on the feature extraction model, feature extraction is performed on the target text class name to obtain text features of the target text class name; The text feature library is constructed based on the target text class name and the text features of the target text class name.

[0093] Based on any of the above embodiments, the image classification unit 830 is specifically used for: Based on each text feature in the text feature library and an image feature of any image to be classified, calculating the similarity between the image feature of any image to be classified and each text feature; The similarities between the image features of any image to be classified and the text features are compared, and the text class name corresponding to the text feature with the greatest similarity is used as the class of any image to be classified.

[0094] Based on any of the above embodiments, the type identification unit 840 is specifically used for: Based on the categories of the images to be classified, determining the number of images in each category; Based on the number of images in each category, the type of the website to be identified is determined.

[0095] Based on any of the above embodiments, the device further includes a model training unit, and the model training unit is used to: Acquire an initial model and a sample data set, wherein the sample data set includes a plurality of sample images and at least one sample text description corresponding to each sample image; Based on each sample image and each sample text description in the sample data set, construct a positive sample pair and a negative sample pair; Determining a contrast loss based on the positive sample pair and the negative sample pair; Based on the contrast loss, parameters of the initial model are iterated to obtain the feature extraction model.

[0096] Based on any of the above embodiments, the model training unit is specifically used for: Based on the image encoding module of the initial model, feature extraction is performed on each sample image to obtain embedded features of each sample image; Based on the text encoding module of the initial model, feature extraction is performed on each sample text description to obtain embedded features of each sample text description; Based on the embedded features of each sample image and the embedded features of each sample text description, positive sample pairs and negative sample pairs are constructed.

[0097] Fig. 9 An example of a physical structure diagram of an electronic device is shown in FIG. Fig. 9As shown, the electronic device may include: a processor 910, a communication interface 920, a memory 930 and a communication bus 940, wherein the processor 910, the communication interface 920 and the memory 930 communicate with each other through the communication bus 940. The processor 910 may call the logic instructions in the memory 930 to execute the website type identification method, which includes: obtaining the URL of the website to be identified, and based on the URL of the website to be identified, obtaining all the images to be classified in the website to be identified; based on the feature extraction model, extracting features from each image to be classified to obtain the image features of each image to be classified, wherein the feature extraction model is obtained by comparative learning and training based on the sample image and the sample text description corresponding to the sample image; determining the category of each image to be classified based on each text feature in the text feature library and the image features of each image to be classified, wherein the text feature library is constructed based on the sample text description; and determining the type of the website to be identified based on the category of each image to be classified.

[0098] In addition, the logic instructions in the above-mentioned memory 930 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the relevant technology or the part of the technical solution, can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program codes.

[0099] On the other hand, the present invention also provides a computer program product, which includes a computer program. The computer program can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the website type identification method provided by the above methods, which method includes: obtaining the URL of the website to be identified, and based on the URL of the website to be identified, obtaining all the images to be classified in the website to be identified; based on a feature extraction model, performing feature extraction on each image to be classified to obtain image features of each image to be classified, and the feature extraction model is obtained by comparative learning and training based on sample images and sample text descriptions corresponding to the sample images; based on each text feature in a text feature library and the image features of each image to be classified, determining the category of each image to be classified, and the text feature library is constructed based on the sample text description; based on the category of each image to be classified, determining the type of the website to be identified.

[0100] On the other hand, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to execute the website type identification method provided by the above-mentioned methods, the method comprising: obtaining the URL of the website to be identified, and based on the URL of the website to be identified, obtaining all the images to be classified in the website to be identified; based on a feature extraction model, performing feature extraction on each image to be classified to obtain image features of each image to be classified, the feature extraction model being obtained by comparative learning and training based on sample images and sample text descriptions corresponding to the sample images; determining the category of each image to be classified based on each text feature in a text feature library and the image features of each image to be classified, the text feature library being constructed based on the sample text description; determining the type of the website to be identified based on the category of each image to be classified.

[0101] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0102] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the relevant technology can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiment.

[0103] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A website type identification method, characterized in that: include: Obtaining the URL of a website to be identified, and based on the URL of the website to be identified, obtaining all images to be classified in the website to be identified; Based on a feature extraction model, feature extraction is performed on each image to be classified to obtain image features of each image to be classified, wherein the feature extraction model is obtained by comparative learning and training based on a sample image and a sample text description corresponding to the sample image; Determining the category of each image to be classified based on each text feature in a text feature library and the image features of each image to be classified, wherein the text feature library is constructed based on the sample text description; Based on the categories of the images to be classified, the type of the website to be identified is determined.

2. The website type identification method according to claim 1, characterized in that: The sample text description is a hierarchical text description that characterizes the category to which the sample image belongs. The sample text description includes a plurality of text class names that are used to characterize the category to which the sample image belongs.

3. The website type identification method according to claim 1, characterized in that: The steps of constructing the text feature library include: Based on the granularity requirement of website type identification, determining a target text class name from the sample text description; Based on the feature extraction model, feature extraction is performed on the target text class name to obtain text features of the target text class name; The text feature library is constructed based on the target text class name and the text features of the target text class name.

4. The website type identification method according to claim 1, characterized in that: The determining the category of each image to be classified based on each text feature in the text feature library and the image feature of each image to be classified includes: Based on each text feature in the text feature library and an image feature of any image to be classified, calculating the similarity between the image feature of any image to be classified and each text feature; The similarities between the image features of any image to be classified and the text features are compared, and the text class name corresponding to the text feature with the greatest similarity is used as the class of any image to be classified.

5. The website type identification method according to any one of claims 1 to 4, characterized in that: The determining the type of the website to be identified based on the category of each image to be classified includes: Based on the categories of the images to be classified, determining the number of images in each category; Based on the number of images in each category, the type of the website to be identified is determined.

6. The website type identification method according to any one of claims 1 to 4, characterized in that: The training steps of the feature extraction model include: Acquire an initial model and a sample data set, wherein the sample data set includes a plurality of sample images and at least one sample text description corresponding to each sample image; Based on each sample image and each sample text description in the sample data set, construct a positive sample pair and a negative sample pair; Determining a contrast loss based on the positive sample pair and the negative sample pair; Based on the contrast loss, parameters of the initial model are iterated to obtain the feature extraction model.

7. The website type identification method according to claim 6, characterized in that: The constructing positive sample pairs and negative sample pairs based on each sample image and each sample text description in the sample data set includes: Based on the image encoding module of the initial model, feature extraction is performed on each sample image to obtain embedded features of each sample image; Based on the text encoding module of the initial model, feature extraction is performed on each sample text description to obtain embedded features of each sample text description; Based on the embedded features of each sample image and the embedded features of each sample text description, positive sample pairs and negative sample pairs are constructed.

8. A website type identification device, characterized in that: include: An image acquisition unit, used to acquire the URL of a website to be identified, and based on the URL of the website to be identified, acquire all images to be classified in the website to be identified; A feature extraction unit, configured to extract features of each image to be classified based on a feature extraction model to obtain image features of each image to be classified, wherein the feature extraction model is obtained by comparative learning and training based on a sample image and a sample text description corresponding to the sample image; An image classification unit, used for determining the category of each image to be classified based on each text feature in a text feature library and image features of each image to be classified, wherein the text feature library is constructed based on the sample text description; The type identification unit is used to determine the type of the website to be identified based on the category of each image to be classified.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that: When the processor executes the computer program, the website type identification method according to any one of claims 1 to 7 is implemented.

10. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the website type identification method according to any one of claims 1 to 7 is implemented.

Images