Generative AI service website identification method based on multi-modal fusion learning

Through multimodal fusion learning technology, the text, image and code features of the website are extracted and identified using the LightGBM model, which solves the problem that the existing technology is difficult to identify artificial intelligence-generated websites, and achieves higher recognition accuracy and adaptability.

CN118965192APending Publication Date: 2024-11-15NAT COMP NETWORK & INFORMATION SECURITY MANAGEMENT CENT
View PDF 0 Cites 5 Cited by

Patent Information

Application Number
CN202410924472.X
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-07-11
Publication Date
2024-11-15

AI Technical Summary

Technical Problem

Existing website classification technology is difficult to accurately identify and segment subcategories of artificial intelligence-generated websites, especially because their subcategory characteristics are highly similar and diverse content have limited generalization capabilities, and traditional methods cannot adapt to changes in emerging websites.

Method used

Using a multimodal fusion learning method, combining hierarchical structure model with program analysis technology, extract the text, image and code features of the website, and use LightGBM to build an integrated learning model to perform multimodal feature fusion recognition to achieve artificial intelligence generation Accurate identification of service websites.

Benefits of technology

It improves the recognition accuracy and segmentation and classification capabilities of artificial intelligence generation service websites, enhances the generalization performance of the model, and can more accurately identify and adapt to diverse website content.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118965192A_ABST
    Figure CN118965192A_ABST
Patent Text Reader

Abstract

The invention provides a generative AI service website identification method based on multi-modal fusion learning. The method comprises the following steps: inputting original multi-dimensional and multi-modal information of an artificial intelligence generation service website, converting the multi-dimensional information into three feature vectors of a text, a picture and a code of the artificial intelligence service website through a multi-modal feature extraction model, and inputting the three feature vectors into a multi-modal feature fusion recognition model, and outputting a judgment result of whether the artificial intelligence generation service website provides the artificial intelligence service or not. According to the method, more accurate artificial intelligence generation service website recognition is realized through feature fusion of texts, images and code modes of websites, so that the generalization performance of a model on emerging and diversified website data is improved, the generalization ability and adaptability of a website recognition and classification model are enhanced, the limitation of classification of a single data type is broken through, and the classification efficiency is improved. And the generalization ability of the classification system is improved, so that the classification system can adapt to continuously changing and diversified artificial intelligence generation type website contents.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present disclosure relates to the field of artificial intelligence, and more specifically, to a generative AI service website identification method based on multimodal fusion learning. Background Art

[0002] Website classification technology is a technology used to classify web pages or websites according to their content, functions or other characteristics. Existing website classification technologies include the following aspects. 1) Based on keyword classification technology, the web page category is determined by analyzing the keywords in the web page; 2) Based on link structure analysis technology, the website category is predicted by using the link relationship between web pages; 3) Based on machine learning technology, machine learning algorithms are used for classification. The method based on machine learning technology uses a classified set of web pages as training data to train a model, and then applies this model to classify new web pages. Common machine learning algorithms include decision trees, support vector machines, deep learning, etc. Among them, deep learning methods such as convolutional neural networks (CNN), recurrent neural networks (RNN) or Transformer can automatically learn complex patterns of web page content, and the classification effect is usually good.

[0003] In recent years, there are two representative solutions: 1) Research [1] (Dhar A, Chauhan I. A Multi-Category Content Classification Approach Using DistilBERT Transformer and BiLSTM) proposed a novel model that comprehensively utilizes the DistilBERT transformer and bidirectional long short-term memory (BiLSTM) architecture. In the experiment, the model accurately classified website content into ten independent categories, including science and technology, food and recipes, NSFW (not safe for work) content, entertainment, politics, business, news, fitness and health, and blogs; 2) Research [2] (Shiming L, Feiyan M, Lijuan Z, et al. A study on a Multi-Model Fusion Framework for Automatic Website Classification and Recognition) extracted website structural features, registration information, text keywords, connectivity and registration status, and checked all features that may contain abnormal information. Using these features, an automatic classification and recognition model for websites was designed. The model integrates two sub-models, CatBoost and FastText, to realize hybrid text information and data mining to improve the accuracy of the model and the efficiency of website prediction.

[0004] Currently, there is no specific recognition and classification technology for AI-generated websites; and due to the rapid development of AI-generated websites and the high similarity of subcategory features of generated websites, traditional web page classification technology has been greatly challenged.

[0005] Problems with existing technologies:

[0006] Traditional website type identification is based on traditional statistical machine learning or naive multimodal machine learning methods. However, there are many subcategories of AI-generated websites (such as text-generated, image-generated, and audio-generated, etc.), and there is a large gap between different categories, which makes the existing website classification model unable to accurately identify generative websites.

[0007] (1) Most existing technologies classify a single data type. When faced with emerging and diverse website content, their generalization capabilities are limited and they are difficult to adapt to various types of web pages.

[0008] (2) It is impossible to perform segmentation and classification for subcategories with similar features. Traditional website classification technology cannot effectively handle such segmentation and classification requirements due to the coarse granularity of feature extraction. Summary of the invention

[0009] The purpose of the embodiments of the present disclosure is to provide a generative AI service website identification method based on multimodal fusion learning. The main research goal of the present invention is to realize the automatic generation of code summary documents with project granularity through the organic combination of hierarchical-based models and program analysis technology. The specific object of the study is English documents, and the programming language targeted is Java. The summary document is mainly reflected in a short natural language summary, which does not involve complex information such as directory structure, instructions for use, etc. Based on this model, an open source code repository summary automatic generation tool was designed and implemented on the Eclipse IDE in the form of an Eclipse plug-in.

[0010] In a general aspect, a generative AI service website identification method based on multimodal fusion learning is provided, wherein the original multi-dimensional multimodal information of an AI-generated service website is input, and the multi-dimensional information is converted into three feature vectors of text, image, and code of the AI-generated service website through a multimodal feature extraction model, and the three feature vectors are input into a multimodal feature fusion identification model, and a judgment result of whether the AI-generated service website provides AI services is output;

[0011] The multimodal feature extraction model includes two parts: a multidimensional static feature extraction model and a hierarchical multimodal feature fusion model. The multimodal feature extraction model first obtains the original multidimensional multimodal information of the generative AI service website, preprocesses the data of each modality, and then extracts features from text, pictures, and codes by the multidimensional static feature extraction model to obtain image features, text features, and code features; then, a hierarchical multimodal feature fusion model is constructed to align and fuse image features, text features, and code features;

[0012] The multimodal feature fusion recognition model uses LightGBM to build an integrated learning model, uses the features of three modalities to establish multiple decision trees, and together constitutes a model. For each decision tree, the forward distribution algorithm is used for greedy training and learning. Each iteration learns a decision tree to fit the residual of the prediction results of the previous t-1 trees and the true value of the training sample, and finally outputs the recognition result of the artificial intelligence generated service website to determine whether the current website provides artificial intelligence services.

[0013] The multi-dimensional information includes the text, images and web page front-end code information of the website.

[0014] The specific method for extracting features from text, pictures, and codes is as follows:

[0015] For the image information of the generative AI service website, the Vision Transformer model is used to extract features. First, the input image is preprocessed into a suitable size and divided into blocks. Then, each image block is converted into an embedding vector through a linear transformation containing weights, and the position code is added to the embedding vector. Then, the embedding vector containing the position code is input into the Vision Transformer model. Finally, the features are subjected to a global average pooling operation.

[0016] For the text information of the AI-generated service website, the BERT model is used for feature extraction. First, the input text is converted into a token sequence, and then the token is converted into a word embedding vector. The result of word embedding plus position encoding is input into a multi-layer Transformer network, and each layer outputs a new word feature sequence. The output of the last layer of the Transformer network is sent to a linear layer to obtain the deep feature representation of the input text. The prediction function of the BERT model is:

[0017] Where s represents the input token;

[0018] For the code information of the AI-generated service website, feature extraction is performed using Code2Vec technology. First, the source code is compiled to obtain the abstract syntax tree of each function; the path is extracted from the abstract syntax tree; the path embedding is trained through a neural network to represent each path as a continuous vector; the embeddings of all paths are aggregated to obtain the vector representation of each code feature.

[0019] The specific implementation method of the hierarchical multimodal feature fusion model is as follows: for the image feature vector, text feature vector and code feature vector, the feature vectors of all modalities are taken as input, and the feature vectors are passed through the router R with a multi-layer perceptron. I ,R T , activate the corresponding expert network, each expert network is a multi-layer perceptron, expressed as:

[0020]

[0021] Among them, W represents the input of different modal features, which is passed through MLP I Get the output and pass the gated routing network R I The output of is used as the weight of the corresponding expert to obtain the expert The final output score is then concatenated into a multimodal feature sequence F static =[W IMG ,W Text ,W code ], enter the multimodal hybrid MOE layer, and obtain the static fusion feature F of the artificial intelligence generated service website static , use the fused features to add the linear layer, then train the classification layer, and use the trained expert model as the extracted features.

[0022] The specific method of the multimodal feature fusion recognition model to determine whether a website provides artificial intelligence services is:

[0023]

[0024] in represents the category of the current website, F represents the input feature, is a collection of classification and regression trees (CART) such that q(x) is a collection of classification and regression trees (CART) such that Map each input feature x to one of the T leaf nodes in the tree.

[0025] The innovation of the embodiment of the present invention lies in:

[0026] 1) The present invention provides a two-stage training mechanism to improve the training efficiency of the multimodal feature fusion model, wherein in the first stage, each modality is predicted using the output of a hybrid expert network to train the feature extraction model. In the second stage, the feature fusion classification layer is implemented based on LightGBM to provide more accurate classification results, improve classification accuracy and detail, and solve the problem that the existing website classification technology cannot subdivide and classify a subcategory of generative artificial intelligence (such as website generation, text generation, code generation, etc.), thereby achieving more detailed subcategory classification of broad categories, so as to more accurately understand and classify website service types.

[0027] 2) The present invention achieves more accurate AI-generated service website identification by fusing the features of the website's text, image, and code modalities, so as to improve the generalization performance of the model on emerging and diversified website data, enhance the generalization ability and adaptability of the website identification and classification model, break through the limitations of classification of a single data type, and improve the generalization ability of the classification system, so that it can adapt to the ever-changing and diversified AI-generated website content. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] The above and other objects and features of the present disclosure will become more apparent from the following description in conjunction with the accompanying drawings.

[0029] Figure 1 is a schematic diagram showing an architecture diagram of a generative AI service website identification method based on multimodal fusion learning according to an embodiment of the present disclosure;

[0030] Figure 2 is a schematic diagram showing a multimodal feature extraction model architecture diagram according to an embodiment of the present disclosure;

[0031] Figure 3 It is a schematic diagram showing a multimodal feature fusion recognition architecture diagram according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0032] The following specific embodiments are provided to help the reader obtain a comprehensive understanding of the methods, devices and / or systems described herein. However, after understanding the disclosure of the present application, various changes, modifications and equivalents of the methods, devices and / or systems described herein will be clear. For example, the order of operations described herein is only an example and is not limited to those orders set forth herein, but can be changed as will be clear after understanding the disclosure of the present application, except for operations that must occur in a specific order. In addition, for greater clarity and simplicity, the description of features known in the art may be omitted.

[0033] The features described herein can be implemented in different forms and should not be construed as being limited to the examples described herein. Rather, the examples described herein have been provided to illustrate only some of the many possible ways to implement the methods, devices, and / or systems described herein, which will be clear after understanding the disclosure of the present application.

[0034] As used herein, the term "and / or" includes any one of the associated listed items and any combination of any two or more.

[0035] Although terms such as "first", "second", and "third" may be used herein to describe various members, components, regions, layers, or portions, these members, components, regions, layers, or portions should not be limited by these terms. Instead, these terms are only used to distinguish one member, component, region, layer, or portion from another member, component, region, layer, or portion. Therefore, without departing from the teachings of the examples described herein, the first member, first component, first region, first layer, or first portion referred to in the examples may also be referred to as the second member, second component, second region, second layer, or second portion.

[0036] In the specification, when an element (such as a layer, a region or a substrate) is described as being “on”, “connected to” or “coupled to” another element, the element may be directly “on”, “connected to” or “coupled to” another element, or one or more other elements may be present therebetween. Conversely, when an element is described as being “directly on”, “directly connected to” or “directly coupled to” another element, there may be no other elements present therebetween.

[0037] The terms used herein are only used to describe various examples and are not intended to limit the disclosure. Unless the context clearly indicates otherwise, the singular form is also intended to include the plural form. The terms "comprise", "include" and "have" indicate the presence of the described features, quantities, operations, components, elements and / or combinations thereof, but do not exclude the presence or addition of one or more other features, quantities, operations, components, elements and / or combinations thereof.

[0038] Unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as those generally understood by a person of ordinary skill in the art to which the present disclosure belongs after understanding the present disclosure. Unless explicitly defined as such herein, terms (such as those defined in a general dictionary) should be interpreted as having a meaning consistent with their meaning in the context of the relevant art and the present disclosure, and should not be interpreted in an idealized or overly formal manner.

[0039] Furthermore, in the description of examples, when it is considered that a detailed description of a well-known related structure or function would cause vague interpretation of the present disclosure, such a detailed description will be omitted.

[0040] Embodiment 1:

[0041] Figure 1 is a schematic diagram showing a generative AI service website identification method based on multimodal fusion learning according to an embodiment of the present disclosure.

[0042] The method inputs multi-dimensional information of the artificial intelligence generated service website, such as the website's text, images, and web page front-end code information, and outputs the discrimination result of the artificial intelligence generated service website. The overall architecture includes two models: multimodal feature extraction and multimodal feature fusion recognition.

[0043] Multimodal feature extraction model:

[0044] The input of this model is the static feature information of the generative AI service website, including the website's text, images, js / html / css and other information, and the output is three feature vectors of the artificial intelligence service website: text, images, and codes.

[0045] First, the original multimodal information of the generative AI service website is obtained, and the data of each modality is preprocessed. Since the generative AI service website contains a wide variety of modal information, which influence and interact with each other, in order to utilize the information association between the same modality and different modalities, the present invention extracts features from static information such as text, images, and codes of the generative AI service website. The model consists of two parts, namely, the extraction of multi-dimensional static features and hierarchical multimodal feature fusion.

[0046] For the image information of the generative AI service website, such as website screenshots, source images cited in web pages, logos in web pages, etc., the Vision Transformer (ViT) model is used for feature extraction. First, the input image needs to be preprocessed into a suitable size and then divided into blocks. Each block will be regarded as a "word" or "token" processed in the Vision Transformer; then, each image block needs to be converted into an embedding vector, which is achieved through a linear transformation containing weights, similar to word embedding in natural language processing tasks; since the Vision Transformer model does not understand the order of tokens, it is necessary to add position encoding to the embedding vector so that the model can understand the relative position of the image blocks; then the embedding vector containing the position encoding is input into the Vision Transformer model, and the Vision Transformer model extracts features through a self-attention mechanism and a fully connected feedforward network; finally, the features are subjected to a global average pooling operation to obtain the feature representation of the Vision Transformer model, as shown in the following formula, where ViT refers to the Vision Transformer model, P represents the block operation, and I represents the input image information.

[0047] F_i=ViT(P(I))

[0048] For text information generated by artificial intelligence service websites, such as website titles, descriptions, keywords and other text information, the BERT model is used for feature extraction. First, the input text is converted into a token sequence, and then the token is converted into a word embedding vector; since the Transformer model does not have the ability to remember the order of sequence elements, BERT needs to add position encoding so that the model knows the order of words in the sequence; the result of word embedding plus position encoding is input into a multi-layer Transformer network, and each layer will output a new word feature sequence; the output of the last layer of the Transformer network will be sent to a linear layer to obtain a deep feature representation of the input text. The prediction function of the BERT model is as follows, where s represents the input token.

[0049]

[0050] For the code information of the AI-generated service website, such as the HTML, CSS, and JS codes of the website platform, feature extraction is performed using Code2Vec technology. First, the source code is compiled to obtain the abstract syntax tree (AST) of each function; paths are extracted from the AST, which describe the interaction between variables and methods in the code, such as the path from variable declaration to its first use, or the path from method call to method declaration; path embedding is trained through a neural network, and each path is represented as a continuous vector that can capture the semantic information of the path; the embeddings of all paths are aggregated to obtain the vector representation of each code feature.

[0051] Subsequently, a hierarchical multimodal feature fusion model is constructed to align and fuse the image features, text features, and code features. The hierarchical multimodal feature fusion model constructed by the present invention is based on the image feature, text feature, and code feature vectors extracted above. The hierarchical multimodal model takes the feature vectors of all modalities as input, and then passes through the routing selector R with a multilayer perceptron. I ,R T , activate the corresponding expert network, each expert network is a multi-layer perceptron, expressed as:

[0052]

[0053] Among them, W represents the input of different modal features, which is passed through MLP I Get the output and pass the gated routing network R I The output of is used as the weight of the corresponding expert to obtain the expert The final output score. Then multiple single-modal features are concatenated into a multi-modal feature sequence F static =[W IMG ,W Text ,W code ], enter the multimodal hybrid MOE layer, and obtain the static fusion feature F of the artificial intelligence generated service website static , use the fused features to add the linear layer, then train the classification layer, and then use the trained expert model as the extracted features.

[0054] Generative AI service website recognition based on multimodal feature association fusion:

[0055] Use LightGBM to build an integrated learning model, and use the features of three modes to build multiple decision trees (CART trees) to form a model. For each CART tree, the forward distribution algorithm is used for greedy training and learning. Each iteration learns a CART tree to fit the residual between the prediction results of the previous t-1 trees and the true value of the training sample. Finally, the recognition results of the AI-generated service website are output to determine whether the current website provides AI services:

[0056]

[0057] Where F represents the input feature, is a collection of classification and regression trees (CART) such that q(x) is a collection of classification and regression trees (CART) such that Map each input feature x to one of the T leaf nodes in the tree.

[0058] Embodiment 2:

[0059] The embodiment selects a certain artificial intelligence generation service website to specifically explain the entire implementation process. Figure 3 As shown, the present invention provides an artificial intelligence generated service website recognition device based on multimodal fusion learning. The input of this model is multi-dimensional information of the artificial intelligence generated service website, such as text, images, and code information of the website, and the output is the judgment result of the artificial intelligence generated service website.

[0060] First, enter the static feature extraction model. The input of this model is the static feature information of the artificial intelligence service website, including the text, images, codes and other information of all websites, and the output is the static fusion feature vector of the artificial intelligence service website. First, the original multimodal information of the social platform obtained is preprocessed. Data preprocessing includes simple data cleaning, conversion of emoticons to English words (calling interface API), etc., and the preprocessed information is used for representation extraction.

[0061] For image data, Vision Transformer (ViT) is used to obtain the representation of each area in the image. ViT first divides the input image into small blocks of fixed size (called patches), and then linearly embeds these patches into a high-dimensional space. Next, ViT inputs these embedded patches and an additional position embedding into a Transformer model, which consists of multiple self-attention layers and feedforward neural network layers. In the Transformer model, the representation of each patch interacts with the representations of all other patches to capture the global contextual information in the image. Finally, ViT uses the output of the Transformer model (usually the output of the first position, also known as the category embedding) for classification. At this point, we can get the visual features of each patch and the global feature representation of the entire image. According to the rule that each patch corresponds to a vector, it can be split into 9 vectors. Taking the website logo image as an example, if the vector dimension is specified to be 768, then after ViT image preprocessing, the website logo image will be converted into:

[0062] W IMG ={w patch_1 ,w patch_2 ,w patch_3 ,w patch_4 ,w patch_5 ,w patch_6 ,...,w patch_I}

[0063] w patch ={0.03213,0.02312,0.35545,...,...,0.57854,0.45855}

[0064] Where I is the number of regions extracted from this image.

[0065] For text data, use existing large-scale pre-trained models such as BERT to obtain the representation of each text word and convert the input text into a feature vector:

[0066] W Text ={w1,···,w T}

[0067] Where T is the number of words in the text. In addition, it is necessary to add a [CLS] tag in front of each sentence and a [SEP] tag between every two sentences. The final word composition is obtained by adding the feature vector and the position vector, and normalized by the linear layer. For the sentence "…artificial intelligence…", each word in the sentence corresponds to a vector. Taking "artificial" as an example, if the vector dimension is specified to be 512, then after BERT text preprocessing, "artificial" will be converted into:

[0068] w2={0.02251,0.12212,-0.3013,...,...,-0.09135,-0.82932,}

[0069] Similarly, other words will also be converted into vectors of the same dimension. In this way, the final vector of the sentence is the vector representation corresponding to the above formula.

[0070] For code data (such as js, css, etc.), feature extraction is performed using Code2Vec technology. First, the source code is compiled to obtain the abstract syntax tree (AST) of each function; paths are extracted from the AST, which describe the interactions between variables and methods in the code; path embedding is trained through a neural network, and each path is represented as a continuous vector that can capture the semantic information of the path; the embeddings of all paths are aggregated to obtain the vector representation W of each code feature. code .

[0071] Subsequently, a hierarchical multimodal feature fusion model is constructed to align and fuse the image features, text features, and code features. The hierarchical multimodal feature fusion model constructed by the present invention is based on the image feature, text feature, and code feature vectors extracted above. The hierarchical multimodal model takes the feature vectors of all modalities as input, and then passes through the routing selector R with a multilayer perceptron. I ,R T , activate the corresponding expert network, each expert network is a multi-layer perceptron, expressed as:

[0072]

[0073] Among them, W represents the input of different modal features, which is passed through MLP I Get the output and pass the gated routing network R I The output of is used as the weight of the corresponding expert to obtain the expert The final output score. Then multiple single-modal features are concatenated into a multi-modal feature sequence F static =[W IMG ,WText ,W code ], enter the multimodal hybrid MOE layer, and obtain the static fusion feature F of the artificial intelligence generated service website static , train the network.

[0074] Artificial intelligence generation service website recognition model based on multimodal ensemble learning: Use LightGBM (LightGradient Boosting Machine) to build an ensemble learning model. IMG ,W Text ,W code ] Input LightGBM to build multiple decision trees (CART trees). For each CART tree, the forward distribution algorithm is used for greedy training and learning. Each iteration learns a CART tree to fit the residual between the prediction results of the previous t-1 trees and the true value of the training sample. Finally, the recognition results of the artificial intelligence generated service website are output to determine whether the website provides artificial intelligence services:

[0075]

[0076] in represents the category of the current website, F represents the input feature, is a collection of classification and regression trees (CART) such that q(x) is a collection of classification and regression trees (CART) such that Map each input feature x to one of the T leaf nodes in the tree.

[0077] Completing the task of AI-generated service website classification is divided into two stages: training stage and prediction stage. In the training stage: use the above two independent feature extraction models to learn and output static fusion representation and dynamic representation respectively. Then use the integrated learning model to perform AI-generated service website classification learning for the above two types of representations. As the model is trained, the model parameters are continuously optimized to obtain the trained model generation service website classification model. In the prediction stage: input the multi-dimensional information of the website, use the trained model to calculate the corresponding static fusion representation and dynamic representation, and finally output the final website category through the integrated learning AI-generated service website classification model.

[0078] Although some embodiments of the present disclosure have been shown and described, it will be appreciated by those skilled in the art that modifications may be made to the embodiments without departing from the principles and spirit of the present disclosure, the scope of which is defined by the claims and their equivalents.

Claims

1. A generative AI service website identification method based on multimodal fusion learning, characterized in that: Input the original multi-dimensional and multi-modal information of the artificial intelligence generation service website, convert the multi-dimensional information into three feature vectors of text, picture and code of the artificial intelligence service website through the multi-modal feature extraction model, input the three feature vectors into the multi-modal feature fusion recognition model, and output the judgment result of whether the website provides artificial intelligence generation services; The multimodal feature extraction model includes two parts: a multi-dimensional static feature extraction model and a hierarchical multimodal feature fusion model. The multimodal feature extraction model first obtains the original multi-dimensional multimodal information of the generative AI service website, pre-processes the data of each modality, and then extracts features from text, pictures, and codes by the multi-dimensional static feature extraction model to obtain image features, text features, and code features; Then, a hierarchical multimodal feature fusion model is constructed to align and fuse image features, text features, and code features; The multimodal feature fusion recognition model uses LightGBM to build an integrated learning model, uses the features of three modalities to establish multiple decision trees, and together constitutes a model. For each decision tree, the forward distribution algorithm is used for greedy training and learning. Each iteration learns a decision tree to fit the residual of the prediction results of the previous t-1 trees and the true value of the training sample, and finally outputs the recognition result of the artificial intelligence generated service website to determine whether the current website provides artificial intelligence services.

2. The generative AI service website identification method based on multimodal fusion learning as claimed in claim 1, characterized in that: The multi-dimensional information includes the text, images and web page front-end code information of the website.

3. A generative AI service website identification method based on multimodal fusion learning as claimed in claim 2, characterized in that: The specific method for extracting features from text, pictures, and codes is as follows: For the image information of the generative AI service website, the Vision Transformer model is used to extract features. First, the input image is preprocessed into a suitable size and divided into blocks. Then, each image block is converted into an embedding vector through a linear transformation containing weights, and the position code is added to the embedding vector. Then, the embedding vector containing the position code is input into the Vision Transformer model. Finally, the features are subjected to a global average pooling operation. For the text information of the AI-generated service website, the BERT model is used for feature extraction. First, the input text is converted into a token sequence, and then the token is converted into a word embedding vector. The result of word embedding plus position encoding is input into a multi-layer Transformer network, and each layer outputs a new word feature sequence. The output of the last layer of the Transformer network is sent to a linear layer to obtain the deep feature representation of the input text. The prediction function of the BERT model is: Where s represents the input token; For the code information of the AI-generated service website, feature extraction is performed using Code2Vec technology. First, the source code is compiled to obtain the abstract syntax tree of each function; the path is extracted from the abstract syntax tree; the path embedding is trained through a neural network to represent each path as a continuous vector; the embeddings of all paths are aggregated to obtain the vector representation of each code feature.

4. A generative AI service website identification method based on multimodal fusion learning as claimed in claim 3, characterized in that: The specific implementation method of the hierarchical multimodal feature fusion model is as follows: for the image feature vector, text feature vector and code feature vector, the feature vectors of all modalities are taken as input, and the feature vectors are passed through the router R with a multi-layer perceptron. I ,R T , activate the corresponding expert network, each expert network is a multi-layer perceptron, expressed as: Among them, W represents the input of different modal features, which is passed through MLP I Get the output and pass the gated routing network R I The output of is used as the weight of the corresponding expert to obtain the expert The final output score is then concatenated into a multimodal feature sequence F static =[W IMG ,W Text ,W code ], enter the multimodal hybrid MOE layer, and obtain the static fusion feature F of the artificial intelligence generated service website static , use the fused features to add the linear layer, then train the classification layer, and use the trained expert model as the extracted features.

5. A generative AI service website identification method based on multimodal fusion learning as claimed in claim 4, characterized in that: The specific method of the multimodal feature fusion recognition model to determine whether a website provides artificial intelligence services is: in represents the category of the current website, F represents the input feature, is a collection of classification and regression trees (CART) such that q(x) is a collection of classification and regression trees (CART) such that Map each input feature x to one of the T leaf nodes in the tree.

Citation Information

Cited By

  • Illegal website gang identification method and system based on multi-dimensional feature fusion

    CN119449481A

  • Website type identification method and device, electronic equipment and storage medium

    CN119598054A

  • Universal two-stage webpage information extraction method and system based on multi-dimensional features

    CN119646331A

  • Dynamic ship classification and identification method based on multi-modal mass change

    CN119810564A

  • Dynamic ship classification and recognition method based on multimodal mass changes

    CN119810564B