Illegal advertisement identification method combining text visual features and text content features

Abstract

The present invention relates to computer technology, and specifically to a method for identifying illegal advertisements that combines text visual features and text content features. The method utilizes three methods to convert a color image into three features: the text visual image itself, the text visual image stroke features, and the image text recognition into text features. VGG16 is used to extract image features. RAN is used to extract the stroke order features of Chinese characters. A CNN-encoder / decoder architecture is used to extract the stroke features of Chinese characters and then embeds and transforms them into image features. DenseNet is used for feature learning and classification. Optical Character Recognition (OCR) is used to convert the text portion of an image into text information. Rule matching and review are performed on the text information. A RoFormer neural network model is used to perform feature learning and splitting on sentence-level text information. The method then identifies illegal advertisements. This method can provide strong support for the advertising review and supervision systems of major website platforms.

Description

Technical Field

[0001] The present invention belongs to the field of computer technology, and in particular relates to a method for identifying illegal advertisements by combining text visual features and text content features. Background Art

[0002] With the development of the internet and the advent of the big data era, the total amount of online data storage has experienced explosive growth. Data has become an integral part of our daily lives, and everyone is a recipient of information. While the internet brings convenience, it also carries security risks. One of these risks is that some criminals exploit the complexity and diversity of online resources, as well as netizens' lack of discernment and vulnerability to online misinformation, to spread illegal information and misleading advertisements online, leading some netizens to believe them and, in turn, threaten their property. Currently, many websites use keyword blocking to block text-based advertisements. Because images are more subtle than text, some advertisements embed text within images. As a result, many platforms cannot distinguish whether images uploaded to their servers are advertisements. Users are often harassed by these advertisements while using chat apps or social media, significantly impacting their online experience. Furthermore, the proliferation of advertisements can even provide a breeding ground for online crimes such as gambling and the spread of cults. Therefore, the use of multimodal technology to identify and review illegal advertisements is beneficial in responding to the national "Clean Internet Campaign" and ensuring a healthy online environment for all netizens.

[0003] Existing illegal ad identification technologies primarily target text-based ads, utilizing techniques like dictionary matching and keyword searches to determine if a text ad is illegal. However, with the increasing variety of online resources, more and more ads are being distributed via images or videos. Technologies designed to review image ads typically rely solely on optical character recognition (OCR) to convert the text within the image into text, which is then reviewed. While this technology is effective for regular printed fonts, it is less effective for the artistic fonts or non-Songti fonts used in many current advertising images, making it difficult to identify and review. Summary of the Invention

[0004] In response to the problems existing in the background technology, the present invention provides a multimodal identification method for illegal advertisements that combines text visual features and text content features.

[0005] To solve the above technical problems, the present invention adopts the following technical solution: a method for identifying illegal advertisements by combining text visual features and text content features, comprising the following steps:

[0006] Step 1: Input advertisement images with various formats and visual text to be detected, and perform image preprocessing steps such as noise reduction and image cropping;

[0007] Step 2: Convert the visualized text in the image into three different feature vector inputs: the features of the text visual image itself, the stroke information features of the text visual image, and the features converted from the image text recognition;

[0008] Step 3: Construct a feature classifier based on a neural network, embed and fuse the image features and stroke information features, and then perform feature classification;

[0009] Step 4: Build a text review system based on rule matching and neural networks to review the text features converted by text recognition to determine whether it contains malicious promotional and illegal advertising content;

[0010] Step 5: Combine the classifier and the audit results to determine whether the advertising image to be tested contains any illegal advertising information.

[0011] In the above-mentioned illegal advertisement identification method that combines text visual features and text content features, the implementation of step 3 involves combining the image features and the stroke order information of the visual text into an embedded representation, and then performing feature learning and classification using the DenseNet neural network. The specific process includes:

[0012] Step 3.1: Input the image to be identified and use the vgg16 neural network to extract the image features.

[0013] Step 3.2: Using the RAN component analysis network to analyze the spatial structure and radical components of the visually appealing characters based on the ten most commonly used spatial structures of Chinese characters, extracting the stroke order feature information of the Chinese characters;

[0014] Step 3.3: Convert the image features and stroke order features into embedded representations and perform concat vector splicing. Use the DenseNet neural network to perform feature classification learning to identify whether the visual text in the image contains any illegal parts.

[0015] In the above-mentioned illegal advertisement identification method that combines text visual features and text content features, the implementation of step 4 involves using OCR technology to extract text features from the image, then using rule matching to verify whether the text contains malicious promotional content linked to e-commerce platforms, and then using a classification model based on the deep learning network RoFormer to classify the text features. The specific process includes:

[0016] Step 4.1: Use OCR technology to extract the text content in the image to be detected;

[0017] Step 4.2: Use regular expression matching to conduct a match review of the text content for malicious promotion, including the review of malicious promotion content related to QQ numbers, WeChat numbers, mobile phone numbers, and platform external links;

[0018] Step 4.3 builds the RoFormer deep learning model, directly inputs the text content at the sentence or paragraph granularity into the model for feature extraction and classification, and identifies information containing illegal advertisements.

[0019] In the above-mentioned illegal advertisement identification method combining text visual features and text content features, the implementation of step 5 includes combining the classification results of the two feature learning processes formed by the three modes of steps 3 and 4, performing illegal advertisement review and identification on the image to be detected and obtaining the identification result.

[0020] Compared with the existing technology, the beneficial effects of the present invention are as follows: it reviews and identifies advertisements with visual text information, adds the RoFormer deep learning model to the traditional OCR-based image advertisement recognition, better learns the characteristics of the text, and increases the accuracy of the OCR recognition technology; at the same time, it proposes a fusion feature learning model based on the image characteristics and the visual stroke order characteristics of Chinese characters, and uses the DenseNet network to learn the characteristics of Chinese characters in unconventional fonts (such as artistic fonts) for classification. The proposed multimodal illegal advertisement recognition technology can well handle the recognition of advertisement texts with certain artistic effects embedded in images, which is more in line with the current needs of diversified advertisement review and can provide strong support for the advertisement review and supervision systems of major website platforms. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] Figure 1 This is a flow chart of an embodiment of the present invention. DETAILED DESCRIPTION

[0022] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the embodiments of the present invention. Obviously, the embodiments described are only part of the embodiments of the present invention, not all of the embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments of the present invention without making any creative efforts shall fall within the scope of protection of the present invention.

[0023] It should be noted that, in the absence of conflict, the embodiments of the present invention and the features in the embodiments may be combined with each other.

[0024] The present invention will be further described below with reference to specific examples, but they are not intended to limit the present invention.

[0025] This example enhances conventional OCR technology by adding a neural network feature extraction component. It uses the image encoder vgg16 to directly extract image features, then employs a Random Access Component (RAN) component analysis network to extract the stroke order features of text. These two features are combined and fed into a DenseNet neural network classifier for feature learning and classification. OCR technology is also used to convert image text into text for review, identifying illegal advertisements using a multimodal learning approach.

[0026] This embodiment is realized by the following technical solutions, which is a method for identifying illegal advertisements that combines text visual features and text content features. For text advertisement pictures with visual effects, three methods are used to convert color images into three different features: text visual image features, text visual image stroke information features, and picture text recognition into text features; the vgg16 image encoder is used to extract the overall image features; the component analysis network RAN ​​is used to analyze the radical components and spatial structure of Chinese characters, and Chinese characters are represented as a hierarchical tree structure, and the stroke order features of Chinese characters are analyzed and extracted; at the same time, the CNN-encoder / Decoder architecture is used to extract the stroke features of Chinese characters and then embed and transform them with the image features themselves; a DenseNet-based neural network is used for feature learning and classification; OCR technology is used to extract and identify the text part of the image and convert it into text information; the text information is subject to rule matching and review to determine whether it contains promotional information; a neural network model based on the pre-trained model RoFormer is used to perform feature learning and splitting on text information at the sentence granularity; the multimodal learning and recognition results are summarized to form the final illegal advertisement recognition result. The specific steps are as follows:

[0027] S1, inputs advertisement images containing visual text to be detected in various formats, and performs image preprocessing steps such as noise reduction and image cropping;

[0028] S2 converts the visual text in the image into three different feature vector inputs: the features of the text visual image itself, the stroke information features of the text visual image, and the text features converted from the image text recognition;

[0029] S3, build a feature classifier based on neural network to embed and fuse the image features and stroke information features and then perform feature classification;

[0030] S4: Build a text review system based on rule matching and neural networks to review the text features converted by text recognition to determine whether it contains advertising content such as malicious promotion;

[0031] S5, combines the learning and recognition of the two modalities to obtain the final illegal advertising recognition results.

[0032] Furthermore, the implementation of S3 involves combining the image features and the stroke order information of the visual text into an embedded representation, and performing feature learning and classification through the DenseNet neural network. The specific process includes:

[0033] S31, input the image to be identified and use the vgg16 neural network to extract the image features;

[0034] S32, using the RAN component analysis network to analyze the spatial structure of the text with visual effects based on the ten common spatial structures of Chinese characters, and extract the stroke order features of the Chinese characters;

[0035] S33 converts the image features and stroke order features into embedded representations and then performs feature splicing. The image features are learned through the DenseNet neural network for feature classification.

[0036] Furthermore, the implementation of S4 involves using OCR technology to identify and convert textual content in images, and then employing both a rule-based review method and a RoFormer deep learning model-based method for feature classification. The specific process includes:

[0037] S41, extract text content using OCR technology;

[0038] S42, write rules and use regular expression matching to conduct multi-directional matching review of text content:

[0039] S43 uses the RoFormer neural network model to learn text features and perform feature classification at the sentence level.

[0040] In addition, the implementation of S5 includes the classification results of the two feature learning processes formed by combining the three modes in S3 and S4, and performs illegal advertising review and identification on the detected images to obtain the identification results.

[0041] In practice, due to the variety of advertising images, this example uses advertising images with visually distinctive text (such as an advertisement containing the word "cheapest" in artistic font) as an example to illustrate a method for identifying illegal advertising images. The method includes the following steps:

[0042] 1) Input the advertisement image to be identified;

[0043] 2) Using three different methods, the visual text in the image is converted into three types of feature vector inputs: the features of the text visual image itself, the stroke information features of the text visual image, and the features converted from image text recognition;

[0044] 3) Construct a neural network-based feature classifier to embed and fuse the image features and stroke information features, and then input them into the DenseNet neural network written based on the PyTorch framework for feature learning and classification;

[0045] The component analysis network RAN ​​is used to analyze the radical components and spatial structure of Chinese characters, representing them as a hierarchical tree structure. A CNN-encoder / decoder architecture is also used to extract the stroke features of Chinese characters and then fuse them with the image features in S2 for classification.

[0046] Specifically, the feature fusion process in step 3) includes:

[0047] 3.1) Use the deep learning model VGG16 to extract image features. The three color channels of the color image are represented as a two-dimensional array and input into the VGG16 neural network consisting of 13 convolutional layers and 3 fully connected layers to extract the image features.

[0048] 3.2) Analyze the spatial structure of Chinese characters using a RAN component analysis network to extract the stroke sequence information of the characters. The RAN component analysis network analyzes the radical components and spatial structure of Chinese characters, representing the characters as a hierarchical tree structure. A CNN-encoder / decoder architecture is used to extract the stroke features of the characters, which are then integrated with the image features from step 2) for classification, extracting the stroke sequence information of the characters.

[0049] 3.3) The image features and stroke order features are converted into embedded representations and then concatenated into vectors. These are then input into a DenseNet neural network for feature learning and classification to identify whether the visual text in the image contains any illegal elements.

[0050] 4) First, use Baidu Feijiang Paddle's OCR toolkit to directly extract text features from the image. Then, use rule matching to verify whether the text contains malicious promotional content such as e-commerce platform links. Then, use the classification model based on the deep learning network RoFormer to classify the text features. The specific steps include:

[0051] 4.1) Use OCR technology to extract the text content in the image to be detected;

[0052] 4.2) Use regular expression matching to conduct matching review of text content for malicious promotion, including review of QQ numbers, WeChat accounts, mobile phone numbers, platform external links, and other malicious promotional content;

[0053] 4.3) Build the RoFormer deep learning model and directly input the text content at the sentence or paragraph granularity into the model for feature extraction and classification to identify information containing illegal advertisements.

[0054] 5) Combined with the results of the multimodal learning classifier, it is determined that the image contains illegal advertising information of "cheapest" suspected of false propaganda.

[0055] The above are only preferred embodiments of the present invention and do not limit the implementation mode and protection scope of the present invention. For those skilled in the art, it should be aware that all solutions obtained by equivalent substitutions and obvious changes made using the contents of the present invention specification should be included in the protection scope of the present invention.

Claims

1. A method for identifying illegal advertisements that combines both visual and content features of text, characterized by: The following steps are involved: Step 1: Input advertisement images with various formats and visual text to be detected, and perform image preprocessing steps such as noise reduction and image cropping; Step 2: Convert the visualized text in the image into three different feature vector inputs: the features of the text visual image itself, the stroke information features of the text visual image, and the features converted from the image text recognition; Step 3: Build a neural network-based feature classifier, embed and fuse the image features and stroke information features, and then perform feature classification. The implementation of step 3 involves combining the image features and the stroke order information of the visual text into an embedded representation, and performing feature learning and classification through the DenseNet neural network. The specific process includes: Step 3.1: Input the image to be identified and use the vgg16 neural network to extract the image features. Step 3.2: Using the RAN component analysis network to analyze the spatial structure and radical components of the visually appealing characters based on the ten most commonly used spatial structures of Chinese characters, extracting the stroke order feature information of the Chinese characters; Step 3.3: Convert the image features and stroke order features into embedded representations and perform concat vector splicing. Then, perform feature classification learning using the DenseNet neural network to identify whether the visual text in the image contains any illegal parts. Step 4: Build a text review system based on rule matching and neural networks to review the text features converted from text recognition to determine whether it contains malicious promotional advertising content. Step 4 involves using OCR technology to extract text features from images, then using rule matching to review whether the text contains malicious promotional content linked to e-commerce platforms, and then using a classification model based on the deep learning network RoFormer to classify the text features. The specific process includes: Step 4.1: Use OCR technology to extract the text content in the image to be detected; Step 4.2: Use regular expression matching to conduct a match review of the text content for malicious promotion, including the review of malicious promotion content related to QQ numbers, WeChat numbers, mobile phone numbers, and platform external links; Step 4.3: Build the RoFormer deep learning model and directly input the text content at the sentence or paragraph level into the model for feature extraction and classification to identify information containing illegal advertisements. Step 5: Combine the classifier and the audit results to determine whether the advertising image to be tested contains any illegal advertising information.

2. The method for identifying illegal advertisements by combining text visual features and text content features according to claim 1, characterized in that: The implementation of step 5 includes combining the classification results of the two feature learning processes formed by the three modes of steps 3 and 4, performing illegal advertising review and identification on the image to be detected, and obtaining the identification results.

Images