Webpage advertisement data processing method and device and electronic equipment

Abstract

The application relates to the field of webpage advertisements and specifically discloses a webpage advertisement data processing method, a webpage advertisement data processing system and electronic equipment. The webpage advertisement data processing method comprises the following steps: webpage advertisement data is captured from an Internet website; and the webpage advertisement data is cleaned to process data that does not meet preset requirements in the webpage advertisements, including the following steps: incomplete data in the webpage advertisement data is filtered out; error data in the webpage advertisement data is corrected based on a semantic understanding model to generate corrected correct data; and the webpage advertisement data with the same advertisement principal, the same advertisement publisher and the same advertisement medium is deleted.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of webpage advertisement, and more particularly, to a webpage advertisement data processing method, device and electronic equipment. BACKGROUND

[0002] In recent years, with the development of Internet technology, various types of advertising alliances and e-commerce platforms have emerged, accompanied by the highlighting of Internet advertising problems. Compared with traditional advertising, Internet advertising is convenient to publish, various in form, and accurate in placement. Illegal advertising hides in it and has strong concealment, which is difficult to be found, tracked and investigated by traditional means. Therefore, it is necessary to regulate Internet advertising.

[0003] The first step of regulating Internet advertising is to capture Internet advertising from various websites and clean the data of Internet advertising to build a large database of Internet advertising.

[0004] The task of data cleaning is to filter out data that does not meet the requirements, to ensure the accuracy and standardization of the data delivered to the downstream as much as possible, and to reduce the cost of manual data filtering. For Internet advertising, its data cleaning task has its characteristics, for example, when the captured data has errors for some reason, such as numerical data is input as full-width number characters, there is a carriage return at the end of string data, the date format is incorrect, the date is out of bounds, etc.

[0005] Therefore, an optimized webpage advertisement data processing scheme is needed. SUMMARY

[0006] In order to solve the above technical problems, the present application is proposed. The embodiments of the present application provide a webpage advertisement data processing method, device and electronic equipment, which captures webpage advertisement data from various Internet websites through a crawler tool, and cleans the webpage advertisement data to process the data in the webpage advertisement that does not meet the preset requirements, thereby constructing an Internet advertising monitoring database.

[0007] According to one aspect of the present application, a webpage advertisement data processing method is provided, which includes:

[0008] capturing webpage advertisement data from Internet websites; and

[0009] cleaning the webpage advertisement data to process the data in the webpage advertisement that does not meet the preset requirements, including: filtering out incomplete data in the webpage advertisement data; correcting error data in the webpage advertisement data based on a semantic understanding model to generate corrected correct data; and deleting the webpage advertisement data that has the same advertiser, the same advertising publisher and the same advertising media.

[0010] In the webpage advertisement data processing method, the webpage advertisement data is captured from an Internet website, including: using a crawler tool to capture webpage advertisement data on the Internet website on different regional computer terminals and mobile terminals, wherein the Internet website includes a portal website, an e-commerce website, a video website, a search engine website, a self-owned website, an advertising alliance, a mobile Internet website, a mobile application, and a WeChat public account.

[0011] In the webpage advertisement data processing method, the webpage advertisement data includes advertisement basic information, advertisement publisher information, advertisement principal information, and corresponding screenshot evidence.

[0012] In the webpage advertisement data processing method, the error data in the webpage advertisement data is corrected based on a semantic understanding model to generate corrected correct data, including: performing word segmentation processing on the webpage advertisement data and dividing each word after word segmentation into two categories of text words and format words; converting the text words and the format words into text vectors and format vectors, respectively, by a word embedding unit of the semantic understanding model; inputting an input vector sequence composed of the text vectors and the format vectors into a mask converter unit of the semantic understanding model to obtain a feature vector sequence containing text feature vectors and format feature vectors; encoding the position of each word after word segmentation to obtain position encoding of each word; inputting the position encoding of each word into the mask converter unit to obtain a position vector sequence composed of position vectors; performing feature fusion on the feature vector sequence and the position vector sequence based on a mask conversion structure of the mask converter unit and through a likelihood maximization mechanism to obtain a fused feature vector; and inputting the fused feature vector into a decoder to generate the corrected correct data.

[0013] In the webpage advertisement data processing method, the position of each word after word segmentation is encoded to obtain position encoding of each word, including: performing sinusoidal encoding on the position of each word after word segmentation according to the following formula to obtain the position encoding of each word, wherein the formula of the sinusoidal encoding is: pos i =sin(i / π i / size ), where i is the initial position of each word, and size is the data size.

[0014] In the webpage advertisement data processing method, the feature vector sequence and the position vector sequence are fused based on the mask conversion structure of the mask converter unit and through the likelihood maximization mechanism to obtain a fused feature vector, including: the feature vector sequence and the position vector sequence are fused based on the mask conversion structure of the mask converter unit and through the likelihood maximization mechanism according to the following formula to obtain the fused feature vector, wherein the formula is:

[0015]

[0016] wherein V 1i and V 2i are each corresponding vectors in the sequence of feature vectors and the sequence of position vectors, respectively, is a mean vector of V 1i and V 2i , d is a distance between V 1i and V 2i , and M indicates whether there is a mask in the encoding process of V 1i or V 2i , and a is a hyperparameter.

[0017] In the above webpage advertisement data processing method, if there is a mask, M takes value x, otherwise takes value -x.

[0018] According to another aspect of the present application, there is provided a webpage advertisement data processing apparatus, comprising:

[0019] a data crawling module for crawling webpage advertisement data from an Internet website; and

[0020] a data cleaning module for cleaning the webpage advertisement data to process data in the webpage advertisement that does not meet a preset requirement;

[0021] wherein the data cleaning module comprises:

[0022] an incomplete data processing unit for filtering out incomplete data in the webpage advertisement data;

[0023] an error data correction unit for correcting error data in the webpage advertisement data based on a semantic understanding model to generate corrected correct data; and

[0024] a duplicate data processing unit for deleting the webpage advertisement data with the same advertiser, the same advertisement publisher and the same advertisement medium.

[0025] In the above webpage advertisement data processing apparatus, the data crawling module is further configured to use a crawler tool to crawl webpage advertisement data on the Internet website on different regional computer terminals and mobile terminals, wherein the Internet website comprises a portal website, an e-commerce website, a video website, a search engine website, a self-owned website, an advertisement alliance, a mobile Internet website, a mobile application and a WeChat public account registered in a jurisdiction.

[0026] In the above webpage advertisement data processing apparatus, the webpage advertisement data comprises advertisement basic information, advertisement publisher information, advertiser information and corresponding screenshot evidence.

[0027] In the webpage advertisement data processing apparatus, the error data correction unit comprises: a word segmentation subunit configured to perform word segmentation processing on the webpage advertisement data and divide each word after word segmentation into two categories of text words and format words; a word embedding subunit configured to convert the text words and the format words into text vectors and format vectors respectively through a word embedding unit of the semantic understanding model; a feature mask conversion subunit configured to input an input vector sequence composed of the text vectors and the format vectors into a mask converter unit of the semantic understanding model to obtain a feature vector sequence containing text feature vectors and format feature vectors; a position encoding unit configured to encode positions of each word after word segmentation to obtain position encodings of each word; a position mask conversion subunit configured to input the position encodings of each word into the mask converter unit to obtain a position vector sequence composed of position vectors; a vector fusion subunit configured to perform feature fusion on the feature vector sequence and the position vector sequence based on a mask conversion structure of the mask converter unit and through a likelihood maximization mechanism to obtain a fused feature vector; and a decoding correction subunit configured to input the fused feature vector into a decoder to generate the corrected correct data.

[0028] In the webpage advertisement data processing apparatus, the position encoding unit is further configured to perform sinusoidal encoding on the positions of each word after word segmentation according to the following formula to obtain the position encodings of each word, where the formula of the sinusoidal encoding is: pos i = sin(i / π i / size ), where i is an initial position of each word, and size is a data size.

[0029] In the webpage advertisement data processing apparatus, the vector fusion subunit is further configured to perform feature fusion on the feature vector sequence and the position vector sequence based on the mask conversion structure of the mask converter unit and through a likelihood maximization mechanism according to the following formula to obtain a fused feature vector, where the formula is:

[0030]

[0031] where V 1i and V 2i are each group of corresponding vectors in the feature vector sequence and the position vector sequence, is a mean vector of V 1i and V 2i , d is a distance between V 1i and V 2i , M indicates whether there is a mask in the encoding process of V 1i or V 2i , and a is a hyperparameter.

[0032] In the webpage advertisement data processing apparatus, if there is a mask, M takes value x, otherwise takes value -x.

[0033] According to still another aspect of the present application, an electronic device is provided, comprising: a processor; and a memory having stored therein computer program instructions, which, when executed by the processor, cause the processor to perform the webpage advertisement data processing method as described above.

[0034] According to still another aspect of the present application, a computer readable medium having stored thereon computer program instructions, which, when executed by a processor, cause the processor to perform the webpage advertisement data processing method as described above.

[0035] Compared with the prior art, the webpage advertisement data processing method, apparatus and electronic device provided by the present application can construct an internet advertisement monitoring database by crawling webpage advertisement data from various internet websites by a crawler tool and cleaning the webpage advertisement data to process data in the webpage advertisement that does not meet preset requirements. BRIEF DESCRIPTION OF DRAWINGS

[0036] The above and other objects, features and advantages of the present application will become more apparent from the following detailed description of embodiments of the present application taken in conjunction with the accompanying drawings. The accompanying drawings are intended to provide a further understanding of embodiments of the present application and are incorporated in and constitute a part of this specification, illustrate embodiments of the present application and serve to explain the present application, but do not limit the present application. In the drawings, like reference numerals refer to like elements or steps throughout.

[0037] Figure 1 A network information acquisition flowchart of the webpage advertisement data processing method according to the embodiments of the present application;

[0038] Figure 2 A flowchart of the webpage advertisement data processing method according to the embodiments of the present application;

[0039] Figure 3 A system architecture schematic diagram of correcting error data in the webpage advertisement data based on a semantic understanding model to generate corrected correct data in the webpage advertisement data processing method according to the embodiments of the present application;

[0040] Figure 4 A flowchart of correcting error data in the webpage advertisement data based on a semantic understanding model to generate corrected correct data in the webpage advertisement data processing method according to the embodiments of the present application;

[0041] Figure 5A block diagram of a webpage advertisement data processing apparatus according to an embodiment of the present application;

[0042] Figure 6 A block diagram of an error data correction unit in a webpage advertisement data processing apparatus according to an embodiment of the present application;

[0043] Figure 7 A block diagram of an electronic device according to an embodiment of the present application. DETAILED DESCRIPTION

[0044] Hereinafter, example embodiments according to the present application will be described in detail with reference to the accompanying drawings. It is obvious that the described embodiments are only a part of the embodiments of the present application, and thus should not be used to limit the present application, and it is understood that the present application is not limited by the example embodiments described herein.

[0045] Summary of the application

[0046] As described above, with the increasing prevalence of the influence of the Internet, Internet advertising has become the mainstream form of advertising, but the existing Internet advertising is numerous and increasingly complex, and there are many problems in data collection and processing and data cleaning processing of the advertisements in the Internet webpage. Therefore, a webpage advertisement collection data analysis algorithm is needed to find the one-to-many and many-to-many relationship between the advertisement starting page and the landing page in the massive Internet pages, thereby saving a lot of repeated collection.

[0047] In data collection, the source of the data mainly includes two parts, one of which is based on distributed search engine active collection and mining to crack different regional advertising acquisition problems. That is, by deploying computer rooms in various places across the country, by using a crawler, different regional PC and mobile Internet advertisements can be actively captured, and at the same time, the complete page HTML information and page screenshot information can be saved, which is convenient for subsequent analysis of the advertising content.

[0048] The other part is to crack the Internet advertising of thousands of people through precise simulation of user portraits. That is, through the PC+mobile+IoT three major entry points to build a comprehensive user behavior chain big data, accurate, detailed, seamless switching using multiple terminals, by breaking the boundaries between scenes, simulating user behavior is very accurate, accurately simulating a variety of types of people through machines to capture mobile advertisements, and through continuous machine learning optimization, the monitored APP or website considers that the browsing behavior is the normal behavior of a certain type of person, thereby obtaining the advertising content.

[0049] The process of obtaining network information is as shown in Figure 1 After crawling into the database, the data between the parties in the system construction needs to go through a data cleaning process, as follows:

[0050] The task of data cleaning refers to filtering data that does not meet the requirements, so as to ensure the accuracy and standardization of the data delivered to the downstream and reduce the cost of manually filtering data. The data that does not meet the requirements mainly includes three categories: incomplete data, incorrect data and repeated data. The incomplete data refers to missing basic information of an advertisement, information of an advertisement publisher and information of an advertisement principal, and one or more parts of corresponding screenshot evidence. The incorrect data mainly includes two categories: one is natural results instead of advertisement results; and the other is that the data captured has errors due to some reasons, such as numerical data being input into full-width number characters, a carriage return after string data, incorrect date format, date out of bounds, etc. The repeated data refers to the same advertisement principal, the same advertisement publisher and the same advertisement media.

[0051] That is, after the data is captured and enters the database, the data between various parties in the system construction needs to undergo the data cleaning process. The task of data cleaning is to filter data that does not meet the requirements, so as to ensure the accuracy and standardization of the data delivered to the downstream and reduce the cost of manually filtering data. For example, when the captured data has errors due to some reasons, such as numerical data being input into full-width number characters, a carriage return after string data, incorrect date format, date out of bounds, etc., these all need to be processed by data cleaning. However, since the natural results are not the object of this monitoring, this will cause interference to the subsequent law enforcement; and the incorrect data may cause data transmission errors and program interruption. Therefore, in order to optimize the filtering of incorrect data, improve the accuracy of obtaining data and ensure the stability of the system, it is expected to provide a webpage advertisement data processing scheme.

[0052] Based on this, the application provides a webpage advertisement data processing method, which includes: capturing webpage advertisement data from an Internet website; and cleaning the webpage advertisement data to process data that does not meet preset requirements in the webpage advertisement, including: filtering out incomplete data in the webpage advertisement data; correcting incorrect data in the webpage advertisement data based on a semantic understanding model to generate corrected correct data; and deleting the webpage advertisement data with the same advertisement principal, the same advertisement publisher and the same advertisement media.

[0053] After introducing the basic principle of the application, various non-limiting embodiments of the application will be specifically introduced with reference to the accompanying drawings.

[0054] Exemplary method

[0055] Figure 2 The flowchart of the webpage advertisement data processing method is shown. As shown in FIG. 1, the webpage advertisement data processing method includes the following steps. Figure 2As shown, the webpage advertisement data processing method according to the embodiment of the present application comprises: S110, webpage advertisement data is scraped from an Internet website; and S120, the webpage advertisement data is cleaned to process data in the webpage advertisement that does not meet preset requirements, comprising: S210, incomplete data in the webpage advertisement data is filtered out; S220, error data in the webpage advertisement data is corrected based on a semantic understanding model to generate corrected correct data; and S230, the webpage advertisement data that is the same in an advertiser, the same in an advertisement publisher and the same in an advertisement medium is deleted.

[0056] In step S110, webpage advertisement data is scraped from an Internet website. It should be understood that, in order to find the one-to-many and many-to-many relationship between an advertisement starting page and a landing page in a mass of Internet pages according to an advertisement collection data analysis algorithm, a large amount of repeated collection is saved. Therefore, in the technical solution of the present application, webpage advertisement data needs to be scraped from the Internet first. In a specific example, a crawler tool is used to scrape webpage advertisement data on the Internet website on a computer terminal and a mobile terminal in different regions, wherein the Internet website comprises a portal website, an e-commerce website, a video website, a search engine website, a self-owned website, an advertisement alliance, a mobile Internet website, a mobile application and a WeChat public account registered in a jurisdiction. It is worth mentioning that here, the webpage advertisement data comprises advertisement basic information, advertisement publisher information, advertiser information and corresponding screenshot evidence. The advertisement forms mainly comprise pictures, texts, graphic links, information streams, videos, search engines and accurate placement, etc.

[0057] Specifically, in the embodiment of the present application, different regional advertisements can be cracked based on a distributed search engine active collection and mining, that is, by deploying a computer room in each region of the country, by using a crawler, Internet advertisements on a PC terminal and a mobile terminal in different regions can be actively scraped, and at the same time, the HTML information and the page screenshot information of the complete page can be saved during scraping, facilitating subsequent analysis of the advertisement content.

[0058] In particular, in the embodiment of the present application, Internet advertisements can also be cracked by accurately simulating user portraits, that is, by building a complete user behavior chain big data through PC+mobile+IoT three major portals, accurately and meticulously switching between multiple terminals, by breaking the boundaries between scenes, simulating user behavior very accurately, accurately simulating a plurality of types of people to scrape mobile terminal advertisements through machines, and through continuous machine learning optimization, making the monitored APP or website think that the browsing behavior is the normal behavior of a certain type of person, thereby obtaining the advertisement content.

[0059] In step S120, the webpage advertisement data is cleaned to process data in the webpage advertisement that does not meet preset requirements. It should be understood that, asFigure 1 As shown, after the crawling is completed and enters the database, the data between various parties in the system construction needs to undergo a data cleaning process. The task of data cleaning is to filter out data that does not meet the requirements, and to ensure the accuracy and standardization of the data passed to the downstream as much as possible, so as to reduce the cost of manually filtering data. Among them, the data that does not meet the requirements mainly includes three categories: incomplete data, incorrect data and duplicate data. Therefore, in the embodiment of the present application, the process of cleaning the webpage advertising data to process the data in the webpage advertising that does not meet the preset requirements includes: first, filtering out the incomplete data in the webpage advertising data; then, correcting the incorrect data in the webpage advertising data based on a semantic understanding model to generate corrected correct data; finally, deleting the webpage advertising data with the same advertiser, the same advertising publisher and the same advertising media.

[0060] In step S210, the incomplete data in the webpage advertising data is filtered out. It should be understood that a complete advertising data should include advertising basic information, advertising publisher information and advertiser information, and corresponding screenshot evidence, which are indispensable. Incomplete data refers to the loss of one or more parts. Moreover, there are many factors that cause the data to be incomplete, such as real-time changes of advertising results, intervention of advertising publishers, sudden failure of websites, etc., which will cause data loss. The loss of any field will affect the subsequent law enforcement. Therefore, in order to avoid wasting labor, in the technical solution of the present application, these incomplete data are automatically filtered out to reduce the cost of manual filtering.

[0061] In step S220, the incorrect data in the webpage advertising data is corrected based on a semantic understanding model to generate corrected correct data. It should be understood that the incorrect data mainly includes two categories: one is the natural result instead of the advertising result; the other is the data that is incorrect due to some reason, such as numerical data input into full-width number characters, a carriage return after string data, incorrect date format, date out of bounds, etc. The natural result is not the object of this monitoring, and will interfere with the subsequent law enforcement. The incorrect data may cause data transmission errors and program interruption. For the incorrect data, a certain fault-tolerant mechanism is needed. In addition to continuously optimizing the algorithm and improving the accuracy of obtaining data through machine learning, the data processing capability of the program needs to be improved to reduce the writing of incorrect data and ensure the stability of the system.

[0062] Correspondingly, for example, in one specific example, a deep learning method for semantic understanding is used to process the incorrect data obtained by crawling to generate corrected correct data.

[0063] Specifically, an encoder-decoder based semantic understanding model can be used, considering that in addition to text data errors, format data errors such as date format errors are also included in the data, so a token-based Bert model can be used as a prototype to perform context-based processing from the overall data perspective, thereby automatically correcting the existing errors.

[0064] Specifically, in the embodiments of the present application, the process of correcting the error data in the web advertising data based on the semantic understanding model to generate the corrected correct data includes: first, performing word segmentation processing on the web advertising data and dividing each word after word segmentation into two categories of text words and format words; then, the word embedding unit of the semantic understanding model respectively converts the text words and the format words into text vectors and format vectors; then, input the input vector sequence composed of the text vectors and the format vectors into the mask converter unit of the semantic understanding model to obtain a feature vector sequence containing text feature vectors and format feature vectors; then, encode the position of each word after word segmentation to obtain the position encoding of each word; then, input the position encoding of each word into the mask converter unit to obtain a position vector sequence composed of position vectors; then, based on the mask conversion structure of the mask converter unit and through the likelihood maximization mechanism, the feature vector sequence and the position vector sequence are fused to obtain a fused feature vector; and finally, input the fused feature vector into the decoder to generate the corrected correct data.

[0065] Figure 3 Fig. 1 illustrates a flowchart of the process of correcting the error data in the web advertising data based on the semantic understanding model to generate the corrected correct data in the web advertising data processing method according to the embodiments of the present application. As shown in Fig. 1, the process of correcting the error data in the web advertising data based on the semantic understanding model to generate the corrected correct data includes the following steps. Figure 3As shown, the semantic understanding model is used to correct the error data in the webpage advertising data to generate corrected correct data, including: S310, performing word segmentation processing on the webpage advertising data, and dividing each word after word segmentation into two categories of text words and format words; S320, converting the text words and the format words into text vectors and format vectors respectively through a word embedding unit of the semantic understanding model; S330, inputting an input vector sequence composed of the text vectors and the format vectors into a mask converter unit of the semantic understanding model to obtain a feature vector sequence containing text feature vectors and format feature vectors; S340, encoding the position of each word after word segmentation to obtain position encoding of each word; S350, inputting the position encoding of each word into the mask converter unit to obtain a position vector sequence composed of position vectors; S360, performing feature fusion on the feature vector sequence and the position vector sequence based on a mask conversion structure of the mask converter unit and through a likelihood maximization mechanism to obtain a fused feature vector; and S370, inputting the fused feature vector into a decoder to generate the corrected correct data.

[0066] Figure 4 FIG. 1 illustrates an architecture schematic diagram of correcting error data in webpage advertising data based on a semantic understanding model to generate corrected correct data in a webpage advertising data processing method according to an embodiment of the present application. As shown in FIG. 1, the semantic understanding model is used to correct the error data in the webpage advertising data to generate corrected correct data, including: Figure 4 As shown in the network architecture of the webpage advertising data processing method, first, the obtained webpage advertising data (for example, IN1 as shown in FIG. 2) is subjected to word segmentation processing, and each word after word segmentation is divided into two categories of text words (for example, P1 as shown in FIG. 2) and format words (for example, P2 as shown in FIG. 2); then, the text words and the format words are converted into text vectors (for example, V1 as shown in FIG. 2) and format vectors (for example, V2 as shown in FIG. 2) respectively through a word embedding unit (for example, SUM as shown in FIG. 2) of the semantic understanding model; then, an input vector sequence composed of the text vectors and the format vectors is input into a mask converter unit (for example, MC as shown in FIG. 2) of the semantic understanding model to obtain a feature vector sequence containing text feature vectors (for example, VF1 as shown in FIG. 2) and format feature vectors (for example, VF2 as shown in FIG. 2); then, the position of each word after word segmentation is encoded to obtain position encoding of each word (for example, P2 as shown in FIG. 2); then, the position encoding of each word is input into the mask converter unit to obtain a position vector sequence composed of position vectors (for example, P2 as shown in FIG. 2); then, the feature vector sequence and the position vector sequence are fused based on a mask conversion structure of the mask converter unit and through a likelihood maximization mechanism to obtain a fused feature vector (for example, F as shown in FIG. 2); and finally, the fused feature vector is input into a decoder (for example, DE as shown in FIG. 2) to generate the corrected correct data (for example, OUT as shown in FIG. 2). Figure 4 Figure 4 Figure 4 Figure 4 Figure 4 Figure 4 Figure 4 Figure 4 Figure 4 Figure 4 ​​​​​​​​​the position encoding of each word is input into the mask converter unit (e.g., as shown in MC) to obtain a sequence of position vectors composed of the position encoding of each word (e.g., as shown in V); then, the sequence of feature vectors and the sequence of position vectors are fused based on the mask conversion structure of the mask converter unit and by a likelihood maximization mechanism to obtain a fused feature vector (e.g., as shown in VF); and, finally, the fused feature vector is input into the decoder (e.g., as shown in D) to generate the corrected correct data. Figure 4 the position encoding of each word is input into the mask converter unit (e.g., as shown in MC) to obtain a sequence of position vectors composed of the position encoding of each word (e.g., as shown in V); then, the sequence of feature vectors and the sequence of position vectors are fused based on the mask conversion structure of the mask converter unit and by a likelihood maximization mechanism to obtain a fused feature vector (e.g., as shown in VF); and, finally, the fused feature vector is input into the decoder (e.g., as shown in D) to generate the corrected correct data. Figure 4 the position encoding of each word is input into the mask converter unit (e.g., as shown in MC) to obtain a sequence of position vectors composed of the position encoding of each word (e.g., as shown in V); then, the sequence of feature vectors and the sequence of position vectors are fused based on the mask conversion structure of the mask converter unit and by a likelihood maximization mechanism to obtain a fused feature vector (e.g., as shown in VF); and, finally, the fused feature vector is input into the decoder (e.g., as shown in D) to generate the corrected correct data. Figure 4 the position encoding of each word is input into the mask converter unit (e.g., as shown in MC) to obtain a sequence of position vectors composed of the position encoding of each word (e.g., as shown in V); then, the sequence of feature vectors and the sequence of position vectors are fused based on the mask conversion structure of the mask converter unit and by a likelihood maximization mechanism to obtain a fused feature vector (e.g., as shown in VF); and, finally, the fused feature vector is input into the decoder (e.g., as shown in D) to generate the corrected correct data. Exemplary system the position encoding of each word is input into the mask converter unit (e.g., as shown in MC) to obtain a sequence of position vectors composed of the position encoding of each word (e.g., as shown in V); then, the sequence of feature vectors and the sequence of position vectors are fused based on the mask conversion structure of the mask converter unit and by a likelihood maximization mechanism to obtain a fused feature vector (e.g., as shown in VF); and, finally, the fused feature vector is input into the decoder (e.g., as shown in D) to generate the corrected correct data.

[0067] In steps S310, S320 and S330, the web advertisement data is processed by word segmentation and each word after word segmentation is divided into two categories of text words and format words, and the text words and the format words are respectively converted into text vectors and format vectors by the word embedding unit of the semantic understanding model, and then the input vector sequence composed of the text vectors and the format vectors is input into the mask converter unit of the semantic understanding model to obtain a sequence of feature vectors containing text feature vectors and format feature vectors. As mentioned above, the deep learning method for semantic understanding is used to process the crawled error data to generate corrected correct data. In a specific example, an encoder-decoder based semantic understanding model can be used. That is, first, the crawled web advertisement data is processed by word segmentation, and each word after word segmentation is divided into two categories of text words and format words. Then, the text words and the format words are respectively processed by the word embedding unit of the semantic understanding model to convert them into text vectors and format vectors. Finally, the input vector sequence composed of the text vectors and the format vectors is input into the mask converter unit of the semantic understanding model to obtain a sequence of feature vectors containing the text feature vectors and the format feature vectors. It can be understood that considering that the data contains not only text data errors but also format data errors such as date format errors, in a specific example, a token-based Bert model can be used as a prototype to perform context-based processing from the overall perspective of the data, thereby automatically correcting the existing errors.

[0068] In step S340 and step S350, the position of each word after the word segmentation is encoded to obtain the position encoding of each word, and the position encoding of each word is input into the mask converter unit to obtain a position vector sequence composed of position vectors. It should be understood that, since the web advertisement data may also have character text errors, such as date character boundary error, etc., in the technical solution of the present application, the position of each word after the word segmentation of the advertisement data also needs to be processed, which can also be implemented using the semantic understanding model. That is, first, the position of each word after the word segmentation is encoded, and in a specific example, sine encoding can be performed, that is, pos i = sin(i / π i / size ), where i is the initial position of each word, for example, n words are 1 to n, and size is the data size, for example, n words. Then, the position encoding of each word is input into the mask converter unit to obtain a position vector, thereby composing a position vector sequence. It is worth mentioning that, by using a loop function to perform position encoding, the generalization ability of the mask converter model can be enhanced to a certain extent.

[0069] Specifically, in the embodiment of the present application, the process of encoding the position of each word after the word segmentation to obtain the position encoding of each word includes: encoding the position of each word after the word segmentation with the following formula to obtain the position encoding of each word, wherein the formula of the sine encoding is: pos i = sin(i / π i / size ), where i is the initial position of each word, and size is the data size.

[0070] In step S360 and step S370, the feature vectors sequence and the position vector sequence are fused based on the mask conversion structure of the mask converter unit and through a likelihood maximization mechanism to obtain a fused feature vector, and the fused feature vector is input into a decoder to generate the corrected correct data. That is, first, based on the mask conversion structure of the mask converter unit, the feature vectors sequence and the position vector sequence are fused through a likelihood maximization mechanism to comprehensively consider the text data errors, the format data errors and the character text errors to obtain a fused feature vector f i (i = 1, …, n). Then, the fused feature vector f i is input into a decoder to obtain the corrected correct data.

[0071] Specifically, in the embodiment of the present application, the process of feature fusion of the feature vector sequence and the position vector sequence based on the mask conversion structure of the mask converter unit and through the likelihood maximization mechanism to obtain the fused feature vector, comprises: feature fusion of the feature vector sequence and the position vector sequence based on the mask conversion structure of the mask converter unit and through the likelihood maximization mechanism to obtain the fused feature vector in the following formula, wherein the formula is:

[0072]

[0073] wherein V 1i and V 2i are each group of corresponding vectors in the feature vector sequence and the position vector sequence, is the mean vector of V 1i and V 2i , d is the distance between V 1i and V 2i , and M represents whether there is a mask in the encoding process of V 1i or V 2i , and a is a hyperparameter. It is worth mentioning that if there is a mask, M takes the value x, otherwise takes the value -x.

[0074] In step S230, the web advertisement data of the same advertiser, the same advertisement publisher and the same advertisement media is deleted. It should be understood that the repeated data refers to the same advertiser, the same advertisement publisher and the same advertisement media. Moreover, the reasons for the repeated data may include: different search terms correspond to the same result; the same advertiser displays in the same media multiple times, etc. Regardless of the cause of the repeated data, in the technical solution of the present application, the data is subjected to secondary verification to delete the data with repeated keyword fields, thereby ensuring the uniqueness of the advertisement data displayed by the same advertiser in the same media.

[0075] In summary, the web advertisement data processing method of the embodiment of the present application is illustrated, which crawls web advertisement data from various Internet websites through a crawler tool, and cleans the web advertisement data to process the data in the web advertisement that does not meet the preset requirements. In this way, an Internet advertisement monitoring database is constructed.

[0076] Figure 5

[0077] Figure 5 Fig. 1 illustrates a block diagram of a web advertisement data processing apparatus according to an embodiment of the present application. As shown in Fig. 1, the web advertisement data processing apparatus comprises a web advertisement data crawling unit 10, a web advertisement data cleaning unit 20 and a web advertisement data processing unit 30. Figure 5As shown, the webpage advertisement data processing apparatus 500 according to the embodiment of the present application comprises: a data crawling module 510, configured to crawl webpage advertisement data from an Internet website; and a data cleaning module 520, configured to clean the webpage advertisement data to process data in the webpage advertisement that does not meet preset requirements.

[0078] As shown, Figure 6 The data cleaning module 520 comprises: an incomplete data processing unit 521, configured to filter out incomplete data in the webpage advertisement data; an error data correction unit 522, configured to correct error data in the webpage advertisement data based on a semantic understanding model to generate corrected correct data; and a repeated data processing unit 523, configured to delete the webpage advertisement data with the same advertiser, the same advertisement publisher and the same advertisement medium.

[0079] In one example, in the webpage advertisement data processing apparatus 500 described above, the data crawling module 510 is further configured to: use a crawler tool to crawl webpage advertisement data on the Internet website on a computer terminal and a mobile terminal in different regions, wherein the Internet website comprises a portal website, an e-commerce website, a video website, a search engine website, a self-owned website, an advertisement alliance, a mobile Internet website, a mobile application and a WeChat public account registered in a jurisdiction.

[0080] In one example, in the webpage advertisement data processing apparatus 500 described above, the webpage advertisement data comprises advertisement basic information, advertisement publisher information, advertiser information and corresponding screenshot evidence.

[0081] In one example, in the webpage advertisement data processing apparatus 500 described above, as shown, Figures 1 to 4As shown, the error data correction unit 522 comprises: a word segmentation subunit 5221 configured to perform word segmentation processing on the webpage advertising data and divide each word after word segmentation into two categories of text words and format words; a word embedding subunit 5222 configured to respectively convert the text words and the format words into text vectors and format vectors through a word embedding unit of the semantic understanding model; a feature mask conversion subunit 5223 configured to input an input vector sequence composed of the text vectors and the format vectors into a mask converter unit of the semantic understanding model to obtain a feature vector sequence containing text feature vectors and format feature vectors; a position encoding unit 5224 configured to encode the position of each word after word segmentation to obtain position encoding of each word; a position mask conversion subunit 5225 configured to input the position encoding of each word into the mask converter unit to obtain a position vector sequence composed of position vectors; a vector fusion subunit 5226 configured to perform feature fusion on the feature vector sequence and the position vector sequence based on a mask conversion structure of the mask converter unit and through a likelihood maximization mechanism to obtain a fused feature vector; and a decoding correction subunit 5227 configured to input the fused feature vector into a decoder to generate the corrected correct data.

[0082] In one example, in the webpage advertising data processing apparatus 500 described above, the position encoding unit 5224 is further configured to perform sinusoidal encoding on the position of each word after word segmentation according to the following formula to obtain the position encoding of each word, where the formula of the sinusoidal encoding is: pos i = sin(i / π i / size ), where i is the initial position of each word, and size is the data size.

[0083] In one example, in the webpage advertising data processing apparatus 500 described above, the vector fusion subunit 5226 is further configured to perform feature fusion on the feature vector sequence and the position vector sequence based on a mask conversion structure of the mask converter unit and through a likelihood maximization mechanism according to the following formula to obtain a fused feature vector, where the formula is:

[0084]

[0085] where V 1i and V 2i are each group of corresponding vectors in the feature vector sequence and the position vector sequence, is a mean vector of V 1i and V 2i , d is a distance between V 1i and V 2i , and M represents a distance between V 1i or V2i whether there is a mask in the encoding process of the above webpage advertisement data, and a is a hyper parameter.

[0086] In one example, in the webpage advertisement data processing apparatus 500 described above, if there is a mask, M takes value x, otherwise takes value -x.

[0087] Here, those skilled in the art can understand that the specific functions and operations of each unit and module in the webpage advertisement data processing apparatus 500 described above have been described in detail above with reference to the description of the webpage advertisement data processing method of the above Exemplary electronic device , and therefore the repeated description thereof will be omitted.

[0088] As described above, the webpage advertisement data processing apparatus 500 according to the embodiments of the present application can be implemented in various terminal devices, such as a server of webpage advertisement data processing algorithm, etc. In one example, the webpage advertisement data processing apparatus 500 according to the embodiments of the present application can be integrated into a terminal device as a software module and / or hardware module. For example, the webpage advertisement data processing apparatus 500 can be a software module in the operating system of the terminal device, or can be an application program developed for the terminal device; of course, the webpage advertisement data processing apparatus 500 can also be one of the many hardware modules of the terminal device.

[0089] Alternatively, in another example, the webpage advertisement data processing apparatus 500 and the terminal device can also be separate devices, and the webpage advertisement data processing apparatus 500 can be connected to the terminal device through a wired and / or wireless network, and transmit interactive information in a conventional data format.

[0090] Figure 7

[0091] Next, the electronic device according to the embodiments of the present application will be described with reference to Figure 7 . As shown in Figure 7 , the electronic device 10 includes one or more processors 11 and a memory 12. The processor 11 can be a central processing unit (CPU) or other forms of processing unit having data processing capability and / or instruction execution capability, and can control other components in the electronic device 10 to perform desired functions.

[0092] The memory 12 can include one or more computer program products that can include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. The volatile memory, for example, can include random access memory (RAM), cache memory, and / or the like. The non-volatile memory, for example, can include read only memory (ROM), hard disk, flash memory, and / or the like. One or more computer program instructions can be stored on the computer-readable storage media, and the processor 11 can execute the program instructions to implement the functions of the web advertisement data processing method of the embodiments of the present application described above and / or other desired functions. Various contents such as a feature vector sequence, a fused feature vector, and the like can also be stored in the computer-readable storage media.

[0093] In one example, the electronic device 10 can further include an input system 13 and an output system 14, which are interconnected through a bus system and / or other forms of connection mechanisms (not shown).

[0094] The input system 13 can include, for example, a keyboard, a mouse, and the like.

[0095] The output system 14 can output various information including correct advertisement data and the like to the outside. The output system 14 can include, for example, a display, a speaker, a printer, a communication network and a remote output device connected thereto, and the like.

[0096] Of course, in order to simplify, Exemplary computer program product and computer readable storage medium Only some of the components in the electronic device 10 related to the present application are shown in the block diagram of FIG. 1, and components such as a bus, an input / output interface, and the like are omitted. In addition to this, the electronic device 10 can include any other appropriate components according to the specific application.

[0097] ​

[0098] In addition to the above-described method and device, the embodiments of the present application can be a computer program product including computer program instructions that, when executed by a processor, cause the processor to perform the steps of the functions in the web advertisement data processing method according to the embodiments of the present application described in the above "Exemplary Methods" section of the specification.

[0099] The computer program product can be written in any combination of one or more programming languages, including an object oriented programming language such as Java, C++ or the like, and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The program code can execute entirely on the user's computing device, partly on the user's computing device, as a stand-alone software package, partly on the user's computing device and partly on a remote computing device or entirely on the remote computing device or server. The embodiments of the present application are not limited by the programming languages made use of, as long as they can be used to implement the methods according to the embodiments of the present application.

[0100] In addition, the embodiments of the present application can also be a computer readable storage medium, which stores computer program instructions, and the computer program instructions, when executed by a processor, cause the processor to perform the steps in the web advertisement data processing method described in the "Exemplary Method" section of the specification.

[0101] The computer readable storage medium can take the form of one or more combinations of any type of computer readable medium. The computer readable medium can be a computer readable signal medium or a computer readable storage medium. The computer readable storage medium can include, for example, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, system or device, or any combination thereof. More specific examples (a non-exhaustive list) of the computer readable storage medium include an electrical connection having one or more wires, a portable disc, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.

[0102] The basic principles of the present application are described above in combination with specific embodiments, but it should be pointed out that the advantages, advantages, effects and the like mentioned in the present application are only examples and are not limiting, and these advantages, advantages, effects and the like cannot be considered as the must-have of each embodiment of the present application. In addition, the above-mentioned specific details are only for the purpose of example and for the purpose of understanding, and are not limited to the above-mentioned specific details, and the above-mentioned details do not limit the present application to the must-use of the above-mentioned specific details.

[0103] The block diagrams of the devices, apparatuses, equipment, systems referred to in this application are only illustrative examples and are not intended to require or imply that the connection, arrangement, configuration must be as shown in the block diagrams. As those skilled in the art will recognize, these devices, apparatuses, equipment, systems can be connected, arranged, configured in any manner. Words such as "include," "contain," "have," and the like are open-ended words that are intended to mean "including but not limited to," and are to be used interchangeably. The words "or" and "and" as used herein are intended to mean "and / or," and are to be used interchangeably. The word "such as" as used herein is intended to mean "such as but not limited to," and is to be used interchangeably.

[0104] It is also important to note that the devices, apparatuses, and methods of the present application can be embodied in a variety of forms without departing from the spirit or essential characteristics thereof. These and other various embodiments of the application are disclosed and can be determined by the claims, the specification and the appended drawings. The embodiments described herein are to be considered in a descriptive sense only and not for purposes of limitation. Therefore, the scope of the present application is indicated by the appended claims rather than by the foregoing description, and all changes that come within the meaning and range of equivalents are intended to be embraced therein.

[0105] The above description of disclosed aspects is given for illustrative purposes only and is not intended to limit the scope of the application. As readily apparent to those skilled in the art, many other alternatives, modifications, and variations can be made to the aspects disclosed herein without departing from the scope of the application. It is therefore intended that the application not be limited, to the exact forms disclosed, but should be construed to cover all modifications that are within the scope of these present principles and their equivalents.

[0106] The above description has been given for illustrative purposes only and is not intended to limit the embodiments of the present application to the forms disclosed herein. Although various example aspects and embodiments have been discussed above, those skilled in the art will recognize certain variations, modifications, changes, additions and sub-combinations thereof.

Claims

1. A method for processing webpage advertising data, characterized in that, include: Scraping webpage advertising data from internet websites; as well as The web page advertising data is cleaned to process data that does not meet preset requirements, including: filtering out incomplete data in the web page advertising data; correcting erroneous data in the web page advertising data based on a semantic understanding model to generate corrected data; and deleting web page advertising data with the same advertiser, the same advertising publisher, and the same advertising media. The process of correcting erroneous data in the webpage advertising data based on a semantic understanding model to generate corrected data includes: The webpage advertising data is segmented into words, and each segmented word is divided into two categories: text words and format words. The semantic understanding model uses word embedding units to convert the text words and format words into text vectors and format vectors, respectively. The input vector sequence consisting of the text vector and the format vector is input into the mask converter unit of the semantic understanding model to obtain a feature vector sequence containing text feature vectors and format feature vectors; The position of each word after word segmentation is encoded to obtain the position code of each word; The positional encoding of each word is input into the mask converter unit to obtain a sequence of positional vectors composed of positional vectors; Based on the mask conversion structure of the mask converter unit, and through a likelihood maximization mechanism, feature fusion is performed on the feature vector sequence and the position vector sequence to obtain a fused feature vector, including: Based on the mask conversion structure of the mask converter unit, and through a likelihood maximization mechanism, feature fusion is performed on the feature vector sequence and the position vector sequence using the following formula to obtain a fused feature vector, wherein the formula is: Where V 1i and V 2i These are the vectors corresponding to each group in the feature vector sequence and the position vector sequence, respectively. It is V 1i and V 2i The mean vector of V, d is the mean vector of V 1i With V 2i The distance between them, and M represents the distance in V. 1i or V 2i Does the encoding process contain a mask, and is α a hyperparameter? The fused feature vector is input into the decoder to generate the corrected data.

2. The webpage advertising data processing method according to claim 1, wherein, Data is scraped from webpage advertising on internet websites, including: Web crawling tools are used to capture webpage advertising data from the aforementioned internet websites on computer and mobile devices in different regions. The internet websites include portal websites, e-commerce websites, video websites, search engine websites, proprietary websites, advertising alliances, mobile internet websites, mobile applications, and WeChat official accounts registered in the jurisdiction.

3. The webpage advertising data processing method according to claim 2, wherein, The webpage advertising data includes basic advertising information, advertising publisher information, advertiser information, and corresponding screenshot evidence.

4. The webpage advertising data processing method according to claim 3, wherein, Encoding the position of each word after word segmentation to obtain the position code of each word includes: The position of each word after segmentation is sinusoidally encoded using the following formula to obtain the position code of each word, wherein the formula for sinusoidal encoding is: pos i =sin(i / π) i / size ), where i is the initial position of each word and size is the data size.

5. The webpage advertising data processing method according to claim 4, wherein, If a mask exists, M takes the value x; otherwise, it takes the value -x.

6. A web page advertising data processing device, characterized in that, include: The data scraping module is used to scrape webpage advertising data from internet websites; as well as The data cleaning module is used to clean the web page advertisement data to process data in the web page advertisement that does not meet preset requirements; The data cleaning module includes: An incomplete data processing unit is used to filter out incomplete data in the web page advertising data; An error data correction unit is used to correct erroneous data in the webpage advertising data based on a semantic understanding model to generate corrected data; and A duplicate data processing unit is used to delete web page advertising data that has the same advertiser, the same advertising publisher, and the same advertising media. The error data correction unit includes: The word segmentation subunit is used to segment the web page advertising data into words and divide each segmented word into two categories: text words and format words. The word embedding subunit is used to convert the text words and the format words into text vectors and format vectors respectively through the word embedding unit of the semantic understanding model; The feature masking conversion subunit is used to input the input vector sequence composed of the text vector and the format vector into the masking converter unit of the semantic understanding model to obtain a feature vector sequence containing the text feature vector and the format feature vector; A position encoding unit is used to encode the position of each word after word segmentation to obtain the position encoding of each word; A positional mask conversion subunit is used to input the positional encoding of each word into the mask converter unit to obtain a positional vector sequence composed of positional vectors; A vector fusion subunit is used to perform feature fusion on the feature vector sequence and the position vector sequence based on the mask transformation structure of the mask converter unit and through a likelihood maximization mechanism to obtain a fused feature vector, including: Based on the mask conversion structure of the mask converter unit, and through a likelihood maximization mechanism, feature fusion is performed on the feature vector sequence and the position vector sequence using the following formula to obtain a fused feature vector, wherein the formula is: Where V 1i and V 2i These are the vectors corresponding to each group in the feature vector sequence and the position vector sequence, respectively. It is V 1i and V 2i The mean vector of V, d is the mean vector of V 1i With V 2i The distance between them, and M represents the distance in V. 1i or V 2i Does the encoding process contain a mask, and is α a hyperparameter? A decoding correction subunit is used to input the fused feature vector into the decoder to generate the corrected data.

7. An electronic device, comprising: processor; as well as A memory storing computer program instructions that, when executed by the processor, cause the processor to perform the web page advertising data processing method as described in any one of claims 1-5.

Images