Text data preprocessing method and device, electronic equipment and medium

Abstract

The present disclosure provides a text data preprocessing method and device for webpage advertisement identification, an electronic device and a medium, the computer technical field, and particularly relates to the computer network technology, the text data mining and the natural language processing technical field. The implementation scheme is: obtaining a resource text of a webpage, the resource text including a text for obtaining a webpage resource; performing word segmentation processing on the resource text to obtain a plurality of first segmented words; performing cutting processing on each first segmented word in the plurality of first segmented words to obtain a plurality of second segmented words; obtaining a semantic feature vector of the plurality of second segmented words; based on the semantic feature vector of the plurality of second segmented words, obtaining a plurality of central semantic vectors; and in response to receiving an identification result that the webpage resource corresponding to the resource text is an advertisement, based on the received instruction of shielding the resource text, shielding the advertisement.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to the technical field of computer, in particular, to the technical field of computer network, text data mining and natural language processing, and specifically, to a method and device for text data preprocessing for webpage advertisement identification, electronic equipment, computer readable storage medium and computer program product. BACKGROUND

[0002] Text data mining is a technology of processing unstructured data into structured data by using natural language processing (NLP) and the like, and the processed structured data can usually be used as input of a machine learning and deep learning model.

[0003] The methods described in this section can not necessarily be the methods previously conceived or used. Unless otherwise indicated, nothing in this section should be assumed to be prior art merely because of its inclusion in this section. Similarly, issues mentioned in this section should not be assumed to have been admitted to be prior art in any jurisdiction unless otherwise indicated. SUMMARY

[0004] The present disclosure provides a method and device for text data preprocessing for webpage advertisement identification, electronic equipment, computer readable storage medium and computer program product.

[0005] According to an aspect of the present disclosure, a method for text data preprocessing for webpage advertisement identification is provided, comprising: obtaining resource text of a webpage, the resource text comprising text for obtaining webpage resources; performing word segmentation processing on the resource text to obtain a plurality of first segmented words; performing cutting processing on each first segmented word in the plurality of first segmented words to obtain a plurality of second segmented words, wherein each second segmented word in the plurality of second segmented words is a segment of continuous characters in its corresponding first segmented word, and the length of each second segmented word in the plurality of second segmented words is less than the length of its corresponding first segmented word; obtaining semantic feature vectors of the plurality of second segmented words; obtaining a plurality of center semantic vectors based on the semantic feature vectors of the plurality of second segmented words, wherein the plurality of center semantic vectors are used to identify whether the webpage resources obtained by the resource text are advertisements; and in response to receiving an identification result that the webpage resources corresponding to the resource text are advertisements, shielding the advertisements based on a received instruction to shield the resource text.

[0006] According to another aspect of the present disclosure, there is provided a text data preprocessing apparatus for webpage advertisement identification, comprising: a first obtaining unit configured to obtain a resource text of a webpage, the resource text comprising text used to obtain a webpage resource; a word segmentation unit configured to perform word segmentation processing on the resource text to obtain a plurality of first segmented words; a word cutting unit configured to perform word cutting processing on each of the plurality of first segmented words to obtain a plurality of second segmented words, wherein each of the plurality of second segmented words is a segment of continuous characters in its corresponding first segmented word, and the length of each of the plurality of second segmented words is less than the length of its corresponding first segmented word; a second obtaining unit configured to obtain semantic feature vectors of the plurality of second segmented words; a third obtaining unit configured to obtain a plurality of center semantic vectors based on the semantic feature vectors of the plurality of second segmented words, wherein the plurality of center semantic vectors are used to identify whether the webpage resource obtained by the resource text is an advertisement; and a shielding unit configured to, in response to receiving an identification result that the webpage resource corresponding to the resource text is an advertisement, shield the advertisement based on a received instruction to shield the resource text.

[0007] According to another aspect of the present disclosure, there is provided an electronic device, comprising: at least one processor; and a memory communicatively connected with the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the above-described text data preprocessing method for webpage advertisement identification.

[0008] According to another aspect of the present disclosure, there is provided a non-transitory computer readable storage medium storing computer instructions, wherein the computer instructions are used to cause a computer to perform the above-described text data preprocessing method for webpage advertisement identification.

[0009] According to another aspect of the present disclosure, there is provided a computer program product comprising a computer program, wherein the computer program, when executed by a processor, implements the above-described text data preprocessing method for webpage advertisement identification.

[0010] According to one or more embodiments of the present disclosure, the amount of data of the resource text can be reduced.

[0011] It should be understood that the content described in this section is not intended to identify key or important features of the embodiments of the present disclosure, nor is it used to limit the scope of the present disclosure. Other features of the present disclosure will become apparent from the following description. BRIEF DESCRIPTION OF DRAWINGS

[0012] The accompanying drawings illustrate exemplary embodiments and together with the description, explain and demonstrate principles of the exemplary embodiments. The drawings illustrate only exemplary embodiments and are therefore not to be considered limiting of the scope of the claims. In all the drawings, like reference numerals refer to like parts throughout the various figures and embodiments.

[0013] Figure 1 A schematic diagram illustrating an exemplary system in which various methods described herein can be implemented according to embodiments of the present disclosure;

[0014] Figure 2 A flowchart illustrating a text data preprocessing method for web advertisement identification according to embodiments of the present disclosure;

[0015] Figure 3 A flowchart illustrating a text data preprocessing method for web advertisement identification according to embodiments of the present disclosure;

[0016] Figure 4 A block diagram illustrating a structure of a text data preprocessing apparatus for web advertisement identification according to embodiments of the present disclosure;

[0017] Figure 5 A block diagram illustrating a structure of a text data preprocessing apparatus for web advertisement identification according to embodiments of the present disclosure;

[0018] Figure 6 A block diagram illustrating a structure of an exemplary electronic device that can be used to implement embodiments of the present disclosure. DETAILED DESCRIPTION

[0019] Exemplary embodiments of the present disclosure are described herein with reference to the accompanying drawings, in which various details are set forth to assist in understanding the embodiments of the present disclosure. However, it will be apparent to those of ordinary skill in the art that various changes and modifications can be made to the embodiments described herein without departing from the scope of the present disclosure. Also, descriptions of well-known functions and constructions can be omitted for clarity and conciseness.

[0020] In the present disclosure, unless otherwise specified, the terms "first", "second", and the like are used to describe various elements not to limit the positional relationship, the chronological relationship, or the importance of the elements, and the terms are used only to distinguish one element from another element. In some examples, the first element and the second element can refer to the same instance of the element, and in some cases, based on the context of the description, they can also refer to different instances.

[0021] The terminology used in the description of the various described examples in the present disclosure is for the purpose of describing particular examples only and is not intended to be limiting. As used in this description, the singular forms "a," "an," and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise. Furthermore, the use of the terms "a" and "an" and "the" and "at least one" are intended to include the meaning of "one or more." Additionally, the use of the term "including," as well as other forms such as "includes" and "included," is intended to be open-ended, meaning that other items not specifically named are also intended to be included. Except to the extent necessary or inherent in the circumstances, no single feature or combination of features is essential to the disclosure.

[0022] Embodiments of the present disclosure will be described in detail with reference to the drawings, where like reference numerals can refer to like elements throughout the various figures.

[0023] Figure 1 A schematic diagram illustrating an example system 100 in which various methods and apparatus described herein can be implemented in accordance with embodiments of the present disclosure is shown. Referring to Figure 1 , the system 100 includes one or more client devices 101, 102, 103, 104, 105, and 106, a server 120, and one or more communication networks 110 coupling the one or more client devices to the server 120. The client devices 101, 102, 103, 104, 105, and 106 can be configured to execute one or more application programs.

[0024] In embodiments of the present disclosure, the server 120 can run one or more services or software applications that enable the execution of a text data pre-processing method for web advertisement identification.

[0025] In certain embodiments, the server 120 can also provide other services or software applications that can include non-virtual environments and virtual environments. In certain embodiments, these services can be provided as web-based services or cloud services, such as to users of the client devices 101, 102, 103, 104, 105, and / or 106 under a software as a service (SaaS) model.

[0026] In Figure 1 In the illustrated configuration, the server 120 can include one or more components that implement the functionality performed by the server 120. These components can include software components that are executable by one or more processors, hardware components, or combinations thereof. Users operating the client devices 101, 102, 103, 104, 105, and / or 106 can in turn utilize one or more client application programs to interact with the server 120 to utilize the services provided by these components. It should be understood that various different system configurations are possible, which can differ from the system 100. Thus, Figure 1 is one example of a system for implementing the various methods described herein and is not intended to be limiting.

[0027] A user can use client devices 101, 102, 103, 104, 105, and / or 106 to obtain resource text of a web page. A client device can provide an interface that enables a user of the client device to interact with the client device. The client device can also output information to the user via the interface. Although Figure 1 Only six client devices are depicted, but those skilled in the art will appreciate that the present disclosure can support any number of client devices.

[0028] Client devices 101, 102, 103, 104, 105, and / or 106 can include various types of computer devices, such as portable handheld devices, general purpose computers (such as personal computers and laptop computers), workstation computers, wearable devices, smart screen devices, self-service kiosk devices, service robots, gaming systems, thin clients, various messaging devices, sensors or other sensing devices, and the like. These computer devices can run various types and versions of software applications and operating systems, such as MICROSOFT Windows, APPLE iOS, UNIX-like operating systems, Linux or Linux-like operating systems (such as GOOGLE Chrome OS); or include various mobile operating systems, such as MICROSOFT Windows Mobile OS, iOS, Windows Phone, Android. Portable handheld devices can include cellular telephones, smartphones, tablet computers, personal digital assistants (PDAs), and the like. Wearable devices can include head-mounted displays (such as smart glasses) and other devices. Gaming systems can include various handheld gaming devices, Internet-enabled gaming devices, and the like. The client devices are capable of executing various different applications, such as various Internet-related applications, communication applications (such as email applications), short message service (SMS) applications, and can use various communication protocols.

[0029] Network 110 can be any type of network familiar to those skilled in the art that can support data communications using any of a variety of available protocols, including without limitation TCP / IP, SNA, IPX, etc. Merely by way of example, one or more of networks 110 can be a local area network (LAN), an Ethernet-based network, a Token-Ring network, a wide area network (WAN), the Internet, a virtual network, a virtual private network (VPN), an intranet, an extranet, a public switched telephone network (PSTN), an infrared network, a wireless network (e.g., a Bluetooth, WIMAX, and / or Wi-Fi network), and / or any combination of these and / or other networks.

[0030] The server 120 can include one or more general purpose computers, special purpose server computers (e.g., PC (personal computer) servers, UNIX servers, mid-range servers), blade servers, mainframe computers, server clusters, or any other appropriate arrangement and / or combination. The server 120 can include one or more virtual machines running virtual operating systems, or other computing architectures involving virtualization (e.g., one or more flexible pools of logical storage devices that can be virtualized to maintain virtual storage devices for the servers). In various embodiments, the server 120 can run one or more services or software applications that provide the functionality described below.

[0031] The computing units in the server 120 can run one or more operating systems including any of the operating systems described above, as well as any commercially available server operating systems. The server 120 can also run any of a variety of additional server applications and / or mid-tier applications, including HTTP servers, FTP servers, CGI servers, JAVA servers, database servers, etc.

[0032] In some embodiments, the server 120 can include one or more applications to analyze and consolidate data feeds and / or event updates from users of the client devices 101, 102, 103, 104, 105, and 106. The server 120 can also include one or more applications to display the data feeds and / or real-time events via one or more display devices of the client devices 101, 102, 103, 104, 105, and 106.

[0033] In some embodiments, the server 120 can be a server of a distributed system, or a server combined with a blockchain. The server 120 can also be a cloud server, or an intelligent cloud computing server or intelligent cloud host with artificial intelligence technology. The cloud server is a host product in the cloud computing service system, to solve the defects of large management difficulty and weak business scalability in traditional physical host and virtual private server (VPS, Virtual Private Server) services.

[0034] The system 100 can also include one or more databases 130. In certain embodiments, these databases can be used to store data and other information. For example, one or more of the databases 130 can be used to store information such as audio files and video files. The databases 130 can reside in various locations. For example, databases used by the server 120 can be local to the server 120 or can be remote from the server 120 and can communicate with the server 120 via a network- or application-specific connection. The databases 130 can be of different types. In certain embodiments, databases used by the server 120 can be, for example, relational databases. One or more of these databases can store, update, and retrieve data to and from the databases in response to commands.

[0035] In certain embodiments, one or more of the databases 130 can also be used by applications to store application data. Databases used by applications can be databases of different types, such as key-value stores, object stores, or regular stores supported by file systems.

[0036] Figure 1 The system 100 can be configured and operated in various ways to enable the application of various methods and apparatuses described in accordance with the present disclosure.

[0037] At present, when the application browser browses the webpage, the loaded webpage often carries floating advertisements and interspersed advertisements, which seriously affects the browsing experience of the user. In order to shield the above-mentioned advertisements, the relevant technical personnel deploys an advertisement recognition model in the browser and the browser mobile terminal application, classifies the webpage resource text through recognition, identifies those resource texts that produce advertisements, and then shields these resource texts, no longer loads the webpage resources requested by these resource texts, thereby achieving the effect of shielding webpage advertisements. However, the process of advertisement recognition often consumes more time, resulting in the problem of slow webpage loading speed, which affects the browsing experience of the user to a certain extent.

[0038] Therefore, according to the embodiments of the present disclosure, as Figure 2As shown, a text data preprocessing method for webpage advertisement identification is provided, comprising: step S201, obtaining resource text of a webpage, the resource text comprising text for obtaining webpage resources; step S202, performing word segmentation processing on the resource text to obtain a plurality of first segmented words; step S203, performing cutting processing on each of the plurality of first segmented words to obtain a plurality of second segmented words, wherein each of the plurality of second segmented words is a segment of continuous characters in its corresponding first segmented word, and the length of each of the plurality of second segmented words is less than the length of its corresponding first segmented word; step S204, obtaining semantic feature vectors of the plurality of second segmented words; step S205, based on the semantic feature vectors of the plurality of second segmented words, obtaining a plurality of central semantic vectors, wherein the plurality of central semantic vectors are used to identify whether the webpage resources obtained by the resource text are advertisements; and step S206, in response to receiving an identification result that the webpage resources corresponding to the resource text are advertisements, based on the received instruction to shield the resource text, shielding the advertisements.

[0039] According to the text data preprocessing method of the embodiments of the present disclosure, the data dimension and data volume of the resource text can be reduced before inputting into the advertisement identification model, thereby reducing the calculation amount of the identification model and improving the calculation speed.

[0040] The resources loaded in webpage loading generally include webpage main frame, script, picture, video and embedded webpage, etc. Among them, the resource text for loading the webpage main frame is generally HTML text, the resource text for loading the script is generally Javascript or XHR type text, and the resource text for loading the embedded webpage is mainly Sub Document type text. Generally, the resource text for loading the script, picture or embedded webpage is more likely to generate advertisements, while the resource text for loading the webpage main frame and video basically will not generate advertisements.

[0041] According to some embodiments, the types of webpage resources to be obtained include at least script, picture and embedded webpage. According to some embodiments, when loading a webpage, a plurality of resource texts of the webpage are obtained, wherein the plurality of resource texts include texts for loading webpage main frame, script, picture, video and embedded webpage; and the resource texts for loading the webpage main frame and video are screened out. Thus, the amount of data to be processed can be further reduced.

[0042] When a resource text is obtained, the text needs to be segmented first to obtain a plurality of first segments. Since the resource text is generally a piece of code, the segmentation can be performed by using an English segmentation tool such as NLTK, SpaCy, StanfordCoreNLP, and the like. It can be understood that a person skilled in the art can select a segmentation tool to be applied by himself / herself, which is not limited herein.

[0043] Since the resource texts of web pages are of different sizes, more time is often needed to identify and classify larger resource texts. It is found through experiments that, for a larger resource text, a segment of text within a limited length range of the resource text can be intercepted to identify the advertisement of the resource text, and the accuracy of the identification is almost not affected.

[0044] According to some embodiments, the segmentation of the resource text to obtain a plurality of first segments includes: segmenting the text within a preset text length in the resource text, wherein the preset text length is less than the text length of the resource text. By intercepting part of the text in a longer resource text, the data amount can be further reduced, and the identification efficiency can be improved. Optionally, the preset text length can be determined by a person skilled in the art, for example, the preset text length can be 3 kB, that is, only the first 3 kB of the resource text is intercepted, which is not limited herein.

[0045] According to some embodiments, the segmentation of the resource text to obtain a plurality of first segments includes: excluding the first segment with a word frequency less than a preset word frequency threshold from the plurality of first segments; and excluding the stop word in the plurality of first segments.

[0046] Some randomly appearing words, such as randomly generated IDs, are often contained in the resource text of a web page. It is found through experiments that, by excluding these randomly appearing words and using the remaining words for classification and identification, the data amount can be reduced, the identification efficiency can be improved, and the accuracy of the identification is not affected. Therefore, after the segmentation of the resource text, the word frequency of each of the plurality of first segments can be counted, and the first segment with a word frequency less than a preset word frequency threshold can be excluded. Optionally, the preset word frequency threshold can be determined by a person skilled in the art, for example, the preset word frequency threshold can be 5, which is not limited herein.

[0047] In addition, the webpage resource text also contains some stop words. The stop words are words that have no actual meaning and are only used to help describe and express in the text, such as "the", "a", "an", "that", and "those" determiners, and for example, "over", "under", "above" prepositions, etc. Excluding these stop words can further reduce the data volume without affecting the recognition accuracy, thereby improving the recognition efficiency. The stop words can be screened out by applying a stop word list. Optionally, those skilled in the art can select the applied stop word list or construct a stop word list according to the needs, which is not limited here.

[0048] According to some embodiments, the cutting processing is performed on each of the plurality of first segmented words to obtain a plurality of second segmented words includes: performing cutting processing on each of the plurality of first segmented words by an N-Gram model.

[0049] The N-Gram model is an algorithm based on a statistical language model. The basic idea is to perform a sliding window operation with a cutting length of N on the content in the text according to the byte, forming a plurality of byte segment sequences. Through the N-Gram model, each first segmented word can be further cut and processed, and in this way, the granularity of the smaller words can be further screened out while supplementing some text features, and the data volume is reduced.

[0050] According to some embodiments, the cutting length parameter applied by the N-Gram model does not exceed a preset cutting length threshold. For example, the cutting length parameter applied by the N-Gram model can be 3, and for a first segmented word "adblock", the N-Gram model is applied to perform cutting processing, and a plurality of second segmented words such as "ad", "adb", "dbl", "blo", "loc", "ock", and "ck" can be obtained. When the applied cutting length parameter is too large, the above effect cannot be achieved, so the cutting length parameter does not exceed the preset cutting length threshold. It can be understood that those skilled in the art can determine the cutting length parameter and the preset cutting length threshold by themselves, which is not limited here.

[0051] Before inputting the obtained plurality of second segmented words into the advertisement recognition model, each second segmented word needs to be converted into a semantic feature vector. Optionally, those skilled in the art can apply a Word2vec model to map the second segmented word from text to a semantic feature vector.

[0052] Since the semantics of some second segmented words are relatively similar, the semantic feature vectors corresponding to the second segmented words with similar semantics also exhibit a clustered distribution in the vector space. Therefore, the semantics of these semantic feature vectors can be uniformly represented by obtaining the central semantic vectors of these semantic feature vectors, and the obtained multiple central semantic vectors can be used as inputs of the advertisement identification model, so as to further reduce the data volume and improve the computational efficiency of the advertisement identification model.

[0053] Optionally, a person skilled in the art can perform clustering analysis on the semantic feature vectors by using the K-means clustering algorithm to obtain the corresponding multiple central semantic vectors. It can be understood that a person skilled in the art can also use other clustering methods to obtain the central semantic vectors, which are not limited herein.

[0054] After obtaining the multiple central semantic vectors, the central semantic vectors can be input into the webpage advertisement identification model to identify whether the webpage resource corresponding to the resource text is an advertisement. The webpage advertisement identification model can use a classification model such as TextCNN or FastText, which is not limited herein.

[0055] When it is identified that the webpage resource corresponding to the resource text is an advertisement, an instruction for shielding the resource text can be input to shield the advertisement and improve the user browsing experience.

[0056] According to some embodiments, a text data preprocessing method for webpage advertisement identification is provided, as shown in Figure 3 S301, obtaining a resource text of a webpage, the resource text including a text for obtaining a webpage resource; S302, performing segmentation processing on the resource text to obtain multiple first segmented words; S303, screening out first segmented words with a word frequency less than a preset word frequency threshold from the multiple first segmented words; S304, screening out stop words from the multiple first segmented words; S305, performing cutting processing on each first segmented word in the multiple first segmented words to obtain multiple second segmented words, wherein each second segmented word in the multiple second segmented words is a segment of continuous characters in the corresponding first segmented word, and the length of each second segmented word in the multiple second segmented words is less than the length of the corresponding first segmented word; S306, obtaining semantic feature vectors of the multiple second segmented words; S307, obtaining multiple central semantic vectors based on the semantic feature vectors of the multiple second segmented words, wherein the multiple central semantic vectors are used to identify whether a webpage resource obtained by the resource text is an advertisement; and S308, in response to receiving an identification result that the webpage resource corresponding to the resource text is an advertisement, shielding the advertisement based on a received instruction for shielding the resource text. Figure 3 The steps S301-S308 in the method are similar to the above embodiments, and are not described herein.

[0057] According to some embodiments, such as Figure 4 As shown, a text data preprocessing apparatus 400 for webpage advertisement recognition is also provided, comprising: a first acquisition unit 410 configured to acquire resource text of a webpage, the resource text including text for acquiring webpage resources; a word segmentation unit 420 configured to perform word segmentation processing on the resource text to acquire a plurality of first words; and a word cutting unit 430 configured to perform word cutting processing on each of the plurality of first words to acquire a plurality of second words, wherein each of the plurality of second words is a continuous segment of characters in its corresponding first word, and the plurality of second words are... The length of each second segment in the binary word segmentation is less than the length of its corresponding first segment; the second acquisition unit 440 is configured to acquire the semantic feature vectors of the plurality of second segments; the third acquisition unit 450 is configured to acquire a plurality of central semantic vectors based on the semantic feature vectors of the plurality of second segments, wherein the plurality of central semantic vectors are used to identify whether the web page resource acquired by the resource text is an advertisement; and the blocking unit 460 is configured to, in response to receiving the identification result that the web page resource corresponding to the resource text is an advertisement, block the advertisement based on the received instruction to block the resource text.

[0058] The operation of units 410-460 of the text data preprocessing device 400 for web page advertisement recognition is similar to the operation of steps S201-S206 of the above-described text data preprocessing method, and will not be described in detail here.

[0059] According to some embodiments, the word segmentation unit of the text data preprocessing device can be further configured to: perform word segmentation processing on the text in the resource text that is within a preset text length, wherein the preset text length is less than the text length of the resource text.

[0060] According to some embodiments, a text data preprocessing apparatus 500 for web page advertisement recognition is also provided, such as... Figure 5 As shown, the word segmentation unit 520 is further configured as follows: a first filtering subunit 521 is configured to filter out first words whose word frequency is less than a preset word frequency threshold among the plurality of first words; and a second filtering subunit 522 is configured to filter out stop words among the plurality of first words.

[0061] The operation of units 510-560 and subunits 521-522 of the text data preprocessing device 500 for web page advertisement recognition is similar to the operation of steps S301-S308 of the above-described text data preprocessing method, and will not be described in detail here.

[0062] According to some embodiments, the word segmentation unit of the text data preprocessing apparatus can be further configured to perform word segmentation processing on each of the first segmented words in the plurality of first segmented words by an N-Gram model.

[0063] According to some embodiments, the N-Gram model applied in the text data preprocessing apparatus has an applied word segmentation length parameter not exceeding a preset word segmentation length threshold.

[0064] According to some embodiments, the types of the processed web resources of the text data preprocessing apparatus at least include scripts, pictures, and embedded web pages.

[0065] According to some embodiments, the text data preprocessing apparatus can further include a fourth acquisition unit configured to acquire a plurality of resource texts of a web page, wherein the plurality of resource texts include texts for loading a plurality of web resources; and a screening unit configured to screen out texts in the plurality of resource texts for loading two types of web resources, i.e., a web main frame and a video.

[0066] According to embodiments of the present disclosure, an electronic device, a readable storage medium, and a computer program product are also provided.

[0067] Reference Figure 6 A block diagram of the structure of an electronic device 600 that can be a server or a client of the present disclosure, which is an example of a hardware device that can be applied to aspects of the present disclosure, will now be described. The electronic device is intended to represent a wide variety of digital electronic computer devices, such as laptops, desktops, workstations, personal digital assistants, servers, blade servers, mainframes, and other appropriate computers. The electronic device can also represent a wide variety of mobile devices, such as personal digital assistants, cellular telephones, smart phones, wearable devices, and other similar computing devices. The components shown here, their connections, and their functions, as well as their interrelationships, are only examples and are not meant to limit the implementations of the present disclosure described and / or claimed in this document.

[0068] As shown in Figure 6 , the electronic device 600 includes a computing unit 601 that can perform various appropriate actions and processes in accordance with a computer program stored in a read-only memory (ROM) 602 or a computer program loaded into a random access memory (RAM) 603 from a storage unit 608. In the RAM 603, various programs and data required for the operation of the electronic device 600 can also be stored. The computing unit 601, the ROM 602, and the RAM 603 are connected to each other through a bus 604. An input / output (I / O) interface 605 is also connected to the bus 604.

[0069] A plurality of components in the electronic device 600 are connected to the I / O interface 605, including: an input unit 606, an output unit 607, a storage unit 608, and a communication unit 609. The input unit 606 can be any type of device that can input information to the electronic device 600, and can receive inputted digital or character information, and generate key signal inputs related to user settings and / or function controls of the electronic device, and can include, but is not limited to, a mouse, a keyboard, a touch screen, a track pad, a track ball, a joystick, a microphone, and / or a remote controller. The output unit 607 can be any type of device that can present information, and can include, but is not limited to, a display, a speaker, a video / audio output terminal, a vibrator, and / or a printer. The storage unit 608 can include, but is not limited to, a magnetic disk, an optical disk. The communication unit 609 allows the electronic device 600 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks, and can include, but is not limited to, a modem, a network card, an infrared communication device, a wireless communication transceiver, and / or a chipset, such as a Bluetooth TM device, an 802.11 device, a WiFi device, a WiMax device, a cellular communication device, and / or the like.

[0070] The computing unit 601 can be various general and / or special purpose processing components having processing and computing capabilities. Some examples of the computing unit 601 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any appropriate processor, controller, microcontroller, etc. The computing unit 601 performs various methods and processes described above, such as the resident point identification method. For example, in some embodiments, the resident point identification method can be implemented as a computer software program that is tangibly embodied in a machine-readable medium, such as the storage unit 608. In some embodiments, part or all of the computer program can be loaded and / or installed onto the electronic device 600 via the ROM 602 and / or the communication unit 609. When the computer program is loaded onto the RAM 603 and executed by the computing unit 601, one or more steps of the resident point identification method described above can be performed. Alternatively, in other embodiments, the computing unit 601 can be configured to perform the resident point identification method by any other appropriate means, such as by means of firmware.

[0071] Various implementations of the systems and techniques described above can be realized in digital electronic circuitry, integrated circuitry, a field programmable gate array (FPGA), an application specific integrated circuit (ASIC), a system on a chip (SOC), a complex programmable logic device (CPLD), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include implementation in one or more computer programs that are executable and / or interpretable on a programmable system including at least one programmable processor, which can be special or general purpose, coupled to receive data and instructions from, and to transmit data and instructions to, a storage system, at least one input device, and at least one output device.

[0072] Program code for carrying out methods of the present disclosure can be written in any combination of one or more programming languages. The program code can be provided to a processor or controller of a general purpose computer, special purpose computer, or other programmable data processing apparatus to produce a machine, such that the program code, when executed by the processor or controller, produces the functions / operations specified in the flowcharts and / or the block diagrams. The program code can be executed entirely on a machine, partially on a machine, partially on a machine as a stand-alone software package, partially on a machine and partially on a remote machine or entirely on a remote machine or server.

[0073] In the context of the present disclosure, a machine-readable medium can be a tangible medium that contains or stores a program for use by or in connection with an instruction execution system, apparatus, or device. The machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can include but is not limited to an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. More specific examples of the machine-readable storage medium will include one or more lines of electrical connections, portable computer disks, hard disk drives, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or Flash memory), optical fibers, portable compact disc read-only memories (CD-ROMs), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0074] To provide for interaction with a user, the systems and techniques described here can be implemented on a computer having a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user and a keyboard and a pointing device (e.g., a mouse or a trackball) by which the user can provide input to the computer. Other kinds of devices can be used to provide for interaction with a user as well; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form, including acoustic, speech, or tactile input.

[0075] The systems and techniques described here can be implemented in a computing system that includes a back end component (e.g., as a data server), or that includes a middleware component (e.g., an application server), or that includes a front end component (e.g., a user computer having a graphical user interface or a Web browser through which a user can interact with an implementation of the systems and techniques described here), or any combination of such back end, middleware, or front end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network (LAN), a wide area network (WAN), and the Internet.

[0076] The computer system can include clients and servers. A client and server are generally remote from each other and typically interact through a communication network. The relationship of client and server is generally established by computer programs running on the respective computers and having a client-server relationship to each other. The servers can be cloud servers, servers of a distributed system, or servers combined with a blockchain.

[0077] It should be understood that the various forms of flow shown above can be re-ordered, added to, or deleted from without departing from the scope of the present disclosure. For example, the steps recited in the present disclosure can be performed in parallel, in series, or in a different order, as long as the desired results of the technology disclosed in the present disclosure are achieved, which is not limited herein.

[0078] While embodiments or examples of this disclosure have been described with reference to the figures, it will be understood that the methods, systems, and devices described above are merely exemplary embodiments or examples, and the scope of the application is not limited to these embodiments or examples. Various elements of the embodiments or examples can be omitted or substituted by equivalents thereof. Furthermore, the steps can be performed in a different order than described in the disclosure. Further, various elements of the embodiments or examples can be combined in various ways. It is important that as technology evolves, many of the elements described herein can be substituted by equivalents which serve the same function.

Claims

1. A text data preprocessing method for webpage advertisement recognition, comprising: Obtaining the resource text of a webpage, wherein the resource text includes code for loading webpage resources, and the types of webpage resources include at least scripts, images, and embedded webpages; the obtained resource text of the webpage includes: Obtain multiple resource texts of a webpage, wherein the multiple resource texts include texts used to load various webpage resources; and Filter out the text in the multiple resource texts that is used to load the following two types of web page resources: web page main frame and video; The resource text is segmented to obtain multiple first segments, each of which is a word in the resource text. Each first word is segmented to obtain multiple second words. The segmentation process is used to segment each first word by bytes using a sliding window with a length equal to the segmentation length parameter, and to filter out first words with a granularity smaller than the segmentation length parameter. The segmentation length parameter does not exceed a preset segmentation length threshold. Each second word is a continuous segment of characters in its corresponding first word and does not have complete semantics. The length of each second word is less than the length of its corresponding first word. Obtain the semantic feature vectors of the multiple second word segments; Based on the semantic feature vectors of the multiple second word segments, multiple central semantic vectors are obtained. These central semantic vectors are obtained through cluster analysis of the semantic feature vectors of the multiple second word segments. The multiple central semantic vectors are then input into a webpage advertisement recognition model to identify whether the webpage resource obtained from the resource text is an advertisement. In response to the recognition result that the webpage resource corresponding to the resource text is an advertisement, the advertisement is blocked when the webpage is loaded based on the received instruction to block the resource text.

2. The method according to claim 1, wherein, The step of segmenting the resource text to obtain multiple first words includes: The text within the preset text length in the resource text is segmented into words, wherein the preset text length is less than the text length of the resource text.

3. The method according to claim 1 or 2, wherein, The step of segmenting the resource text to obtain multiple first words includes: Filter out the first word segments whose word frequency is less than a preset word frequency threshold from the plurality of first word segments; and Stop words are filtered out from the multiple first word segments.

4. The method according to claim 1 or 2, wherein, The step of performing word segmentation on each of the plurality of first word segments to obtain a plurality of second word segments includes: Each of the multiple first word segments is segmented using the N-Gram model.

5. A text data preprocessing apparatus for webpage advertisement recognition, comprising: The first acquisition unit is configured to acquire the resource text of a webpage, wherein the resource text includes code for loading webpage resources, and the types of webpage resources include at least scripts, images, and embedded webpages. The acquired resource text of the webpage includes: Obtain multiple resource texts of a webpage, wherein the multiple resource texts include texts used to load various webpage resources; and Filter out the text in the multiple resource texts that is used to load the following two types of web page resources: web page main frame and video; The word segmentation unit is configured to perform word segmentation processing on the resource text to obtain a plurality of first words, each of which is a word in the resource text. The word segmentation unit is configured to perform word segmentation processing on each first word to obtain multiple second words. The word segmentation processing is used to segment each first word by a sliding window with a length equal to the word segmentation length parameter in bytes, and to filter out first words with a granularity smaller than the word segmentation length parameter. The word segmentation length parameter does not exceed a preset word segmentation length threshold. Each second word is a continuous segment of characters in its corresponding first word and does not have complete semantics. The length of each second word is less than the length of its corresponding first word. The second acquisition unit is configured to acquire the semantic feature vectors of the plurality of second word segments; The third acquisition unit is configured to acquire multiple central semantic vectors based on the semantic feature vectors of the multiple second word segments. These multiple central semantic vectors are obtained through cluster analysis of the semantic feature vectors of the multiple second word segments. The multiple central semantic vectors are used as input to a webpage advertisement recognition model to identify whether the webpage resource obtained from the resource text is an advertisement. The blocking unit is configured to, in response to receiving a recognition result that the webpage resource corresponding to the resource text is an advertisement, block the advertisement based on the received instruction to block the resource text when loading the webpage.

6. The apparatus according to claim 5, wherein, The word segmentation unit is further configured as follows: The text within the preset text length in the resource text is segmented into words, wherein the preset text length is less than the text length of the resource text.

7. The apparatus according to claim 5 or 6, wherein, The word segmentation unit is further configured as follows: The first filtering subunit is configured to filter out first word segments whose word frequency is less than a preset word frequency threshold from the plurality of first word segments; and The second filtering subunit is configured to filter out stop words from the plurality of first word segments.

8. The apparatus according to claim 5 or 6, wherein, The word segmentation unit is further configured as follows: Each of the multiple first word segments is segmented using the N-Gram model.

9. An electronic device, comprising: At least one processor; as well as A memory that is communicatively connected to the at least one processor; in The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the method of any one of claims 1-4.

10. A non-transitory computer-readable storage medium storing computer instructions, wherein, The computer instructions are used to cause the computer to perform the method according to any one of claims 1-4.

11. A computer program product comprising a computer program, wherein, The computer program, when executed by a processor, implements the method of any one of claims 1-4.

Images