Method and device for classifying crawled articles and medium

Automatically classify crawled article data through web page crawler technology and machine learning classifiers, solving the problem of time-consuming manual classification and improving data processing efficiency.

CN118981533AInactive Publication Date: 2024-11-19ANRUI DIGITAL INFORMATION TECH CO LTD
View PDF 6 Cites 0 Cited by

Patent Information

Application Number
CN202411006452.0
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-07-25
Publication Date
2024-11-19
Estimated Expiration
Not applicable · inactive patent

AI Technical Summary

Technical Problem

In the prior art, crawled article data requires manual classification and archive, which consumes a lot of time and lacks intelligent classification functions.

Method used

Article data is crawled through web page crawler technology, and article classification model is generated using support vector classifier and naive Bayes classifier to realize automatic classification of articles.

Benefits of technology

It realizes intelligent classification of crawled article data, reduces the workload of manual operations, and improves the efficiency of data acquisition and sorting.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118981533A_ABST
    Figure CN118981533A_ABST
Patent Text Reader

Abstract

The invention discloses a method and device for classifying crawled articles and a medium. The method comprises the following steps: performing article crawling on a to-be-crawled page through a webpage page crawler technology to obtain article data; article titles and article contents in the article data are stored in two columns of a CSV file respectively, and CSV file data are obtained; using an article title in the CSV file data as a label, using article content as training data, and performing training through a support vector machine classifier to generate an article classification model; and inputting the crawled to-be-classified articles into an article classification model to obtain a classification result of the to-be-classified articles.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the technical field of data classification, and more specifically, to a method, device and medium for classifying crawled articles. Background Art

[0002] When crawling article data from a specified web page, the crawled data needs to be manually classified and archived, which is very time-consuming. Therefore, how to develop an intelligent classification function to automatically classify and archive the crawled data has become a technical problem that needs to be urgently solved in this field. Summary of the invention

[0003] In view of the deficiencies in the prior art, the present invention provides a method, device and medium for classifying crawled articles.

[0004] According to one aspect of the present invention, a method for classifying crawled articles is provided, comprising:

[0005] Use web page crawler technology to crawl articles on the crawled page and obtain article data;

[0006] Store the article title and article content in the article data into two columns of the CSV file respectively, and obtain the CSV file data;

[0007] The article titles in the CSV file data are used as labels, and the article contents are used as training data. The support vector machine classifier is trained to generate an article classification model.

[0008] Input the crawled articles to be classified into the article classification model to obtain the classification results of the articles to be classified.

[0009] Optionally, it also includes: preprocessing the article data, removing stop words in the article data, stemming, lemmatization and tokenization.

[0010] Optionally, the article content is used as training data and trained through a support vector machine classifier to generate an article classification model, including:

[0011] The bag-of-words model is used to extract features from the article content and obtain numerical feature vectors;

[0012] The numerical feature vector is trained through the Naive Bayes classifier to generate an article classification model.

[0013] Optionally, a bag-of-words model is used to extract features from the article content to obtain a numerical feature vector, including:

[0014] Perform vocabulary division according to the control of the article content and punctuation marks to obtain the vocabulary data of the article content;

[0015] Number each word in the vocabulary data;

[0016] Obtain the number of occurrences of each word in the article content according to the number of each word, and determine the word frequency of each word;

[0017] Combine the word frequency of each word in the article content into a vector to generate a numerical feature vector.

[0018] According to another aspect of the present invention, there is provided a device for classifying crawled articles, comprising:

[0019] A crawling module is used to crawl articles on the crawled page through web page crawler technology to obtain article data;

[0020] The acquisition module is used to store the article title and article content in the article data into two columns of the CSV file respectively, and obtain the CSV file data;

[0021] A generation module is used to use the article titles in the CSV file data as labels and the article contents as training data, and train the support vector machine classifier to generate an article classification model;

[0022] The classification module is used to input the crawled articles to be classified into the article classification model to obtain the classification results of the articles to be classified.

[0023] According to another aspect of the present invention, a computer-readable storage medium is provided, wherein the storage medium stores a computer program, and the computer program is used to execute the method described in any one of the above aspects of the present invention.

[0024] According to another aspect of the present invention, an electronic device is provided, comprising: a processor; a memory for storing instructions executable by the processor; the processor is configured to read the executable instructions from the memory and execute the instructions to implement the method described in any one of the above aspects of the present invention.

[0025] Therefore, this application can more efficiently obtain and organize the required data and reduce the workload of manual operations by training the article classification model to intelligently classify the crawled article data. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] A more complete understanding of exemplary embodiments of the present invention may be obtained by referring to the following drawings:

[0027] Figure 1 is a flowchart of a method for classifying crawled articles provided by an exemplary embodiment of the present invention;

[0028] Figure 2is a schematic diagram of the structure of a device for classifying crawled articles provided by an exemplary embodiment of the present invention;

[0029] Figure 3 This is a structure of an electronic device provided by an exemplary embodiment of the present invention. DETAILED DESCRIPTION

[0030] Below, the exemplary embodiments according to the present invention will be described in detail with reference to the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all the embodiments of the present invention, and it should be understood that the present invention is not limited to the exemplary embodiments described here.

[0031] It should be noted that the relative arrangement of components and steps, the numerical expressions and numerical values ​​set forth in these embodiments do not limit the scope of the present invention unless specifically stated otherwise.

[0032] Those skilled in the art can understand that the terms "first" and "second" in the embodiments of the present invention are only used to distinguish different steps, devices or modules, etc., and neither represent any specific technical meaning nor indicate the necessary logical order between them.

[0033] It should also be understood that, in the embodiments of the present invention, “plurality” may refer to two or more than two, and “at least one” may refer to one, two or more than two.

[0034] It should also be understood that any component, data or structure mentioned in the embodiments of the present invention can generally be understood as one or more, unless explicitly limited or otherwise indicated in the context.

[0035] In addition, the term "and / or" in the present invention is only a description of the association relationship of the associated objects, indicating that there can be three relationships. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in the present invention generally indicates that the associated objects before and after are in an "or" relationship.

[0036] It should also be understood that the description of the various embodiments of the present invention focuses on the differences between the various embodiments, and the same or similar aspects thereof can be referenced to each other, and for the sake of brevity, they will not be described one by one.

[0037] At the same time, it should be understood that for the convenience of description, the sizes of the various parts shown in the drawings are not drawn according to the actual proportional relationship.

[0038] The following description of at least one exemplary embodiment is merely illustrative in nature and is in no way intended to limit the invention, its application, or uses.

[0039] Technologies, methods, and equipment known to ordinary technicians in the relevant art may not be discussed in detail, but where appropriate, the technologies, methods, and equipment should be considered part of the specification.

[0040] It should be noted that like reference numerals and letters refer to similar items in the following figures, and therefore, once an item is defined in one figure, it need not be further discussed in subsequent figures.

[0041] Embodiments of the present invention can be applied to electronic devices such as terminal devices, computer systems, servers, etc., which can operate with many other general or special computing system environments or configurations. Examples of well-known terminal devices, computing systems, environments and / or configurations suitable for use with electronic devices such as terminal devices, computer systems, servers, etc. include, but are not limited to: personal computer systems, server computer systems, thin clients, thick clients, handheld or laptop devices, microprocessor-based systems, set-top boxes, programmable consumer electronics, network personal computers, small computer systems, large computer systems, and distributed cloud computing technology environments including any of the above systems, etc.

[0042] Electronic devices such as terminal devices, computer systems, servers, etc. can be described in the general context of computer system executable instructions (such as program modules) executed by computer systems. Generally, program modules can include routines, programs, object programs, components, logic, data structures, etc., which perform specific tasks or implement specific abstract data types. Computer systems / servers can be implemented in a distributed cloud computing environment, where tasks are performed by remote processing devices linked through a communication network. In a distributed cloud computing environment, program modules can be located on local or remote computing system storage media including storage devices.

[0043] Exemplary Methods

[0044] Figure 1 FIG. 1 is a flow chart of a method for classifying crawled articles provided by an exemplary embodiment of the present invention. This embodiment can be applied to electronic devices, such as Figure 1 As shown, the method 100 for classifying crawled articles includes the following steps:

[0045] Step 101, crawling articles on the page to be crawled by using web page crawler technology to obtain article data;

[0046] Step 102, storing the article title and article content in the article data into two columns of the CSV file respectively, and obtaining the CSV file data;

[0047] Step 103, using the article titles in the CSV file data as labels and the article contents as training data, training is performed through a support vector machine classifier to generate an article classification model;

[0048] Step 104, input the crawled articles to be classified into the article classification model to obtain the classification results of the articles to be classified.

[0049] Specifically, in order to classify and archive the crawled data, the present invention uses text classification technology and methods, and uses a machine learning model to classify articles. The specific steps are as follows:

[0050] 1. Store the crawled articles, article titles, and article contents into two columns in a CSV file.

[0051] The method of crawling data is as follows:

[0052] 1) Use the requests library to send HTTP requests to obtain page content, and use the BeautifulSoup library to parse the page:

[0053] Python

[0054] url="https: / / example.com / articles"

[0055] response = requests.get(url)

[0056] soup=BeautifulSoup(response.content,"html.parser")

[0057] 2) Locate the HTML element where the article content is located: Use the selector function of the BeautifulSoup library to locate the HTML element containing the article content.

[0058] Python

[0059] articles=soup.find_all("div",class_="article")

[0060] 3) Extract article data: For each article element, extract the required data, such as title, author, publication time, etc.

[0061] Python

[0062] for article in articles:

[0063] title=article.find("h2").text

[0064] author=article.find("span",class_="author").text

[0065] date=article.find("span",class_="date").text

[0066] # Further process or store the acquired data

[0067] print("Title:",title)

[0068] print("Author:",author)

[0069] print("Date:",date).

[0070] 2. Use the `pandas` library to load the data and separate the text content `X` and the label `y`.

[0071] 3. Next, use `TfidfVectorizer` to convert the text content into feature vectors. `TfidfVectorizer` calculates the TF-IDF weight of each word and represents the article in vector form.

[0072] 4. Then, use the `train_test_split` function to split the dataset into training and test sets.

[0073] 5. Next, use the support vector machine (SVM) classifier as an example and train the classifier model using the training set data.

[0074] The text classification method is as follows: to classify articles according to their content, natural language processing (NLP) technology and machine learning methods are used.

[0075] 1). Collect and prepare data sets: Collect a batch of article data sets that have been annotated according to classification labels. Make sure that each sample in the data set has a corresponding classification label.

[0076] 2) Text preprocessing: Perform text preprocessing on the article, including removing stop words, stemming or lemmatization, tokenization, etc. to reduce noise and extract useful features.

[0077] 3) Feature extraction: Use the bag-of-words model to convert text into numerical features that can be used in machine learning models.

[0078] 4). Divide into training and test sets: Divide the dataset into training and test sets for training and evaluating the model.

[0079] 5) Training model: Select a machine learning algorithm or deep learning model suitable for the task, such as the Naive Bayes classifier. Use the training set to train the model.

[0080] 6) Model evaluation: Use the test set to evaluate the trained model and calculate indicators such as classification accuracy, precision, recall, F1 score, etc. to evaluate the performance of the model.

[0081] 7). Prediction classification: Use the trained model to classify and predict new unknown articles.

[0082] The present invention uses the CountVectorizer class of the sklearn library to calculate the bag-of-words vector, as follows:

[0083] 1). Create an instance of CountVectorizer and make necessary configurations by setting some parameters, such as setting the maximum number of features, setting stop words, etc.

[0084] 2.) Call the instance's fit_transform() method and pass the text data to be processed to this method. This method will convert the text data into the representation of the bag-of-words model.

[0085] 3.) The CountVectorizer class will automatically perform the following steps to calculate the bag-of-words vector:

[0086] a. Word segmentation: Split text data into individual words. It can be segmented based on spaces, punctuation marks, etc.

[0087] b. Build a vocabulary: Count all the words that appear in the text data and assign a unique number to each word.

[0088] c. Calculate word frequency: count the number of occurrences of each word in each text data.

[0089] d. Construct a bag-of-words vector: Combine the word frequencies of each text data into a vector. Each dimension in the vector corresponds to a word in the vocabulary, and the value is the number of times the word appears in the text.

[0090] 4). Use the get_feature_names() method to get the vocabulary, that is, all the words that appear in the text.

[0091] 5). Use the toarray() method to represent the bag-of-words vector as an array.

[0092] 6. Then, use the trained classifier to make classification predictions on the test set data.

[0093] 7. Finally, use the `accuracy_score` function to calculate the accuracy of the prediction results to evaluate the performance of the classifier.

[0094] Therefore, this application can more efficiently obtain and organize the required data and reduce the workload of manual operations by training the article classification model to intelligently classify the crawled article data.

[0095] Exemplary Devices

[0096] Figure 2 FIG. 1 is a schematic diagram of a structure of an apparatus for classifying crawled articles provided by an exemplary embodiment of the present invention. Figure 2 As shown, the device 200 includes:

[0097] The crawling module 210 is used to crawl articles on the crawled page by using the web page crawler technology to obtain article data;

[0098] The acquisition module 220 is used to store the article title and the article content in the article data into two columns of the CSV file respectively, and acquire the CSV file data;

[0099] A generating module 230 is used to use the article titles in the CSV file data as labels and the article contents as training data, and to generate an article classification model by training a support vector machine classifier;

[0100] The classification model,240,is used to input the crawled articles to be classified into the,article classification model to obtain the classification results of the articles to be classified.

[0101] Optionally, the device 200 further includes: a preprocessing module, which is used to preprocess the article data, remove stop words, stem, lemmatize and tokenize the article data.

[0102] Optionally, the generating module 230 includes:

[0103] The acquisition submodule is used to extract features from the article content using the bag-of-words model and obtain numerical feature vectors;

[0104] The generation submodule is used to train the numerical feature vector through the naive Bayes classifier to generate an article classification model.

[0105] Optionally, obtain a submodule for, including:

[0106] An acquisition unit, used for performing vocabulary division according to the control and punctuation marks of the article content, and acquiring vocabulary data of the article content;

[0107] A numbering unit, used for numbering each word in the vocabulary data;

[0108] A determination unit, used to obtain the number of occurrences of each word in the article content according to the number of each word, and determine the word frequency of each word;

[0109] The generating unit is used to combine the word frequency of each word in the article content into a vector to generate a numerical feature vector.

[0110] Exemplary Electronic Devices

[0111] Figure 3 This is a structure of an electronic device provided by an exemplary embodiment of the present invention. Figure 3 As shown, the electronic device 30 includes one or more processors 31 and a memory 32 .

[0112] The processor 31 may be a central processing unit (CPU) or other forms of processing units having data processing capabilities and / or instruction execution capabilities, and may control other components in the electronic device to perform desired functions.

[0113] The memory 32 may include one or more computer program products, and the computer program product may include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. The volatile memory may include, for example, random access memory (RAM) and / or cache memory (cache), etc. The non-volatile memory may include, for example, read-only memory (ROM), hard disk, flash memory, etc. One or more computer program instructions may be stored on the computer-readable storage medium, and the processor 31 may run the program instructions to implement the methods of the software programs of the various embodiments of the present invention described above and / or other desired functions. In one example, the electronic device may also include: an input device 33 and an output device 34, which are interconnected via a bus system and / or other forms of connection mechanisms (not shown).

[0114] In addition, the input device 33 may also include, for example, a keyboard, a mouse, and the like.

[0115] The output device 34 can output various information to the outside. The output device 34 can include, for example, a display, a speaker, a printer, a communication network and a remote output device connected thereto.

[0116] Of course, to simplify, Figure 3 Only some of the components related to the present invention in the electronic device are shown, and components such as a bus, an input / output interface, etc. are omitted. In addition, according to specific application conditions, the electronic device may also include any other appropriate components.

[0117] Exemplary computer program products and computer-readable storage media

[0118] In addition to the above-mentioned methods and devices, an embodiment of the present invention may also be a computer program product, which includes computer program instructions, which, when executed by a processor, enable the processor to execute the steps of the method according to various embodiments of the present invention described in the above-mentioned "Exemplary Method" section of this specification.

[0119] The computer program product may be written in any combination of one or more programming languages ​​to write program code for performing the operations of the embodiments of the present invention, including object-oriented programming languages ​​such as Java, C++, etc., and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user computing device, partially on the user device, as a separate software package, partially on the user computing device and partially on a remote computing device, or entirely on a remote computing device or server.

[0120] In addition, an embodiment of the present invention may also be a computer-readable storage medium having computer program instructions stored thereon, which, when executed by a processor, enables the processor to execute the steps of the method for information mining of historical change records according to various embodiments of the present invention described in the above “Exemplary Method” section of this specification.

[0121] The computer readable storage medium can adopt any combination of one or more readable media. The readable medium can be a readable signal medium or a readable storage medium. The readable storage medium can include, for example, but is not limited to, a system, system or device of electricity, magnetism, light, electromagnetic, infrared, or semiconductor, or any combination of the above. More specific examples (non-exhaustive list) of readable storage media include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.

[0122] The basic principle of the present invention is described above in conjunction with specific embodiments. However, it should be pointed out that the advantages, strengths, effects, etc. mentioned in the present invention are only examples and not limitations, and it cannot be considered that these advantages, strengths, effects, etc. must be possessed by each embodiment of the present invention. In addition, the specific details disclosed above are only for the purpose of illustration and facilitation of understanding, rather than limitation, and the above details do not limit the present invention to being implemented by adopting the above specific details.

[0123] Each embodiment in this specification is described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the embodiments can be referred to each other. For the system embodiment, since it basically corresponds to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0124] The block diagrams of the devices, systems, equipment, and systems involved in the present invention are only illustrative examples and are not intended to require or imply that they must be connected, arranged, and configured in the manner shown in the block diagram. As will be appreciated by those skilled in the art, these devices, systems, equipment, and systems can be connected, arranged, and configured in any manner. Words such as "including", "comprising", "having", etc. are open words, referring to "including but not limited to", and can be used interchangeably with them. The words "or" and "and" used here refer to the words "and / or" and can be used interchangeably with them, unless the context clearly indicates otherwise. The word "such as" used here refers to the phrase "such as but not limited to", and can be used interchangeably with it.

[0125] The method and system of the present invention may be implemented in many ways. For example, the method and system of the present invention may be implemented by software, hardware, firmware or any combination of software, hardware, firmware. The above order of steps for the method is only for illustration, and the steps of the method of the present invention are not limited to the order specifically described above, unless otherwise specifically stated. In addition, in some embodiments, the present invention may also be implemented as a program recorded in a recording medium, which includes machine-readable instructions for implementing the method according to the present invention. Thus, the present invention also covers a recording medium storing a program for executing the method according to the present invention.

[0126] It should also be noted that in the system, device and method of the present invention, each component or each step can be decomposed and / or recombined. These decompositions and / or recombinations should be regarded as equivalent schemes of the present invention. The above description of the disclosed aspects is provided to enable any technician in the field to make or use the present invention. Various modifications to these aspects are very obvious to those skilled in the art, and the general principles defined here can be applied to other aspects without departing from the scope of the present invention. Therefore, the present invention is not intended to be limited to the aspects shown here, but in accordance with the widest range consistent with the principles and novel features disclosed here.

[0127] The above description has been given for the purpose of illustration and description. In addition, this description is not intended to limit the embodiments of the present invention to the forms disclosed herein. Although a number of example aspects and embodiments have been discussed above, those skilled in the art will recognize certain variations, modifications, changes, additions and sub-combinations thereof.

Claims

1. A method for classifying crawled articles, It is characterized in that include: Use web page crawler technology to crawl articles on the crawled page and obtain article data; The article title and article content in the article data are stored in two columns of a CSV file respectively, to obtain CSV file data; The article titles in the CSV file data are used as labels, and the article contents are used as training data, and a support vector machine classifier is used for training to generate an article classification model; The crawled articles to be classified are input into the article classification model to obtain the classification results of the articles to be classified.

2. The method according to claim 1, It is characterized in that Also includes: The article data is preprocessed to remove stop words, stem, lemmatize, and tokenize the article data.

3. The method according to claim 1, It is characterized in that The article content is used as training data and trained through a support vector machine classifier to generate an article classification model, including: Using a bag-of-words model to extract features from the article content to obtain a numerical feature vector; The numerical feature vector is trained by a naive Bayes classifier to generate the article classification model.

4. The method according to claim 1, It is characterized in that The bag-of-words model is used to extract features from the article content to obtain numerical feature vectors, including: Perform vocabulary division according to the control and punctuation marks of the article content to obtain vocabulary data of the article content; Numbering each word in the vocabulary data; Obtain the number of occurrences of each word in the article content according to the number of each word, and determine the word frequency of each word; The word frequencies of each word in the article content are combined into a vector to generate the numerical feature vector.

5. A device for classifying crawled articles, It is characterized in that include: A crawling module is used to crawl articles on the crawled page through web page crawler technology to obtain article data; An acquisition module, used for storing the article title and article content in the article data into two columns of a CSV file respectively, and acquiring the CSV file data; A generation module, used to use the article titles in the CSV file data as labels and the article contents as training data, train a support vector machine classifier, and generate an article classification model; The classification module is used to input the crawled articles to be classified into the article classification model to obtain the classification results of the articles to be classified.

6. The device according to claim 5, It is characterized in that Also includes: The preprocessing module is used to preprocess the article data, remove stop words, stem, restore lemmas and tokenize the article data.

7. The device according to claim 5, It is characterized in that Generate modules, including: An acquisition submodule is used to extract features from the article content using a bag-of-words model to obtain a numerical feature vector; The generation submodule is used to train the numerical feature vector through a naive Bayes classifier to generate the article classification model.

8. The device according to claim 5, It is characterized in that Get submodules for, including: An acquisition unit, used for performing vocabulary division according to the control and punctuation marks of the article content, and acquiring vocabulary data of the article content; A numbering unit, used for numbering each word in the vocabulary data; A determination unit, used to obtain the number of occurrences of each word in the article content according to the number of each word, and determine the word frequency of each word; A generating unit is used to combine the word frequency of each word in the article content into a vector to generate the numerical feature vector.

9. A computer-readable storage medium, It is characterized in that The storage medium stores a computer program, and the computer program is used to execute the method according to any one of claims 1 to 4.

10. An electronic device, It is characterized in that The electronic device comprises: processor; a memory for storing instructions executable by the processor; The processor is used to read the executable instructions from the memory and execute the instructions to implement the method described in any one of claims 1 to 4.

Citation Information

Patent Citations

  • Classifier training method and apparatus, electronic device and computer readable medium

    CA3063580A1

  • IT (Information Technology) information aggregation reading personalized recommendation method

    CN105243087A

  • Malicious webpage classification model construction method and system and storage medium

    CN116881601A

  • Anti-fraud method and device based on ChatGPT model

    CN117407866A

  • Method and System for Document Classification

    US20100185568A1