Method and system for processing web page data

Abstract

The application discloses a webpage data processing method and system. The method is applied to the analysis field, and comprises the following steps: in response to webpage data of a preset webpage changing, obtaining new webpage data of the preset webpage; respectively processing title content and text content in the new webpage data to obtain a keyword set corresponding to the title content and a text vector corresponding to the text content; determining a screening coefficient of the new webpage data based on the keyword set and the text vector; and in response to the screening coefficient being greater than a preset screening coefficient, storing the new webpage data. The application solves the technical problem of low matching degree of screened text.

Description

Technical Field

[0001] This invention relates to the field of analysis, and more specifically, to a method and system for processing web page data. Background Technology

[0002] Currently, the external environment for the power industry is constantly changing. Failure to promptly capture important policies and directions will adversely affect the company's operations and development, necessitating the ability to quickly obtain relevant data when analyzing text. However, current technology for filtering webpage content involves segmenting the webpage title and matching the obtained segments to determine if the webpage content meets the filtering requirements. Since searching based on the title cannot perfectly match the content, the filtering results are not accurate enough.

[0003] There is currently no effective solution to the above problems. Summary of the Invention

[0004] This invention provides a method and system for processing web page data, which at least solves the technical problem of low matching degree of filtered text.

[0005] According to one aspect of the present invention, a method for processing web page data is provided, comprising: in response to a change in the web page data of a preset web page, acquiring new web page data of the preset web page; processing the title content and body content in the new web page data respectively to obtain a keyword set corresponding to the title content and a text vector corresponding to the body content; determining a filtering coefficient for the new web page data based on the keyword set and the text vector, wherein the filtering coefficient is used to characterize the degree of matching between the new web page data and preset filtering conditions; and storing the new web page data in response to the filtering coefficient being greater than the preset filtering coefficient.

[0006] Optionally, the filtering coefficients for new webpage data are determined based on a keyword set and text vectors, including: obtaining a preset keyword set and a preset text vector set; matching each keyword in the keyword set with the preset keyword set to obtain a first coefficient for the keyword set, wherein the first coefficient is used to characterize the degree of matching between the keyword set and the preset keyword set; matching the text vectors with the preset text vector set to obtain a second coefficient for the text vectors, wherein the second coefficient is used to characterize the degree of matching between the text vectors and the preset text vector set; and weighting the first coefficient and the second coefficient to obtain the filtering coefficients.

[0007] Optionally, each keyword in the keyword set is matched with a preset keyword set to obtain a first coefficient of the keyword set, including: determining the number of keywords in the keyword set to obtain a first number; matching each keyword in the keyword set with the preset keyword set to obtain a score for each keyword, wherein the score is used to represent whether each keyword is successfully matched with the preset keyword set; obtaining the sum of the scores of all keywords in the keyword set to obtain a total score; obtaining the product of the first number and the preset score to obtain a target score, wherein the preset score is used to represent the average score of all preset keywords in the preset keyword set; and obtaining the ratio of the total score to the target total score to obtain the first coefficient.

[0008] Optionally, matching the text vector with a preset set of text vectors to obtain a second coefficient of the text vector includes: determining the number of preset text vectors in the preset set of text vectors to obtain a second number, wherein different preset text vectors in the preset set of text vectors are used to represent templates of different types of body text content; obtaining the similarity between the text vector and multiple preset text vectors in the preset set of text vectors to obtain multiple similarities; obtaining the sum of multiple similarities to obtain a total similarity; obtaining the product of the second number and the preset similarities to obtain a target similarity; and obtaining the ratio of the total similarity to the target similarity to obtain a second coefficient.

[0009] Optionally, the title content and body content in the new webpage data are processed separately to obtain a keyword set corresponding to the title content and a text vector corresponding to the body content, including: filtering the new webpage data to obtain the text to be filtered; performing word segmentation on the title content in the text to be filtered to obtain a keyword set; and performing semantic analysis on the body content in the text to be filtered to obtain a text vector.

[0010] Optionally, the title content in the text to be filtered is segmented to obtain a keyword set, including: filtering the title content to obtain the first text; and segmenting the first text using a word segmentation algorithm to obtain the keyword set.

[0011] Optionally, semantic analysis is performed on the main text content of the text to be filtered to obtain a text vector, including: segmenting the main text content using a word segmentation algorithm to obtain a second text; determining the weights of words in the second text; obtaining the vector representation of the second text; and determining the text vector based on the weights and vector representation.

[0012] Optionally, determining the weight of words in the second text includes: processing the words using a term frequency-inverse document frequency algorithm to obtain initial weights; determining a first weight parameter for the words based on the first word located on a different page in the second text; determining a second weight parameter for the words based on the second word located on a different line in the second text; determining a third weight parameter for the words based on all words in the second text; and adjusting the initial weights based on the first, second, and third weight parameters to obtain the weights of the words in the second text.

[0013] According to another aspect of the present invention, a webpage data processing apparatus is also provided, comprising: an acquisition module, configured to acquire new webpage data of the preset webpage in response to a change in the webpage data of the preset webpage; a processing module, configured to process the title content and body content of the new webpage data respectively to obtain a keyword set corresponding to the title content and a text vector corresponding to the body content; a determination module, configured to determine a filtering coefficient of the new webpage data based on the keyword set and the text vector, wherein the filtering coefficient is used to characterize the degree of matching between the new webpage data and preset filtering conditions; and a storage module, configured to store the new webpage data in response to the filtering coefficient being greater than a preset filtering coefficient.

[0014] According to another aspect of the present invention, a webpage data processing system is also provided, comprising: a monitoring module, configured to acquire new webpage data of a preset webpage in response to a change in the webpage data of the preset webpage; a filtering module, connected to the monitoring module, configured to process the title content and body content of the new webpage data respectively to obtain a keyword set corresponding to the title content and a text vector corresponding to the body content, and to determine a filtering coefficient of the new webpage data based on the keyword set and the text vector, wherein the filtering coefficient is used to characterize the degree of matching between the new webpage data and preset filtering conditions; and a database module, configured to store the new webpage data in response to the filtering coefficient being greater than a preset filtering coefficient.

[0015] According to another aspect of the present invention, a computer-readable storage medium is also provided, the computer-readable storage medium including a stored program, wherein, when the program is executed, it controls the device where the computer-readable storage medium is located to perform any of the methods described in the above embodiments.

[0016] According to another aspect of the present invention, a processor is also provided, which is used to run a program, wherein the program executes the method of any one of the above embodiments when it runs.

[0017] In this embodiment of the invention, in response to a change in the webpage data of a preset webpage, new webpage data of the preset webpage is obtained. The title content and body content of the new webpage data are processed separately to obtain a keyword set corresponding to the title content and a text vector corresponding to the body content. Then, a filtering coefficient for the new webpage data is determined based on the keyword set and the text vector. The filtering coefficient is used to characterize the degree of matching between the new webpage data and the preset filtering conditions. Furthermore, in response to a filtering coefficient greater than the preset filtering coefficient, the new webpage data is stored. It should be noted that determining the filtering coefficient based on the keyword set and the text vector makes the filtering coefficient more accurate, thereby improving the matching degree of the filtered text and achieving the technical effect of a high degree of matching between the filtered text and the data, thus solving the technical problem of low matching degree of the filtered text. Attached Figure Description

[0018] The accompanying drawings, which are included to provide a further understanding of the invention and form part of this application, illustrate exemplary embodiments of the invention and, together with their description, serve to explain the invention and do not constitute an undue limitation thereof. In the drawings:

[0019] Figure 1 This is a flowchart of a web page data processing method according to an embodiment of the present invention;

[0020] Figure 2 This is a schematic diagram of an optional network processing module according to an embodiment of the present invention;

[0021] Figure 3 This is a flowchart of an optional overall web page data processing method according to an embodiment of the present invention;

[0022] Figure 4 This is a schematic diagram of a web page data processing apparatus according to an embodiment of the present invention. Detailed Implementation

[0023] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0024] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0025] Example 1

[0026] According to an embodiment of the present invention, an embodiment of a method for processing web page data is provided. It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions. Furthermore, although a logical order is shown in the flowchart, in some cases, the steps shown or described may be executed in a different order than that shown here.

[0027] Figure 1 This is a flowchart of a webpage data processing method according to an embodiment of the present invention, such as... Figure 1 As shown, the method includes the following steps:

[0028] Step S102: In response to a change in the webpage data of the preset webpage, obtain the new webpage data of the preset webpage.

[0029] The aforementioned preset webpage can be a webpage pre-set for processing webpage data, and can be, but is not limited to, webpages used to publish information about the power industry and official policy announcements. Webpage data can be data from the current preset webpage, and new webpage data can be new data resulting from changes to the preset webpage's data.

[0030] In one optional embodiment, new webpage data can be obtained in real time by subscribing to RSS (Really Simple Syndication) messages. When the webpage data in a preset webpage changes, the new webpage data corresponding to the preset webpage with the changed webpage data can be obtained in real time.

[0031] Step S104: Process the title content and body content in the new webpage data separately to obtain the keyword set corresponding to the title content and the text vector corresponding to the body content.

[0032] The title content mentioned above can be related to the title in the webpage data. The body content can be related to the body content in the webpage data. The keyword set can be a collection of all relevant keywords in the title content. The text vector can be a numerical vector representing the text of the body content.

[0033] In one optional embodiment, new webpage data is filtered to remove invalid information from updated webpages, obtaining text to be filtered. The title content is then filtered to obtain first text. This first text is then segmented using a word segmentation algorithm to obtain a keyword set. The main text is then segmented using the same algorithm to obtain second text. The words are further processed using a term frequency-inverse document frequency algorithm to obtain initial word weights. The first weight parameter is then obtained according to the following formula:

[0034] Where h represents the page number of the segmented text, page g (t) represents the number of t items contained in page g, numpage g Let represent the total number of words on page t, and obtain the second weight parameter according to the following formula:

[0035] Where p represents the line number of the first page in the segmented text, and num q (t) represents the number of t items in the q-th row, num q This represents the total number of this graph contained in the q-th row. The third weight parameter is calculated using the following formula:

[0036] Where num(t) represents the number of word segments t contained in the segmented text, num alpg This represents the total number of all word segments. η represents a preset constant parameter, and Ф represents a control parameter, Ф∈(0.2,0.8). The initial weights are adjusted according to the first weight parameter, the second weight parameter, and the third weight parameter to obtain the weights of words in the second text, thus obtaining the vector representation of the second text. Further, based on the weights and vector representation, the text vector is determined.

[0037] Step S106: Based on the keyword set and text vector, determine the filtering coefficient of the new webpage data, whereby the filtering coefficient is used to characterize the degree of matching between the new webpage data and the preset filtering conditions.

[0038] The filtering coefficients mentioned above can be the matching coefficients after filtering new webpage data. Preset filtering conditions can be conditions that are preset for filtering as needed. Matching degree can be the degree of match between the new webpage data and the preset filtering conditions.

[0039] In one optional embodiment, a preset keyword set and a preset text vector set are obtained, and the number of keywords in the keyword set is determined to obtain a first number. Each keyword in the keyword set is then matched with the preset keyword set to obtain a score corresponding to each keyword. These scores are summed to obtain a total score. The first number is then multiplied by the preset score to obtain a target score. The total score is then compared with the target total score to obtain a first coefficient for the keyword set. A second number corresponding to the preset text vectors in the preset text vector set is determined, and the similarity between the text vectors and the preset text vectors to multiple preset text vectors is obtained, resulting in multiple similarities. These multiple similarities are summed to obtain a total similarity. The second number is then multiplied by the preset similarity to obtain a target similarity. The total similarity is then compared with the target similarity to obtain a second coefficient for the text vectors, calculated according to the following formula: The selection coefficient is obtained by weighting the first coefficient and the second coefficient, where simldx represents the selection coefficient, λ represents the weight coefficient, λ∈(0,1), and vat u This represents the score of word 'u' in bstu. If word 'u' exists in the preset title keyword set tlkyset, then vat... u The score is 1, otherwise, vat u The score is 0, nbstu represents the total number of words contained in bstu, aplst represents the standard value of the average score of the preset title keywords, and sim v,webvr This indicates the relevance between the text vector webvr and the element v in the set bstv, where bstv represents the set of text vectors of templates for the main text content of the preset nbstv types, and webst represents the preset relevance standard value.

[0040] Step S108: In response to the filtering coefficient being greater than the preset filtering coefficient, the new webpage data is stored.

[0041] The preset screening coefficients mentioned above can be preset according to the actual situation.

[0042] In one optional embodiment, when the filtering coefficient is greater than the preset filtering coefficient, it indicates that the text to be filtered meets the filtering requirements, and the database module stores the text that meets the filtering requirements.

[0043] It should be noted that due to the specific nature of the power industry, the database module is typically hosted on a local server. This allows power company users to access this data without needing an external network connection.

[0044] Through the above steps, in response to changes in the webpage data of a preset webpage, new webpage data can be obtained. The title and body text of the new webpage data are then processed separately to obtain a keyword set corresponding to the title and a text vector corresponding to the body text. A filtering coefficient is then determined based on the keyword set and text vector. This filtering coefficient characterizes the degree of matching between the new webpage data and the preset filtering conditions. If the filtering coefficient is greater than the preset filtering coefficient, the new webpage data is stored. It should be noted that determining the filtering coefficient based on the keyword set and text vector makes the filtering coefficient more accurate, thereby improving the matching degree of the filtered text. This achieves the technical effect of a higher matching degree between the filtered text and the data, thus solving the technical problem of low matching degree of the filtered text.

[0045] Optionally, the filtering coefficients for new webpage data are determined based on a keyword set and text vectors, including: obtaining a preset keyword set and a preset text vector set; matching each keyword in the keyword set with the preset keyword set to obtain a first coefficient for the keyword set, wherein the first coefficient is used to characterize the degree of matching between the keyword set and the preset keyword set; matching the text vectors with the preset text vector set to obtain a second coefficient for the text vectors, wherein the second coefficient is used to characterize the degree of matching between the text vectors and the preset text vector set; and weighting the first coefficient and the second coefficient to obtain the filtering coefficients.

[0046] The aforementioned preset keyword set can be a set of keywords pre-defined as needed. The preset text vector set can be a set of text vectors pre-defined as needed.

[0047] The first coefficient mentioned above can be the keyword set matching coefficient obtained by matching the obtained keyword set with a preset keyword set. The second coefficient can be the text vector matching coefficient obtained by matching the obtained text vector with a preset text vector set.

[0048] In one optional embodiment, a preset keyword set and a preset text vector set are obtained. Each keyword in the keyword set is matched with the preset keyword set to obtain a first coefficient of the keyword set, and the text vectors are matched with the preset text vector set to obtain a second coefficient of the text vectors. The coefficients are calculated according to the following formula: The first coefficient and the second coefficient are weighted and summed to obtain the screening coefficient.

[0049] It should be noted that in setting the calculation functions, this invention assigns different calculation weights to the calculation results of the title and the body text, thereby improving the accuracy of the calculation results. `tlkyset` contains various pre-set keywords. For example, for policy information, keywords such as "regulations," "trial implementation," and "notice" can be set. For industry dynamic information, keywords such as "conference," "meeting," and "opening" can be set. `bstv` mainly consists of different webpage templates. These templates, based on the type of information, set different vectors of frequently occurring sentences and words, thereby improving the accuracy of the results obtained from the filtering coefficients.

[0050] Further, each keyword in the keyword set is matched with a preset keyword set to obtain a first coefficient for the keyword set, including: determining the number of keywords in the keyword set to obtain a first number; matching each keyword in the keyword set with the preset keyword set to obtain a score for each keyword, where the score is used to represent whether each keyword is successfully matched with the preset keyword set; obtaining the sum of the scores of all keywords in the keyword set to obtain a total score; obtaining the product of the first number and the preset score to obtain a target score, where the preset score is used to represent the average score of all preset keywords in the preset keyword set; and obtaining the ratio of the total score to the target total score to obtain the first coefficient.

[0051] The first quantity mentioned above can be the number of keywords in the keyword set. The score can be the sum of the matching scores obtained after each keyword matches the preset keyword set. The preset score can be the average of the scores corresponding to the preset keywords. The target score can be the score obtained by multiplying the number of keywords by the preset score.

[0052] In one optional embodiment, the number of keywords in the keyword set (nbstu) is determined, and each keyword in the keyword set is matched with a preset keyword set to obtain a score for each keyword, and the sum of the scores of all keywords is obtained. Further multiply the number of keywords (nbstu) by the preset score (aplst) to obtain the target score, according to the following formula: Obtain the ratio of the total score to the target total score, and determine the first coefficient.

[0053] Optionally, matching the text vector with a preset set of text vectors to obtain a second coefficient of the text vector includes: determining the number of preset text vectors in the preset set of text vectors to obtain a second number, wherein different preset text vectors in the preset set of text vectors are used to represent templates of different types of body text content; obtaining the similarity between the text vector and multiple preset text vectors in the preset set of text vectors to obtain multiple similarities; obtaining the sum of multiple similarities to obtain a total similarity; obtaining the product of the second number and the preset similarities to obtain a target similarity; and obtaining the ratio of the total similarity to the target similarity to obtain a second coefficient.

[0054] The second quantity mentioned above can be the number of preset text vectors in the preset text vector set.

[0055] The aforementioned similarity can be the similarity between a text vector and multiple preset text vectors, where the total similarity can be the sum of multiple similarities. The target similarity can be the similarity between the number of preset text vectors and the preset similarity, where the preset similarity can be the average of the corresponding similarities of the preset text vectors.

[0056] In one optional embodiment, the number of preset text vectors in the preset text vector set is determined, and the similarity between the text vectors and multiple preset text vectors in the preset text vector set is obtained, and the sum of multiple similarity scores is obtained. The second quantity is further multiplied by the preset similarity to obtain the target similarity, according to the formula: The ratio of total similarity to target similarity is used as the second coefficient.

[0057] Optionally, the title content and body content in the new webpage data are processed separately to obtain a keyword set corresponding to the title content and a text vector corresponding to the body content, including: filtering the new webpage data to obtain the text to be filtered; performing word segmentation on the title content in the text to be filtered to obtain a keyword set; and performing semantic analysis on the body content in the text to be filtered to obtain a text vector.

[0058] The text to be filtered mentioned above can be text that has not yet been filtered, or text that has had invalid information removed from the new webpage data and has not been filtered.

[0059] In one optional embodiment, the filtering module removes invalid information from the new webpage data to obtain the text to be filtered, and determines whether the text to be filtered meets the filtering requirements. Text that meets the filtering requirements is then sent to the database module. The title in the text to be filtered is segmented to obtain a set of title keywords, and the body content in the text to be filtered is semantically analyzed to obtain the text vector of the body content.

[0060] It should be noted that the invalid information can include, but is not limited to: content unrelated to the text such as advertisements, pop-ups, videos, etc.

[0061] Furthermore, perform word segmentation on the title content in the text to be screened to obtain a keyword set, including: filtering the title content to obtain a first text; using a word segmentation algorithm to perform word segmentation on the first text to obtain a keyword set.

[0062] The above-mentioned first text can be the text after filtering the title. The word segmentation algorithm can be a mechanical word segmentation method based on rules. Mainly by maintaining a dictionary, when splitting a sentence, each string of the sentence is matched one by one with the words in the word list, and if found, it is split. It can be, but is not limited to: forward maximum matching method, backward maximum matching method, and bidirectional maximum matching method.

[0063] In an optional embodiment, remove modal particles, adverbs, prepositions, and conjunctions in the title to obtain a sentence to be segmented, use a word segmentation algorithm to perform word segmentation on the sentence to be segmented, and store the obtained words in the set of title keywords.

[0064] It should be noted that modal particles are a type of particle, which are function words used in a sentence to indicate pauses and emphasize the tone at the end of the sentence. When read in a sentence, they are not translated and represent a slow pause. When emphasizing the tone at the end of the sentence or enhancing the momentum through repeated parallel sentences. Modal particles include "ah, ah, la, alas, ne, ba, le, wa, ya, ma, o, ha, yo, me", etc. An adverb is a word that expresses the characteristics of an action or state in a sentence. It is used to modify a verb, an adjective, another adverb, or the whole sentence to express concepts such as time, place, degree, and manner. Adverbs can be divided into time adverbs, frequency adverbs, position adverbs, manner adverbs, degree adverbs, interrogative adverbs, conjunctive adverbs, relative adverbs, demonstrative order adverbs, and demonstrative completion adverbs. Adverbs include very, absolutely, also, all, first, second, surely, exactly, etc. A preposition is a word or affix used to express the grammatical function of a character in grammar. It is generally placed in front of a pronoun or a noun phrase, and forms a prepositional structure with these words to express location, time, state, manner, reason, purpose, comparison object, etc. The original meaning of the sentence will change after removing the preposition. A conjunction is a function word used to connect words and words, phrases and phrases, or sentences and sentences, and express a certain logical relationship. Conjunctions can express relationships such as parallelism, transition, hypothesis, choice, progress, condition, causality, etc. Common conjunctions include and, with, and, then, and, then, etc.

[0065] Optionally, perform semantic analysis on the body content in the text to be screened to obtain a text vector, including: using a word segmentation algorithm to perform word segmentation on the body content to obtain a second text; determining the weight of the words in the second text in the second text; obtaining the vector representation of the second text; based on the weight and the vector representation, determining the text vector.

[0066] The semantic analysis described above can be a logical stage in the compilation process, performing context-dependent property checks and type checks on structurally correct source programs. The second text can be the text obtained after word segmentation of the main content. The weight can be the relative importance of words in the second text compared to the main text.

[0067] In one optional embodiment, the main text is segmented using a word segmentation algorithm to obtain a second text. The weight of each word segment in the segmented text is calculated using the following function: Where nft represents the number of words t in the segmented text, nfalwrd represents the total number of words in each segment, tfidf(t) represents the weight value of word t calculated using the TF-IDF algorithm, Ea(f) represents the first weight parameter of word t, Eb(t) represents the second weight parameter of word t, Ec(t) represents the third weight parameter of word t, and divweb represents the set of all words in the segmented text. The vector representation of the segmented text can be obtained using the Word2Vec (Word to Vectot, a group of related models used to generate word vectors) algorithm. The text vector webvr of the main content is then calculated based on the weights and vector representation. The specific calculation formula is as follows: Among them, webvct s Let s represent the text vector, wgt(s) represent the weight of the text vector s, nfwrd represent the number of distinct words in the segmented text, and vct represent the weight of the text vector s. v Let w represent the v-th dimension of the text vector s, and w represent the total number of dimensions of the vector.

[0068] It's important to note that obtaining the weights of word segments is a prerequisite for subsequent text vector calculation. Generally, weights are calculated directly using the TF-IDF algorithm. However, the TF-IDF algorithm doesn't consider the different frequencies of word segmentation across different pages and within the same page; it directly calculates the frequency of occurrence across all words. If the number of word segments 't' on page i is 'numi', and the number of word segments 't' on the remaining pages is 'numot', where 'numi' >> 'numot', then the total number of word segments 't' is 'numi' + 'numot'. However, according to the TF-IDF algorithm's calculation method, this can lead to an underestimation of the value of word segment 't'. Therefore, we need to consider the occurrence of word segment 't' across different pages. This invention, in calculating weights, considers not only the TF-IDF algorithm's results but also the distribution of word segment 't' across different pages and within the first page. A first weight parameter reflects the distribution of word segment 't' across different pages, while second and third weight parameters reflect its distribution within the first page, thus improving the accuracy of the calculated weights.

[0069] In addition, by calculating the text vector of each word separately, the text vector of the entire main text can be composed of the text vectors of all words in WebVR.

[0070] Further, determining the weights of words in the second text includes: processing the words using a term frequency-inverse document frequency algorithm to obtain initial weights; determining a first weight parameter for the words based on the first word located on a different page in the second text; determining a second weight parameter for the words based on the second word located on a different line in the second text; determining a third weight parameter for the words based on all words in the second text; and adjusting the initial weights based on the first, second, and third weight parameters to obtain the weights of the words in the second text.

[0071] The term frequency-inverse document frequency (TFM) algorithm described above is a feature vectorization algorithm that calculates the general importance of a word. The initial weights can be the weights obtained after processing words using the TFM algorithm.

[0072] The first word mentioned above can be a word located on a different page in the second text. The first weight parameter can be the weight parameter of the word relative to its corresponding word on a different page in the second text. The second word can be a word located on a different line in the second text. The second weight parameter can be the weight parameter of the word relative to its corresponding word on a different line in the second text. The third weight parameter can be the weight parameter of the word relative to all words in the second text.

[0073] In one optional embodiment, a first weight parameter is determined based on the first word in the second text located on a different page, using the following formula:

[0074] The second weight parameter of a word is determined based on the word and the second word in the second text that are located on different lines. The specific formula is as follows:

[0075] The third weight parameter of the word is determined based on all words in the second text and the word itself. The specific formula is as follows:

[0076]

[0077] It should be noted that the first page typically features a summary of policies or industry information; therefore, its content is quite important. Two weighted parameters are used to reflect the distribution of word segmentation 't'. The second and third weighted parameters are calculated based on the distribution of word segmentation 't' on the first page, providing a comprehensive picture of the word segmentation 't' distribution.

[0078] In addition, in this embodiment of the invention, a schematic diagram of the network processing module is shown below. Figure 2 As shown, the analysis module in this invention is used to analyze text stored in the database. The analysis module may include a login unit, a display unit, and an input unit. The login unit is used to identify the employee's identity and determine whether the employee can use the analysis module. The display unit is used to display the information retrieved from the database module by the identified employee. The input unit is used by the identified employee to input the analysis results. The login unit identifies the employee's identity through facial recognition, including: acquiring the employee's facial image; determining whether the facial image meets the requirements; if it does, performing facial recognition on the employee based on the facial image; if it does not, re-acquiring the facial image. Determining whether the facial image meets the requirements requires calculating comprehensive parameters of the facial image, using the following formula: Where zesidx represents the synthesis parameters, fcset represents the set of pixels in the face image, and gry z Gs represents the grayscale value of pixel z. z This represents the result of calculating pixel z using the Gaussian filtering algorithm, where thrz represents the preset first comparison coefficient, and dtl... zThe gradient value of pixel z in the horizontal direction is represented by NFC, the total number of pixels in the face image, and NFCE, the total number of pixels in the skin region of the face image. d1, d2, and d3 represent three set scaling factors, and vst represents the set standard deviation of the gradient value. If the comprehensive parameter is greater than the set parameter threshold, the face image meets the requirements; if the comprehensive parameter is less than or equal to the set parameter threshold, the face image does not meet the requirements. In the process of face recognition, this invention does not directly recognize the face image after acquisition, but first calculates the comprehensive parameter to determine whether it meets the requirements. This invention only performs face recognition on face images that meet the requirements, thereby improving the efficiency of face recognition. If the face image does not meet the requirements, recognition may fail, requiring the acquisition of a new face image. However, face recognition requires processing time, which reduces its efficiency and results in poor identity recognition performance of the detection and analysis system of this invention. Therefore, the above-mentioned setting of this invention can effectively improve the efficiency of face recognition. In calculating the comprehensive coefficient, this invention comprehensively calculates it from several aspects, including the variance of the differences in pixel values ​​after Gaussian processing, the variance of the gradient values, and the number of pixels belonging to the skin region. The smaller the variance of the differences in pixel values ​​after Gaussian processing, the larger the variance of the gradient values, and the more pixels belonging to the skin region, the larger the comprehensive coefficient. The above-described embodiments of this invention can accurately select facial images that meet the requirements.

[0079] The following is combined Figure 3 A preferred embodiment of the present invention will be described in detail, such as... Figure 3 As shown, the title of the text to be filtered is segmented to obtain the set of title keywords bstu. The body content of the text to be filtered is semantically analyzed to obtain the text vector webvr of the body content. Based on bstu and webvr, the filtering coefficient is calculated. If the filtering coefficient is greater than the set filtering coefficient threshold value, it means that the text to be filtered meets the filtering requirements. If the filtering coefficient is less than or equal to the set filtering coefficient threshold value, it means that the text to be filtered does not meet the filtering requirements.

[0080] Example 2

[0081] According to another aspect of the present invention, a web page data processing apparatus is also provided. This apparatus can execute the vehicle control method described in the above embodiments. The specific implementation and preferred application scenarios are the same as those described in the above embodiments, and will not be repeated here.

[0082] Figure 4 This is a schematic diagram of a web page data processing apparatus according to an embodiment of the present invention, such as... Figure 4As shown, the device includes the following components: an acquisition module 40, a processing module 42, a determination module 44, and a storage module 46.

[0083] The acquisition module 40 is used to acquire new webpage data of the preset webpage in response to changes in the webpage data of the preset webpage.

[0084] Processing module 42 is used to process the title content and body content in the new web page data respectively to obtain the keyword set corresponding to the title content and the text vector corresponding to the body content;

[0085] The determination module 44 is used to determine the filtering coefficient of new web page data based on the keyword set and text vector, wherein the filtering coefficient is used to characterize the degree of matching between the new web page data and the preset filtering conditions;

[0086] Storage module 46 is used to store new webpage data in response to a filtering coefficient being greater than a preset filtering coefficient.

[0087] Optionally, the determining module includes: a first acquisition unit, used to acquire a preset keyword set and a preset text vector set; a first matching unit, used to match each keyword in the keyword set with the preset keyword set to obtain a first coefficient of the keyword set, wherein the first coefficient is used to characterize the degree of matching between the keyword set and the preset keyword set; a second matching unit, used to match the text vector with the preset text vector set to obtain a second coefficient of the text vector, wherein the second coefficient is used to characterize the degree of matching between the text vector and the preset text vector set; and a second acquisition unit, used to perform a weighted sum of the first coefficient and the second coefficient to obtain a filtering coefficient.

[0088] The first matching unit includes: a first determining subunit, used to determine the number of keywords in the keyword set, obtaining a first quantity; a first matching subunit, used to match each keyword in the keyword set with a preset keyword set, obtaining a score for each keyword, wherein the score is used to represent whether each keyword is successfully matched with the preset keyword set; a first obtaining subunit, used to obtain the sum of the scores of all keywords in the keyword set, obtaining a total score; a second obtaining subunit, used to obtain the product of the first quantity and the preset score, obtaining a target score, wherein the preset score is used to represent the average score of all preset keywords in the preset keyword set; and a third obtaining subunit, used to obtain the ratio of the total score to the target total score, obtaining a first coefficient.

[0089] The second matching unit includes: a second determining subunit, used to determine the number of preset text vectors in a preset text vector set to obtain a second number, wherein different preset text vectors in the preset text vector set are used to represent the text vectors of templates for different types of main text content; a fourth obtaining subunit, used to obtain the similarity between a text vector and multiple preset text vectors in the preset text vector set to obtain multiple similarities; a fifth obtaining subunit, used to obtain the sum of multiple similarities to obtain a total similarity; a sixth obtaining subunit, used to obtain the product of the second number and the preset similarities to obtain a target similarity; and a seventh obtaining subunit, used to obtain the ratio of the total similarity to the target similarity to obtain a second coefficient.

[0090] The processing module includes: a first filtering unit for filtering new webpage data to obtain text to be filtered; a first processing unit for performing word segmentation on the title content of the text to be filtered to obtain a keyword set; and a first analysis unit for performing semantic analysis on the body content of the text to be filtered to obtain a text vector.

[0091] The first processing unit includes a filtering subunit for filtering the title content to obtain the first text; and a first processing subunit for segmenting the first text using a word segmentation algorithm to obtain a keyword set.

[0092] The first filtering unit includes: a second processing subunit, used to segment the main text using a word segmentation algorithm to obtain the second text; a third determining subunit, used to determine the weights of words in the second text; an eighth obtaining subunit, used to obtain the vector representation of the second text; and a fourth determining subunit, used to determine the text vector based on the weights and vector representation.

[0093] The third determining subunit also includes: processing words using the term frequency-inverse document frequency algorithm to obtain the initial weight of the words; determining the first weight parameter of the words based on the first word located on a different page in the second text; determining the second weight parameter of the words based on the second word located on a different line in the second text; determining the third weight parameter of the words based on all words in the second text; and adjusting the initial weights based on the first, second, and third weight parameters to obtain the weight of the words in the second text.

[0094] Example 3

[0095] According to another aspect of the present invention, a webpage data processing apparatus is also provided, comprising: an acquisition module, configured to acquire new webpage data of the preset webpage in response to a change in the webpage data of the preset webpage; a processing module, configured to process the title content and body content of the new webpage data respectively to obtain a keyword set corresponding to the title content and a text vector corresponding to the body content; a determination module, configured to determine a filtering coefficient of the new webpage data based on the keyword set and the text vector, wherein the filtering coefficient is used to characterize the degree of matching between the new webpage data and preset filtering conditions; and a storage module, configured to store the new webpage data in response to the filtering coefficient being greater than a preset filtering coefficient.

[0096] Example 4

[0097] According to another aspect of the present invention, a computer-readable storage medium is also provided, the computer-readable storage medium including a stored program, wherein, when the program is executed, it controls the device where the computer-readable storage medium is located to perform any of the methods described in the above embodiments.

[0098] Example 5

[0099] According to another aspect of the present invention, a processor is also provided, which is used to run a program, wherein the program executes the method of any one of the above embodiments when it runs.

[0100] The sequence numbers of the above embodiments of the present invention are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0101] In the above embodiments of the present invention, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0102] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. The device embodiments described above are merely illustrative; for example, the division of units can be a logical functional division, and in actual implementation, there may be other division methods. For instance, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the displayed or discussed mutual coupling, direct coupling, or communication connection may be through some interfaces; the indirect coupling or communication connection between units or modules may be electrical or other forms.

[0103] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0104] Furthermore, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0105] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, read-only memory (ROM), random access memory (RAM), portable hard drives, magnetic disks, or optical disks.

[0106] The above description is only a preferred embodiment of the present invention. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.

Claims

1. A method for processing webpage data, characterized in that, include: In response to a change in the webpage data of a preset webpage, the new webpage data of the preset webpage is obtained; The title content and body content in the new webpage data are processed separately to obtain the keyword set corresponding to the title content and the text vector corresponding to the body content; Based on the keyword set and the text vector, a filtering coefficient for the new webpage data is determined, wherein the filtering coefficient is used to characterize the degree of matching between the new webpage data and the preset filtering conditions; In response to the filtering coefficient being greater than a preset filtering coefficient, the new webpage data is stored; Determining the filtering coefficient for the new webpage data based on the keyword set and the text vector includes: obtaining a preset keyword set and a preset text vector set; matching each keyword in the keyword set with the preset keyword set to obtain a first coefficient for the keyword set, wherein the first coefficient is used to characterize the degree of matching between the keyword set and the preset keyword set; matching the text vector with the preset text vector set to obtain a second coefficient for the text vector, wherein the second coefficient is used to characterize the degree of matching between the text vector and the preset text vector set; and performing a weighted sum of the first coefficient and the second coefficient to obtain the filtering coefficient.

2. The method according to claim 1, characterized in that, Matching each keyword in the keyword set with the preset keyword set to obtain a first coefficient of the keyword set includes: Determine the number of keywords in the keyword set to obtain a first number; Each keyword in the keyword set is matched with the preset keyword set to obtain a score for each keyword, wherein the score is used to characterize whether each keyword is successfully matched with the preset keyword set; The total score is obtained by summing the scores of all keywords in the keyword set. The target score is obtained by multiplying the first quantity by a preset score, wherein the preset score is used to represent the average score of all preset keywords in the preset keyword set; The ratio of the total score to the target total score is obtained to obtain the first coefficient.

3. The method according to claim 1, characterized in that, Matching the text vector with the preset text vector set to obtain the second coefficient of the text vector includes: The number of preset text vectors in the preset text vector set is determined to obtain a second number, wherein different preset text vectors in the preset text vector set are used to represent the text vectors of templates for different types of body text content; The similarity between the text vector and multiple preset text vectors among the preset text vectors is obtained to obtain multiple similarity scores; The sum of the multiple similarities is obtained to get the total similarity. The target similarity is obtained by multiplying the second quantity and the preset similarity. The ratio of the total similarity to the target similarity is obtained to obtain the second coefficient.

4. The method according to claim 1, characterized in that, The title content and body content of the new webpage data are processed separately to obtain the keyword set corresponding to the title content and the text vector corresponding to the body content, including: The new webpage data is filtered to obtain the text to be filtered; The title content in the text to be filtered is segmented into words to obtain the keyword set; Semantic analysis is performed on the main text content of the text to be filtered to obtain the text vector.

5. The method according to claim 4, characterized in that, The title content in the text to be filtered is segmented into words to obtain the keyword set, including: The title content is filtered to obtain the first text; The first text is segmented using a word segmentation algorithm to obtain the keyword set.

6. The method according to claim 4, characterized in that, Semantic analysis is performed on the main text content of the text to be filtered to obtain the text vector, including: The main text is segmented using a word segmentation algorithm to obtain the second text; Determine the weights of words in the second text. Obtain the vector representation of the second text; The text vector is determined based on the weights and the vector representation.

7. The method according to claim 6, characterized in that, Determining the weights of words in the second text includes: The initial weights of the words are obtained by processing the words using the term frequency-inverse document frequency algorithm. Based on the word and the first word in the second text located on different pages, a first weight parameter for the word is determined; Based on the word and a second word located on a different line in the second text, a second weight parameter for the word is determined; Based on the stated word and all words in the second text, a third weight parameter for the stated word is determined; The initial weights are adjusted based on the first weight parameter, the second weight parameter, and the third weight parameter to obtain the weight of the word in the second text.

8. A web page data processing device, characterized in that, include: The acquisition module is used to acquire new webpage data of the preset webpage in response to a change in the webpage data of the preset webpage; The processing module is used to process the title content and body content in the new webpage data respectively to obtain the keyword set corresponding to the title content and the text vector corresponding to the body content; A determining module is used to determine the filtering coefficient of the new webpage data based on the keyword set and the text vector, wherein the filtering coefficient is used to characterize the degree of matching between the new webpage data and the preset filtering conditions; The storage module is used to store the new webpage data in response to the filtering coefficient being greater than a preset filtering coefficient; The determining module is further configured to obtain a preset keyword set and a preset text vector set; match each keyword in the keyword set with the preset keyword set to obtain a first coefficient of the keyword set, wherein the first coefficient is used to characterize the degree of matching between the keyword set and the preset keyword set; match the text vector with the preset text vector set to obtain a second coefficient of the text vector, wherein the second coefficient is used to characterize the degree of matching between the text vector and the preset text vector set; and perform a weighted sum of the first coefficient and the second coefficient to obtain the filtering coefficient.

9. A web page data processing system, characterized in that, include: The monitoring module is used to obtain new webpage data of the preset webpage in response to changes in the webpage data of the preset webpage; A filtering module, connected to the monitoring module, is used to process the title content and body content in the new webpage data respectively to obtain a keyword set corresponding to the title content and a text vector corresponding to the body content, and to determine the filtering coefficient of the new webpage data based on the keyword set and the text vector, wherein the filtering coefficient is used to characterize the degree of matching between the new webpage data and the preset filtering conditions; A database module is used to store the new webpage data in response to the filtering coefficient being greater than a preset filtering coefficient; The filtering module is further configured to obtain a preset keyword set and a preset text vector set; match each keyword in the keyword set with the preset keyword set to obtain a first coefficient of the keyword set, wherein the first coefficient is used to characterize the degree of matching between the keyword set and the preset keyword set; match the text vector with the preset text vector set to obtain a second coefficient of the text vector, wherein the second coefficient is used to characterize the degree of matching between the text vector and the preset text vector set; and perform a weighted sum of the first coefficient and the second coefficient to obtain the filtering coefficient.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium includes a stored program, wherein, when the program is executed, it controls the device on which the computer-readable storage medium is located to perform the method according to any one of claims 1 to 7.

11. A processor, characterized in that, The processor is used to run a program, wherein the program executes the method according to any one of claims 1 to 7 when it runs.

Images