Distributed acquisition method based on webpage big data

Abstract

The embodiment of the invention relates to the field of data collection, in particular to a distributed collection method based on webpage big data. A specific embodiment of the method comprises the following steps: a data processing subsystem obtains webpage key information sent by a user terminal; the data retrieval subsystem in the at least one data retrieval subsystem responds to the received webpage key information sent by the data processing subsystem and retrieves the webpage key information to generate an initial data retrieval information set; the data processing subsystem performs data cleaning processing on each piece of initial data retrieval information in the at least one initial data retrieval information set in response to the received at least one initial data retrieval information set sent by the at least one data retrieval subsystem; and the data processing subsystem performs sorting processing on the data retrieval cleaning information set to generate a data retrieval sorting information sequence. According to the embodiment, the waste of webpage pushing resources can be reduced.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of data collection, and specifically to a distributed collection method based on web page big data. Background Art

[0002] Web big data can be, but is not limited to: web data of movie information, web data of book information. The processing of web big data mainly involves data collection, processing, analysis and visualization. On the technical level, this includes the use of various technologies such as Flume, Sqoop, Kettle for data collection, HDFS, HBase, MongoDB, etc. for data storage, and Spark technology stack for data analysis. In addition, data visualization is usually achieved through technologies such as HTML, CSS, JavaScript, etc., so that data can be displayed on a large screen in the form of charts, images, etc., which is convenient for monitoring, analysis and display of data.

[0003] ‌By sending web page information with high relevance to key web page information to user terminals through search engines, web page information that meets user needs can be sent to user terminals. At present, when collecting data, the commonly used method is to collect web page information stored in a single server through a single search engine, and then send the collected web page information to the user terminal.

[0004] However, in practice, it is found that when the above method is used for data collection, the following technical problems often occur: First, the web page information collected by a single search engine is relatively single, which makes it difficult for the collected web page information to meet user needs, making it difficult to send web page information that meets user needs to user terminals, wasting web page push resources; In the process of adopting technical solutions to solve the above-mentioned technical problem one, the following problems often occur: when searching each web page information stored by different search engines through a full-text search algorithm, since the web page information contains a lot of information and the amount of web page information is large, it is necessary to consume a lot of computing resources to traverse all the information contained in each web page information, resulting in a waste of computing resources. In response to the above problems, the conventional solution is generally: by splitting the web page information and constructing a web page information graph, there is no need to traverse the entire web page information. However, the above-mentioned conventional solution still has the following technical problem two: since the amount of web page information retrieved by each search engine is large, and some of the retrieved web page information does not meet user needs, when the user terminal views the web page information, it is difficult to directly view the web page information that meets the user's needs, wasting web page push resources; Third, sending web page information that is not sorted according to user needs to the user terminal will make it difficult for the user terminal to view the web page information that best meets the user's needs, wasting web page push resources. Summary of the invention

[0005] The content of this application is used to introduce concepts in a brief form, which will be described in detail in the detailed implementation section below. The content of this application is not intended to identify the key features or essential features of the technical solution claimed for protection, nor is it intended to limit the scope of the technical solution claimed for protection.

[0006] Some embodiments of the present application propose a distributed collection method based on web page big data to solve one or more of the technical problems mentioned in the above background technology section.

[0007] In a first aspect, some embodiments of the present application provide a distributed collection method based on web page big data, which is applied to a data collection system, wherein the data collection system includes: a data processing subsystem, at least one data retrieval subsystem, and a memory. The method includes: the data processing subsystem obtains key information of the web page sent by the user terminal; the data processing subsystem sends the key information of the web page to the at least one data retrieval subsystem; the data retrieval subsystem in the at least one data retrieval subsystem retrieves and processes the key information of the web page in response to receiving the key information of the web page sent by the data processing subsystem to generate an initial data retrieval information set, and sends the initial data retrieval information set to the storage device. to the above-mentioned data processing subsystem; in response to receiving at least one initial data retrieval information set sent from the above-mentioned at least one data retrieval subsystem, the above-mentioned data processing subsystem performs data cleaning processing on each initial data retrieval information in the above-mentioned at least one initial data retrieval information set to generate data retrieval cleaning information and obtain a data retrieval cleaning information set; the above-mentioned data processing subsystem performs sorting processing on the above-mentioned data retrieval cleaning information set to generate a data retrieval sorting information sequence, and sends the above-mentioned data retrieval sorting information sequence to the above-mentioned user terminal; the above-mentioned data processing subsystem sends the above-mentioned data retrieval sorting information sequence to the above-mentioned memory to store the above-mentioned data retrieval sorting information sequence in the above-mentioned memory.

[0008] The above-mentioned embodiments of the present application have the following beneficial effects: through the distributed collection method based on web page big data of some embodiments of the present application, the waste of web page push resources can be reduced. Specifically, the reason for wasting web page push resources is that the web page information collected by a single search engine is relatively single, which makes it difficult for the collected web page information to meet the user's needs, and it is difficult to send web page information that meets the user's needs to the user terminal. Based on this, in the distributed collection method based on web page big data of some embodiments of the present application, first, the data processing subsystem obtains the key information of the web page sent by the user terminal. Thus, the key information of the web page can be obtained so that the web page information that meets the user's needs can be retrieved according to the key information of the web page. Secondly, the data processing subsystem sends the key information of the web page to the at least one data retrieval subsystem. Then, the data retrieval subsystem in the at least one data retrieval subsystem responds to receiving the key information of the web page sent by the data processing subsystem, retrieves and processes the key information of the web page to generate an initial data retrieval information set, and sends the initial data retrieval information set to the data processing subsystem. Thus, different data retrieval subsystems can retrieve the initial data retrieval information sets that meet the user's needs under different search engines. Then, in response to receiving at least one initial data retrieval information set sent by the at least one data retrieval subsystem, the data processing subsystem performs data cleaning processing on each initial data retrieval information in the at least one initial data retrieval information set to generate data retrieval cleaning information and obtain a data retrieval cleaning information set. Thus, a data retrieval cleaning information set that meets user needs after data cleaning can be obtained. Afterwards, the data processing subsystem performs sorting processing on the data retrieval cleaning information set to generate a data retrieval sorting information sequence, and sends the data retrieval sorting information sequence to the user terminal. Thus, a data retrieval information sequence that meets user needs after sorting can be obtained. Finally, the data processing subsystem sends the data retrieval sorting information sequence to the memory to store the data retrieval sorting information sequence in the memory. Thus, the data retrieval sorting information sequence can be stored in the memory, and when the webpage key information sent by the user terminal is the same, the data retrieval sorting information sequence stored in the memory can be directly called to reduce the waiting time of the user terminal. Therefore, the initial data retrieval information set can be collected from different search engines by at least one data retrieval subsystem, and the initial data retrieval information set can be sorted to obtain a data retrieval sorting information sequence that meets user needs. Thus, a data retrieval sorting information sequence that meets the user's needs can be sent to the user terminal, thereby reducing the waste of web page push resources. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] The above and other features, advantages and aspects of the embodiments of the present application will become more apparent with reference to the following detailed description in conjunction with the accompanying drawings. Throughout the accompanying drawings, the same or similar reference numerals represent the same or similar elements. It should be understood that the drawings are schematic and that components and elements are not necessarily drawn to scale.

[0010] Figure 1 is a flow chart of some embodiments of the distributed collection method based on web page big data according to the present application; Figure 2 It is a schematic diagram of the structure of some embodiments of the data acquisition system according to the present application. DETAILED DESCRIPTION

[0011] Embodiments of the present application will be described in more detail below with reference to the accompanying drawings. Although certain embodiments of the present application are shown in the accompanying drawings, it should be understood that the present application can be implemented in various forms and should not be construed as being limited to the embodiments set forth herein. On the contrary, these embodiments are provided to provide a more thorough and complete understanding of the present application. It should be understood that the drawings and embodiments of the present application are only for exemplary purposes and are not intended to limit the scope of protection of the present application.

[0012] It should also be noted that, for ease of description, only the parts related to the invention are shown in the drawings. In the absence of conflict, the embodiments and features in the embodiments of the present application can be combined with each other.

[0013] It should be noted that the concepts such as "first" and "second" mentioned in this application are only used to distinguish different devices, modules or units, and are not used to limit the order or interdependence of the functions performed by these devices, modules or units.

[0014] It should be noted that the modifications of "one" and "plurality" mentioned in the present application are illustrative rather than restrictive, and those skilled in the art should understand that unless otherwise clearly indicated in the context, it should be understood as "one or more".

[0015] The names of the messages or information exchanged between multiple devices in the embodiments of the present application are only used for illustrative purposes and are not used to limit the scope of these messages or information.

[0016] The present application will be described in detail below with reference to the accompanying drawings and in combination with embodiments.

[0017] Figure 1 The process 100 of some embodiments of the distributed collection method based on web page big data according to the present application is shown. The distributed collection method based on web page big data is applied to a data collection system and includes the following steps: Step 101: The data processing subsystem obtains key information of a web page sent by a user terminal.

[0018] In some embodiments, the data processing subsystem can obtain the key information of the web page sent by the user terminal from the storage terminal through a wired connection or a wireless connection. The key information of the web page can represent the keywords of the information that the user terminal wants to view. For example, the key information of the web page can be: information representing the name of a movie, information representing the name of a book, and information representing the name of a website. Here, the execution subject of the distributed collection method based on web page big data can be a data collection system.

[0019] Here, the data acquisition system can refer to Figure 2 The structure diagram of some embodiments of the data acquisition system 200 is shown. Figure 2 As shown, the data acquisition system 200 includes: a data processing subsystem 201, at least one data retrieval subsystem 202, and a memory 203. The at least one data retrieval subsystem 202 may include: a data retrieval subsystem 2021 and a data retrieval subsystem 2022. For example, the data processing subsystem 201 may be a central processing unit. The memory 203 may be, but is not limited to: a random access memory, a read-only memory. The data retrieval subsystem 2021 in the at least one data retrieval subsystem 202 may be: a system with a search engine. The data retrieval subsystem 2022 in the at least one data retrieval subsystem 202 may be: a system with a search engine.

[0020] It should be noted that the above wireless connection methods may include but are not limited to 3G / 4G connection, WiFi connection, Bluetooth connection, WiMAX connection, Zigbee connection, UWB (ultra wideband) connection, and other wireless connection methods currently known or to be developed in the future.

[0021] Step 102: The data processing subsystem sends the key information of the web page to at least one data retrieval subsystem.

[0022] In some embodiments, the data processing subsystem may send the webpage key information to the at least one data retrieval subsystem. In practice, the data processing subsystem may send the webpage key information to each data retrieval subsystem in the at least one data retrieval subsystem.

[0023] Step 103, in response to receiving the web page key information sent by the data processing subsystem, the data retrieval subsystem in at least one data retrieval subsystem retrieves and processes the web page key information to generate an initial data retrieval information set, and sends the initial data retrieval information set to the data processing subsystem.

[0024] In some embodiments, the data retrieval subsystem in the at least one data retrieval subsystem mentioned above can, in response to receiving the web page key information sent by the above-mentioned data processing subsystem, retrieve and process the above-mentioned web page key information to generate an initial data retrieval information set, and can send the above-mentioned initial data retrieval information set to the above-mentioned data processing subsystem.

[0025] In practice, in response to receiving the webpage key information sent by the data processing subsystem, the data retrieval subsystem in the at least one data retrieval subsystem may perform retrieval processing on the webpage key information through the following steps to generate an initial data retrieval information set: The first step is to perform data retrieval processing on the above web page key information to generate an initial web page retrieval information set.

[0026] In practice, the data retrieval subsystem may perform data retrieval processing on the above webpage key information through the following sub-steps to generate an initial webpage retrieval information set: The first sub-step is to obtain an initial web page information set. In practice, the data retrieval subsystem may obtain the initial web page information set from the terminal device through a wired connection or a wireless connection. The initial web page information in the initial web page information set may represent a web page including key web page information. For example, the initial web page information in the initial web page information set may represent web page big data (e.g., web page data of movie information, web page data of book information).

[0027] The second sub-step is to perform feature extraction processing on each initial web page information in the initial web page information set to generate initial feature web page information and obtain the initial feature web page information set. In practice, the data retrieval subsystem can perform feature extraction processing on each initial web page information in the initial web page information set to generate initial feature web page information by the following extraction steps: In the first extraction step, the initial web page information is split and processed to generate an initial web page split information set. In practice, the data retrieval subsystem can split and process the initial web page information using a preset splitting algorithm to generate an initial web page split information set. For example, the preset splitting algorithm can be: a word segmentation algorithm (e.g., a maximum matching algorithm, a shortest path algorithm). Here, the initial web page split information in the initial web page split information set can represent words.

[0028] The second extraction step is to construct a web page information graph based on the above-mentioned initial web page splitting information set. In practice, the data retrieval subsystem can construct a web page information graph based on the above-mentioned initial web page splitting information set by a preset construction algorithm. For example, the preset construction algorithm can be: using the initial web page splitting information in the initial web page splitting information set as a node, and using the connection line between two initial web page splitting information in the initial web page splitting information set that meets the preset connection condition as an edge to construct the web page information graph. For example, the preset connection condition can be: two initial web page splitting information in the initial web page splitting information set are adjacent in the initial web page information.

[0029] The third extraction step is to determine the initial web page splitting weight set corresponding to the initial web page splitting information set based on the web page information graph. The initial web page splitting weights in the initial web page splitting weight set may correspond to the initial web page splitting information in the initial web page splitting information set. In practice, based on the web page information graph, the data retrieval subsystem may determine the initial web page splitting weight set corresponding to the initial web page splitting information set by a preset weight algorithm. For example, the preset weight algorithm may be a PageRank algorithm.

[0030] In the fourth extraction step, each initial web page splitting information corresponding to each initial web page splitting weight satisfying the preset weight condition in the initial web page splitting weight set is determined as the initial feature web page information. The preset weight condition may be: the initial web page splitting weight is greater than or equal to the preset web page weight. For example, the preset web page weight may be 0.8.

[0031] The third sub-step is to perform the following determination steps for each initial feature web page information in the initial feature web page information set: The first determination step is to determine the initial similarity information corresponding to the initial feature web page information and the web page key information. In practice, the data retrieval subsystem can determine the initial similarity information corresponding to the initial feature web page information and the web page key information by using a preset similarity algorithm. For example, the preset similarity algorithm can be: TF-IDF (TermFrequency-Inverse Document Frequency) algorithm.

[0032] In the second determination step, in response to determining that the initial similarity information satisfies a preset similarity condition, the initial web page information corresponding to the initial similarity information is determined as the initial web page search information. The preset similarity condition may be: the value represented by the initial similarity information is greater than a preset similarity value. For example, the preset similarity value may be 0.8.

[0033] The fourth sub-step is to determine the determined individual initial web page search information as an initial web page search information set.

[0034] The second step is to perform feature extraction processing on each initial web page search information in the initial web page search information set to generate web page feature information and obtain a web page feature information set. In practice, the specific implementation method and technical effects of performing feature extraction processing on each initial web page search information in the initial web page search information set to generate web page feature information and obtain a web page feature information set can refer to step 103 in the above embodiment, which will not be repeated here.

[0035] The third step is to input the above web page key information and the above web page feature information set into a pre-trained web page feature ranking model to obtain a web page feature ranking information sequence. Among them, the pre-trained web page feature ranking model can be a pre-trained model that takes the web page key information and the web page feature information set as input and takes the web page feature ranking information sequence as output. The web page feature ranking information in the above web page feature ranking information sequence can include: data retrieval weight. Here, the arrangement order of the web page feature ranking information in the web page feature ranking information sequence can be in descending order according to the data retrieval weight. The data retrieval weight included in the web page feature ranking information in the web page feature ranking information sequence can represent: the similarity between the web page feature information corresponding to the web page feature ranking information and the web page key information.

[0036] Step 4: determine each initial web page search information corresponding to each web page feature sorting information satisfying the preset sorting condition in the web page feature sorting information sequence as the initial data search information set. The preset sorting condition may be: the first preset sorting number of web page feature sorting information in the web page feature sorting information sequence. For example, the preset sorting number may be 5.

[0037] Optionally, before inputting the webpage key information and the webpage feature information set into a pre-trained webpage feature ranking model to obtain a webpage feature ranking information sequence, the method further includes: The first step is to obtain the training sample set.

[0038] In some embodiments, the data retrieval subsystem may obtain the training sample set through a wired connection or a wireless connection. The training samples in the training sample set include: sample web page key information, sample web page feature information set and sample web page feature ranking information sequence.

[0039] The second step is to determine the initial web page feature ranking model.

[0040] In some embodiments, the data retrieval subsystem may determine an initial web page feature ranking model, wherein the initial web page feature ranking model may include: an initial clustering model, an initial weight model, and an initial ranking model.

[0041] Here, the initial clustering model may be a model that takes the sample web page feature information set as input and takes the initial clustering feature information set as output. For example, the initial clustering model may be a Gaussian mixture model.

[0042] The initial weight model may be a model that takes the initial clustering feature information and the sample webpage key information as input and takes the initial feature information weight as output. For example, the initial weight model may be a text similarity algorithm (e.g., cosine similarity algorithm, Pearson correlation coefficient algorithm, standard Euclidean distance algorithm) model. Here, the initial feature information weight may represent the similarity between the initial clustering feature information and the sample webpage key information.

[0043] The initial sorting model may be a model that takes the initial feature information weight set and the sample web page feature information set as inputs, and takes the initial web page feature sorting information sequence as output. For example, the initial sorting model is used to: first, the initial feature information weight set may be sorted by a preset feature sorting algorithm to generate an initial feature information sorting weight sequence. Then, for each initial feature information sorting weight in the initial feature information sorting weight sequence, the sample web page feature information corresponding to the initial feature information sorting weight in the sample web page feature information set is determined as the initial web page feature sorting information. Finally, the determined initial web page feature sorting information is determined as the initial web page feature sorting information sequence. For example, the preset feature sorting algorithm may be, but is not limited to, a linear regression model, a decision tree model, and a quick sorting algorithm model.

[0044] The third step is to select target training samples from the above training sample set.

[0045] In some embodiments, the data retrieval subsystem may randomly select a target training sample from the training sample set.

[0046] The fourth step is to input the sample web page feature information set included in the selected target training sample into the initial clustering model to obtain an initial clustering feature information set.

[0047] In some embodiments, the data retrieval subsystem may input the sample web page feature information set included in the selected target training sample into the initial clustering model to obtain an initial clustering feature information set.

[0048] In the fifth step, for each initial clustering feature information group in the initial clustering feature information group set, each initial clustering information satisfying a preset clustering condition in the initial clustering feature information group is determined as the initial clustering feature information group.

[0049] In some embodiments, the data retrieval subsystem may determine the initial clustering information in the initial clustering feature information group that satisfies a preset clustering condition as the initial clustering feature information for each initial clustering feature information group in the initial clustering feature information group set. The preset clustering condition may be: the initial clustering feature information is the initial clustering feature information in the initial clustering feature information group that represents the cluster center.

[0050] In the sixth step, each determined initial clustering feature information is determined as an initial clustering feature information set.

[0051] In some embodiments, the data retrieval subsystem may determine the determined individual pieces of initial clustering feature information as an initial clustering feature information set.

[0052] In the seventh step, each initial clustering feature information and the key information of the sample web page in the initial clustering feature information set are input into the initial weight model to generate the initial feature information weights and obtain the initial feature information weight set.

[0053] In some embodiments, the data retrieval subsystem may input each initial clustering feature information and sample web page key information in the initial clustering feature information set into an initial weight model to generate initial feature information weights and obtain an initial feature information weight set.

[0054] In the eighth step, the initial feature information weight set and the sample web page feature information set are input into the initial sorting model to obtain the initial web page feature sorting information sequence.

[0055] In some embodiments, the data retrieval subsystem may input the initial feature information weight set and the sample web page feature information set into an initial ranking model to obtain an initial web page feature ranking information sequence.

[0056] In the ninth step, based on a preset loss function, a difference value between the initial web page feature ranking information sequence and the sample web page feature ranking information sequence included in the selected target training sample is determined.

[0057] In some embodiments, based on a preset loss function, the data retrieval subsystem can determine the difference between the initial web page feature sorting information sequence and the sample web page feature sorting information sequence included in the selected target training sample. The preset loss function can be, but is not limited to: mean square error loss function (MSE), hinge loss function (SVM), cross entropy loss function (CrossEntropy), 0-1 loss function, absolute value loss function, log loss function, square loss function, exponential loss function, etc.

[0058] In a tenth step, in response to determining that the difference value is greater than or equal to a preset difference value, adjusting network parameters of the initial web page feature ranking model.

[0059] In some embodiments, the data retrieval subsystem may adjust the network parameters of the initial web page feature ranking model in response to determining that the difference value is greater than or equal to the preset difference value. For example, the difference between the difference value and the preset difference value may be calculated. On this basis, the network parameters of the initial malicious code detection model are adjusted using methods such as back propagation and gradient descent. There is no limitation on the setting of the preset difference value, for example, the preset difference value may be 0.1.

[0060] Optionally, in response to determining that the difference value is less than the preset difference value, the initial web page feature ranking model is determined as the trained web page feature ranking model.

[0061] In some embodiments, in response to determining that the difference value is less than the preset difference value, the data retrieval subsystem may determine the initial web page feature ranking model as the trained web page feature ranking model.

[0062] The relevant content of step 103, as an inventive point of an embodiment of the present application, solves the second technical problem mentioned in the background technology, "wasting web page push resources". Among them, the factors that waste web page push resources are often as follows: since the amount of web page information retrieved by each search engine is large, and part of the retrieved web page information does not meet the user's needs, it is difficult for the user terminal to directly view the web page information that meets the user's needs when viewing the web page information. If the above factors are solved, the effect of reducing the waste of web page push resources can be achieved. In order to achieve this effect, the present application firstly trains the initial clustering model to obtain the initial clustering feature information group set after clustering, so as to screen out the initial clustering feature information representing the cluster center. Then, by training the initial weight information, the initial feature information weight representing the similarity between the initial clustering feature information and the web page key information can be obtained. Then, by training the initial sorting model, the sample web page feature information set can be sorted according to the initial feature information weight set to generate an initial web page feature sorting information sequence. Therefore, a relatively accurate web page feature ranking model can be obtained by training an initial web page feature ranking model including an initial clustering model, an initial weight model, and an initial ranking model, so that each initial cluster feature information representing the cluster center can be retrieved from the web page feature information set through the relatively accurate web page feature ranking model, and each web page feature information corresponding to each initial cluster feature information is sorted to obtain a web page feature ranking information sequence. Thus, the web page feature ranking information in the obtained web page feature ranking information sequence all meets the user's needs. Furthermore, through the web page feature ranking information that meets the user's needs, the web page information that meets the user's needs can be sent to the user terminal, which can reduce the waste of web page push resources.

[0063] Step 104, the data processing subsystem, in response to receiving at least one set of initial data retrieval information sent by at least one data retrieval subsystem, performs data cleaning processing on each piece of initial data retrieval information in the at least one set of initial data retrieval information to generate data retrieval cleaning information, thereby obtaining a data retrieval cleaning information set.

[0064] In some embodiments, the above-mentioned data processing subsystem, in response to receiving at least one set of initial data retrieval information sent by the at least one data retrieval subsystem, may perform data cleaning processing on each piece of initial data retrieval information in the at least one set of initial data retrieval information to generate data retrieval cleaning information, thereby obtaining a data retrieval cleaning information set.

[0065] In practice, the above-mentioned data processing subsystem may perform data cleaning processing on each piece of initial data retrieval information in the at least one set of initial data retrieval information through the following steps to generate data retrieval cleaning information: First step, perform screening processing on the above-mentioned initial data retrieval information to generate initial data screening information. In practice, the above-mentioned data processing subsystem may remove the information indicating emptiness in the above-mentioned initial data retrieval information to perform screening processing on the above-mentioned initial data retrieval information and obtain initial data screening information.

[0066] Second step, perform data standardization processing on the above-mentioned initial data screening information to generate initial data standardization information. In practice, the above-mentioned data processing subsystem may perform data standardization processing on the initial data screening information representing the initial format through a preset data standardization algorithm to generate initial data standardization information representing the target format. For example, the preset data standardization algorithm may be: standardization algorithm, normalization algorithm, centering algorithm. Here, the initial format may represent Chinese, English, French. The target format may represent Chinese.

[0067] Third step, perform text cleaning processing on the above-mentioned initial data standardization information to generate data retrieval cleaning information. In practice, the above-mentioned data processing subsystem may perform text cleaning processing on the above-mentioned initial data standardization information through a preset text cleaning algorithm to generate data retrieval cleaning information. For example, the preset text cleaning algorithm may be a regular expression algorithm. Thus, through the regular expression algorithm, punctuation marks, stop words (e.g., "de", "shi"), and the case and spelling errors in the initial data standardization information can be removed.

[0068] Step 105, the data processing subsystem performs sorting processing on the data retrieval cleaning information set to generate a data retrieval sorting information sequence, and sends the data retrieval sorting information sequence to the user terminal.

[0069] In some embodiments, the data processing subsystem performs sorting processing on the data retrieval cleaning information set to generate a data retrieval sorting information sequence, and sends the data retrieval sorting information sequence to the user terminal.

[0070] In practice, the data processing subsystem may sort the data retrieval cleaning information set to generate a data retrieval sorting information sequence by the following steps: The first step is to vectorize each data retrieval cleaning information in the above data retrieval cleaning information set to generate a data retrieval cleaning vector and obtain a data retrieval cleaning vector set. In practice, the above data processing subsystem can vectorize each data retrieval cleaning information in the above data retrieval cleaning information set through a preset vectorization algorithm to generate a data retrieval cleaning vector and obtain a data retrieval cleaning vector set. For example, the preset vectorization algorithm can be: One-Hot encoding algorithm, TF-IDF (Term Frequency–Inverse Document Frequency) algorithm, Word2vec (word to vector) algorithm.

[0071] The second step is to select a data center vector group from the data retrieval cleaning vector set. In practice, the data processing subsystem may randomly select a preset number of data retrieval cleaning vectors from the data retrieval cleaning vector set as the data center vector group. For example, the preset number of centers may be 5.

[0072] The third step is to determine the initial data clustering vector group corresponding to each data center vector in the data center vector group to obtain an initial data clustering vector group set. In practice, first, for each data center vector in the data center vector group, the data processing subsystem may determine the data center vector as the initial data clustering vector group. Then, the data processing subsystem may determine the determined initial data clustering vector groups as the initial data clustering vector group set.

[0073] In the fourth step, the data retrieval cleaning vector set with the data center vector group removed is determined as the data removal vector set.

[0074] In the fifth step, based on the data removal vector set and the data center vector group, the following processing sub-steps are performed: In the first sub-step, for each data removal vector in the data removal vector set, perform the following adding steps: The first adding step is to determine, for each data center vector in the data center vector group, a data center distance value between the data center vector and the data removal vector. In practice, for each data center vector in the data center vector group, the data processing subsystem may determine the data center distance value between the data center vector and the data removal vector by using a preset distance algorithm. For example, the preset distance algorithm may be: a Euclidean distance algorithm.

[0075] The second adding step adds the data removal vector to the data center vector corresponding to the smallest data center distance value among the determined data center distance values ​​and the corresponding initial data clustering vector group, so as to update the initial data clustering vector group.

[0076] The second sub-step is to determine each updated initial data clustering vector group as a data clustering update vector group set.

[0077] The third sub-step is to determine the data center update vector corresponding to each data cluster update vector group in the data cluster update vector group set to obtain the data center update vector group. The data center update vector in the data center update vector group corresponds to the data cluster update vector group in the data cluster update vector group set. In practice, the above-mentioned data processing subsystem can determine the data center update vector corresponding to each data cluster update vector group in the data cluster update vector group set by a preset center algorithm to obtain the data center update vector group. For example, the preset center algorithm can be: the mean of each data cluster update vector in the data cluster update vector group is determined as the data center update vector.

[0078] In a fourth sub-step, in response to determining that the data cluster update vector set and the data center update vector set meet a preset matching condition, the data cluster update vector set is determined as a target data cluster vector set. The preset matching condition may be: for each data cluster update vector in the data cluster update vector set, the update distance value between the data cluster update vector and the data center update vector corresponding to the data cluster update vector is the minimum value of each update distance value between the data cluster update vector and each data center update vector in the data center update vector set. Here, the update distance value may be the distance value between the data cluster update vector and the data center vector determined by the above-mentioned preset distance algorithm.

[0079] Step 6: For each target data clustering vector group in the target data clustering vector group set, the target data clustering vector in the target data clustering vector group that meets the preset center condition is determined as the target data clustering selection vector. The preset center condition may be: the distance value between the target data clustering vector and the data center update vector corresponding to the target data clustering vector is less than a preset distance value. For example, the preset distance value may be 50.

[0080] In the seventh step, each determined target data cluster selection vector is determined as a target data cluster selection vector set.

[0081] The eighth step is to determine the data retrieval weight corresponding to each target data cluster selection vector in the target data cluster selection vector set to obtain a data retrieval weight set. In practice, first, for each target data cluster selection vector in the target data cluster selection vector set, the data processing subsystem may determine the data retrieval weight including the initial data retrieval information corresponding to the target data cluster selection vector and the corresponding web page feature ranking information. Then, the data processing subsystem may determine the determined data retrieval weights as a data retrieval weight set.

[0082] The ninth step is, in response to determining that the data retrieval weight set satisfies the preset splitting condition, sorting the data retrieval weight set to generate a data retrieval sorting weight sequence. The preset splitting condition may be: the number of data retrieval weights in the data retrieval weight set is a preset number. For example, the preset number may be 2. In practice, in response to determining that the data retrieval weight set satisfies the preset splitting condition, the data processing subsystem may sort the data retrieval weight set using a first preset sorting algorithm to generate a data retrieval sorting weight sequence. For example, the first preset sorting algorithm may be: arranging each data retrieval weight in the data retrieval weight set in order from large to small.

[0083] Step 10: In response to determining that the data retrieval weight set does not satisfy the above preset splitting condition, the following sorting steps are performed: In the first sorting step, the data retrieval weight set is split to generate an initial data split weight group set. In practice, the data processing subsystem can split the data retrieval weight set by a preset weight splitting algorithm to generate an initial data split weight group set. For example, the preset weight splitting algorithm can be: binary division.

[0084] The second sorting step is to sort the initial data splitting weight group in response to determining that the initial data splitting weight group in the initial data splitting weight group set meets the above-mentioned preset splitting condition, so as to generate an initial data sorting weight sequence. In practice, in response to determining that the initial data splitting weight group in the initial data splitting weight group set meets the above-mentioned preset splitting condition, the data processing subsystem can sort the initial data splitting weight group by a second preset sorting algorithm to generate an initial data sorting weight sequence. For example, the second preset sorting algorithm can be: arranging each initial data splitting weight in the initial data splitting weight group in order from large to small.

[0085] In a third sorting step, in response to determining that the initial data splitting weight group set satisfies the preset data splitting condition, each generated initial data sorting weight sequence is sorted to generate a data retrieval sorting weight sequence. The preset data splitting condition may be that each initial data splitting weight group in the initial data splitting weight group set satisfies the preset splitting condition.

[0086] In the eleventh step, the data retrieval cleaning information corresponding to each data retrieval ranking weight in the generated data retrieval ranking weight sequence is determined as the data retrieval ranking information to obtain a data retrieval ranking information sequence.

[0087] Optionally, in response to determining that the data cluster update vector group set and the data center update vector group do not satisfy the above-mentioned preset matching condition, the data cluster update vector group set with the data center update vector group removed is determined as the data removal vector set, and the data center update vector group is determined as the data center vector group, so as to perform the above-mentioned processing steps again.

[0088] In some embodiments, in response to determining that the data cluster update vector group set and the data center update vector group do not satisfy the preset matching condition, the data processing subsystem determines the data cluster update vector group set with the data center update vector group removed as the data removal vector set, and determines the data center update vector group as the data center vector group, so as to execute the above processing steps again.

[0089] Optionally, in response to determining that the initial data splitting weight group in the initial data splitting weight group set does not meet the above-mentioned preset splitting condition, the initial data splitting weight group in the initial data splitting weight group set that does not meet the preset splitting condition is used as a data retrieval weight set for performing the above-mentioned sorting step again.

[0090] In some embodiments, in response to determining that an initial data splitting weight group in the initial data splitting weight group set does not satisfy the above-mentioned preset splitting condition, the above-mentioned data processing subsystem uses the initial data splitting weight group in the initial data splitting weight group set that does not satisfy the preset splitting condition as a data retrieval weight set for performing the above-mentioned sorting step again.

[0091] The relevant content of step 105, as an inventive point of an embodiment of the present application, solves the technical problem three mentioned in the background technology, "waste of web page push resources". Among them, the factors that waste web page push resources are often as follows: sending web page information that is not sorted according to user needs to the user terminal will make it difficult for the user terminal to view the web page information that best meets the user's needs. If the above factors are solved, the effect of reducing the waste of web page push resources can be achieved. In order to achieve this effect, the present application can first convert the data retrieval cleaning information set into a data retrieval cleaning vector set through a vectorization algorithm, so as to subsequently cluster the data retrieval cleaning vector set. Then, a data center vector group can be randomly selected, and each data retrieval cleaning vector in the data retrieval cleaning vector set can be added to the initial data clustering vector group corresponding to the nearest data center vector to generate an updated data clustering update vector group to obtain a data clustering update vector group set. Then, the data center update vector group representing the cluster center in each data cluster update vector group is re-determined, and each data cluster update vector is redistributed to the cluster corresponding to the data center update vector closest to the cluster center, until all data cluster update vectors are in the cluster closest to the cluster center, and the target data cluster vector group set is obtained. Then, each target data cluster selection vector set that meets the preset center condition can be selected to obtain a target data cluster selection vector set that better meets user needs. Then, the data retrieval weight set corresponding to the target data cluster selection vector set can be determined to obtain the data retrieval weight representing the correlation between the target data cluster selection vector and the key information of the web page. Then, the data retrieval weights can be arranged in order from large to small to obtain a data retrieval sorting weight sequence, so as to obtain a sorted data retrieval sorting information sequence. Therefore, the data retrieval cleaning information sets retrieved by different search engines can be clustered by a clustering algorithm to screen out target data cluster selection vectors that are more in line with user needs, and the data retrieval cleaning information sets retrieved by different search engines can be sorted according to the data retrieval weight set that represents the relevance between the data retrieval cleaning information and the key information of the web page, so as to arrange the data retrieval cleaning information that best meets the user's needs at the front to obtain a data retrieval sorting information sequence. Thus, the data retrieval sorting information sequence sorted according to the user's needs can be sent to the user terminal, which can facilitate the user terminal to view the web page information that best meets the user's needs. Furthermore, the waste of web page push resources can be reduced.

[0092] Step 106: The data processing subsystem sends the data retrieval sorting information sequence to the memory to store the data retrieval sorting information sequence in the memory.

[0093] In some embodiments, the data processing subsystem sends the data retrieval sorting information sequence to the memory to store the data retrieval sorting information sequence in the memory.

[0094] The above-mentioned embodiments of the present application have the following beneficial effects: through the distributed collection method based on web page big data of some embodiments of the present application, the waste of web page push resources can be reduced. Specifically, the reason for wasting web page push resources is that the web page information collected by a single search engine is relatively single, which makes it difficult for the collected web page information to meet the user's needs, and it is difficult to send web page information that meets the user's needs to the user terminal. Based on this, in the distributed collection method based on web page big data of some embodiments of the present application, first, the data processing subsystem obtains the key information of the web page sent by the user terminal. Thus, the key information of the web page can be obtained so that the web page information that meets the user's needs can be retrieved according to the key information of the web page. Secondly, the data processing subsystem sends the key information of the web page to the at least one data retrieval subsystem. Then, the data retrieval subsystem in the at least one data retrieval subsystem responds to receiving the key information of the web page sent by the data processing subsystem, retrieves and processes the key information of the web page to generate an initial data retrieval information set, and sends the initial data retrieval information set to the data processing subsystem. Thus, different data retrieval subsystems can retrieve the initial data retrieval information sets that meet the user's needs under different search engines. Then, in response to receiving at least one initial data retrieval information set sent by the at least one data retrieval subsystem, the data processing subsystem performs data cleaning processing on each initial data retrieval information in the at least one initial data retrieval information set to generate data retrieval cleaning information and obtain a data retrieval cleaning information set. Thus, a data retrieval cleaning information set that meets user needs after data cleaning can be obtained. Afterwards, the data processing subsystem performs sorting processing on the data retrieval cleaning information set to generate a data retrieval sorting information sequence, and sends the data retrieval sorting information sequence to the user terminal. Thus, a data retrieval information sequence that meets user needs after sorting can be obtained. Finally, the data processing subsystem sends the data retrieval sorting information sequence to the memory to store the data retrieval sorting information sequence in the memory. Thus, the data retrieval sorting information sequence can be stored in the memory, and when the webpage key information sent by the user terminal is the same, the data retrieval sorting information sequence stored in the memory can be directly called to reduce the waiting time of the user terminal. Therefore, the initial data retrieval information set can be collected from different search engines by at least one data retrieval subsystem, and the initial data retrieval information set can be sorted to obtain a data retrieval sorting information sequence that meets user needs. Thus, a data retrieval sorting information sequence that meets the user's needs can be sent to the user terminal, thereby reducing the waste of web page push resources. The technical features of the above embodiments can be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0095] The above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, a person skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features may be replaced by equivalents. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the protection scope of the technical solutions of the embodiments of the present application.

Claims

1. A distributed collection method based on web page big data, applied to a data collection system, the data collection system include: A data processing subsystem, at least one data retrieval subsystem, a memory, comprising: The data processing subsystem obtains key information of the web page sent by the user terminal; The data processing subsystem sends the webpage key information to the at least one data retrieval subsystem; The data retrieval subsystem in the at least one data retrieval subsystem performs retrieval processing on the webpage key information in response to receiving the webpage key information sent by the data processing subsystem to generate an initial data retrieval information set, and sends the initial data retrieval information set to the data processing subsystem; In response to receiving at least one initial data retrieval information set sent by the at least one data retrieval subsystem, the data processing subsystem performs data cleaning processing on each initial data retrieval information in the at least one initial data retrieval information set to generate data retrieval cleaning information and obtain a data retrieval cleaning information set; The data processing subsystem performs sorting processing on the data retrieval cleaning information set to generate a data retrieval sorting information sequence, and sends the data retrieval sorting information sequence to the user terminal; The data processing subsystem sends the data retrieval sorting information sequence to the memory to store the data retrieval sorting information sequence in the memory.

2. According to claim 1, the distributed collection method based on web page big data, in, The retrieving and processing the webpage key information to generate an initial data retrieval information set includes: Performing data retrieval processing on the webpage key information to generate an initial webpage retrieval information set; Performing feature extraction processing on each initial web page search information in the initial web page search information set to generate web page feature information, thereby obtaining a web page feature information set; Inputting the web page key information and the web page feature information set into a pre-trained web page feature ranking model to obtain a web page feature ranking information sequence; The initial web page search information corresponding to each web page feature sorting information satisfying the preset sorting condition in the web page feature sorting information sequence is determined as an initial data search information set.

3. According to claim 2, the distributed collection method based on web page big data, in, The performing data retrieval processing on the webpage key information to generate an initial webpage retrieval information set includes: Get the initial web page information set; Performing feature extraction processing on each initial web page information in the initial web page information set to generate initial feature web page information, thereby obtaining an initial feature web page information set; For each initial characteristic web page information in the initial characteristic web page information set, the following determination steps are performed: Determine initial similarity information corresponding to the initial feature web page information and the web page key information; In response to determining that the initial similarity information satisfies a preset similarity condition, determining the initial web page information corresponding to the initial similarity information as initial web page search information; The determined pieces of initial web page search information are determined as an initial web page search information set.

4. The distributed collection method based on web page big data according to claim 3, in, The step of performing feature extraction processing on each initial web page information in the initial web page information set to generate initial feature web page information includes: Splitting the initial web page information to generate an initial web page split information set; Based on the initial web page splitting information set, construct a web page information graph; Based on the web page information graph, determining an initial web page splitting weight set corresponding to the initial web page splitting information set, wherein the initial web page splitting weights in the initial web page splitting weight set correspond to the initial web page splitting information in the initial web page splitting information set; The initial web page splitting information corresponding to each initial web page splitting weight satisfying the preset weight condition in the initial web page splitting weight set is determined as the initial characteristic web page information.

5. The distributed collection method based on web page big data according to claim 1, in, The step of performing data cleaning on each piece of initial data retrieval information in the at least one initial data retrieval information set to generate data retrieval cleaning information includes: Screening the initial data retrieval information to generate initial data screening information; Performing data standardization processing on the initial data screening information to generate initial data standardization information; The initial data standardization information is subjected to text cleaning processing to generate data retrieval cleaning information.

Images