Automatic crawling method of website data based on Internet of Things

Abstract

The present invention relates to the technical field of data compression, and in particular to an automatic website data crawling method based on the Internet of Things. The method obtains data information to be crawled in a website and constructs a two-dimensional matrix, divides the two-dimensional matrix into at least one submatrix, obtains the data complexity of the submatrix, and performs assimilation processing on elements in a target submatrix according to the data complexity of each submatrix to obtain a target two-dimensional matrix; scans the elements in the target two-dimensional matrix using at least two preset scanning methods to obtain a corresponding one-dimensional data sequence, obtains the optimal scanning degree according to the length of continuous identical data in each sequence, and then obtains an optimal scanning method, performs run-length coding compression on the corresponding target data sequence that can be maximally compressed obtained by scanning the target two-dimensional matrix using the optimal scanning method, obtains data information in the website by automatically crawling the compressed data and restoring the crawled data, so as to achieve the effect of maximizing the data compression.

Description

Technical Field

[0001] The present invention relates to the technical field of data compression, and in particular to an automatic crawling method for website data based on the Internet of Things. Background Art

[0002] Automated data crawling, also known as crawler, is an automated program used to access Internet resources, usually used for search engines or data collection. It starts from one or more starting web pages, automatically discovers and accesses linked web pages, thereby crawling web page content, and then storing the crawled information in the computer.

[0003] When automatically crawling web pages, a large amount of data will be obtained, including the website's user visits, click-through rate, behavioral data, etc. In order to crawl data efficiently, the data to be crawled must first be compressed and preprocessed. The commonly used compression method is run-length encoding compression. The advantages of this method are low computational complexity and fast decompression process. However, for data with few repeated values, the compression of run-length encoding is low and may even cause the compressed data volume to be larger than the original data volume, resulting in reduced crawling efficiency.

[0004] Therefore, how to maximize data compression and improve the efficiency of automated data crawling has become an urgent problem that needs to be solved. Summary of the invention

[0005] In view of this, an embodiment of the present invention provides a method for automatically crawling website data based on the Internet of Things to solve the problem of maximizing data compression and improving the efficiency of automatically crawling data.

[0006] An embodiment of the present invention provides a method for automatically crawling website data based on the Internet of Things, the method comprising the following steps:

[0007] Acquire data information to be crawled from the website, perform base conversion on the data information to be crawled, obtain a corresponding numerical sequence, and construct a two-dimensional matrix of the numerical sequence;

[0008] Divide the two-dimensional matrix into at least one submatrix of a preset size, obtain the data complexity of each submatrix according to the element difference between the two-dimensional matrix and each submatrix, and perform an assimilation operation on the elements in the two-dimensional matrix according to the data complexity of each submatrix to obtain a target two-dimensional matrix;

[0009] For any scanning mode among at least two preset scanning modes, use the scanning mode to scan the elements in the target two-dimensional matrix to obtain a corresponding one-dimensional data sequence, obtain the scanning optimality of the scanning mode according to the continuous identical data in the one-dimensional data sequence, and obtain the optimal scanning mode according to the scanning optimality of each scanning mode;

[0010] The elements in the target two-dimensional matrix are scanned using the optimal scanning method to obtain a target data sequence, the target data sequence is run-length encoded and compressed to obtain corresponding compressed data, and the crawled compressed data obtained by automatically crawling the compressed data is decompressed and restored to obtain the crawled data information in the website.

[0011] Preferably, obtaining the data complexity of each submatrix according to the element difference between the two-dimensional matrix and each submatrix includes:

[0012] For any submatrix, according to the difference in elements and the difference in the number of elements between the two-dimensional matrix and the submatrix, obtain the discreteness of the submatrix;

[0013] Obtaining the degree of deviation of the submatrix according to the difference between each element in the submatrix;

[0014] The data complexity of the submatrix is ​​obtained according to the discreteness and the deviation of the submatrix.

[0015] Preferably, obtaining the discreteness of the submatrix according to the difference in elements and the difference in the number of elements between the two-dimensional matrix and the submatrix includes:

[0016] Obtaining a first difference between a maximum element and a minimum element in the submatrix, obtaining a second difference between a maximum element and a minimum element in the two-dimensional matrix, and calculating a difference ratio between the first difference and the second difference;

[0017] Counting the number of occurrences of each element in the submatrix to obtain a maximum number of occurrences, calculating a quantity ratio between the maximum number of occurrences and a total number of elements in the submatrix, and obtaining a difference between a constant 1 and the quantity ratio;

[0018] A weighted sum is performed on the difference ratio and the difference, and the corresponding weighted sum result is used as the discreteness of the submatrix.

[0019] Preferably, obtaining the degree of deviation of the submatrix according to the difference between each element in the submatrix includes:

[0020] The average deviation of all elements in the submatrix is ​​calculated, and the average deviation is negatively mapped using an exponential function with a natural constant as the base to obtain a corresponding mapping value, and a subtraction result between a constant 1 and the mapping value is obtained as the deviation degree of the submatrix.

[0021] Preferably, obtaining the data complexity of the submatrix according to the discreteness and deviation of the submatrix includes:

[0022] A weighted sum is performed on the discreteness of the submatrix and the deviation of the submatrix, and the corresponding weighted sum result is used as the data complexity of the submatrix.

[0023] Preferably, performing an assimilation operation on the elements in the two-dimensional matrix according to the data complexity of each of the submatrices to obtain a target two-dimensional matrix includes:

[0024] Obtain a preset data complexity threshold, and if the data complexity of any submatrix is ​​less than the data complexity threshold, use the submatrix as a target submatrix;

[0025] For any target submatrix, calculate the element mean of all elements in the target submatrix, replace all elements in the target submatrix with the element mean, and obtain the assimilated submatrix;

[0026] Each of the assimilated sub-matrices is used to replace the corresponding target sub-matrix in the two-dimensional matrix to obtain a target two-dimensional matrix.

[0027] Preferably, obtaining the scanning optimality of the scanning mode according to the continuous identical data in the one-dimensional data sequence includes:

[0028] Counting the lengths of subsequences of consecutive identical data in the one-dimensional data sequence, obtaining a maximum subsequence length, counting the total sequence length of the one-dimensional data sequence, and calculating a first length ratio between the maximum subsequence length and the total sequence length;

[0029] Obtaining a preset length threshold, counting the number of subsequences whose subsequence lengths are greater than the length threshold, and calculating a second length ratio between the number of subsequences and the total sequence length;

[0030] A weighted sum is performed on the first length ratio and the second length ratio to obtain a corresponding weighted sum result as the scanning optimality of the scanning method.

[0031] Preferably, obtaining the optimal scanning mode according to the scanning optimality of each scanning mode includes:

[0032] According to the scanning optimality of each scanning mode, the scanning mode corresponding to the maximum scanning optimality is obtained as the optimal scanning mode.

[0033] Compared with the prior art, the embodiments of the present invention have the following beneficial effects:

[0034] The present invention obtains data information to be crawled in a website, performs base conversion on the data information to be crawled, obtains a corresponding numerical sequence, and constructs a two-dimensional matrix of the numerical sequence; divides the two-dimensional matrix into at least one submatrix of a preset size, obtains the data complexity of each submatrix according to the element difference between the two-dimensional matrix and each submatrix, performs assimilation operation on the elements in the two-dimensional matrix according to the data complexity of each submatrix, and obtains a target two-dimensional matrix; for any scanning method among at least two preset scanning methods, uses the scanning method to scan the elements in the target two-dimensional matrix to obtain a corresponding one-dimensional data sequence, obtains the scanning optimality of the scanning method according to the continuous identical data in the one-dimensional data sequence, and obtains an optimal scanning method according to the scanning optimality of each scanning method; uses the optimal scanning method to scan the elements in the target two-dimensional matrix to obtain a target data sequence, performs run-length coding compression on the target data sequence to obtain corresponding compressed data, and decompresses and restores the crawled compressed data obtained by automatically crawling the compressed data to obtain the crawled data information in the website. Among them, since there are fewer repeated values ​​in the data to be crawled in the network, and the lower the data complexity of the submatrix, the higher the degree of assimilation of the elements in the corresponding submatrix, some elements in the two-dimensional matrix are assimilated to obtain the target two-dimensional matrix, thereby achieving the purpose of simplifying complexity and improving computational efficiency; since the number and length of subsequences of continuous identical data in the one-dimensional data sequence corresponding to the target two-dimensional matrix obtained by different scanning methods are different, the scanning optimality of each scanning method is analyzed according to the continuous identical data in the one-dimensional data sequence obtained under each scanning method to obtain the optimal scanning method, and the scanning path of the optimal scanning method has the most identical data, so that the repeated values ​​when the target two-dimensional matrix is ​​run-length encoded using the optimal scanning method are the most, so that the data compression degree is maximized, thereby improving the efficiency of automated crawling of data information. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0036] Figure 1 This is a method flow chart of a method for automatically crawling website data based on the Internet of Things provided by the first embodiment of the present invention;

[0037] Figure 2 It is a schematic diagram of a two-dimensional matrix provided in Embodiment 1 of the present invention. DETAILED DESCRIPTION

[0038] Embodiments of the present disclosure are described in detail below, and examples of the embodiments are shown in the accompanying drawings. The embodiments described below with reference to the accompanying drawings are exemplary and are intended to be used to explain the present disclosure, but should not be understood as limiting the present disclosure.

[0039] It should be noted that the terms "first", "second", etc. in the specification of the present disclosure and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present disclosure described herein can be implemented in an order other than those illustrated or described herein. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present disclosure. Instead, they are merely examples of devices and methods consistent with some aspects of the present disclosure.

[0040] In order to illustrate the technical solution of the present invention, specific embodiments are provided below for illustration.

[0041] See also Figure 1 , is a method flow chart of a method for automatically crawling website data based on the Internet of Things provided by the first embodiment of the present invention, such as Figure 1 As shown, the method may include:

[0042] Step S101, obtaining data information to be crawled in the website, converting the data information to be crawled into a base, obtaining a corresponding numerical sequence, and constructing a two-dimensional matrix of the numerical sequence.

[0043] Usually, the data information to be crawled in the website contains a variety of important data indicators, such as visits, click-through rate, behavioral data, etc. In addition to decimal numerical data, the data information to be crawled also contains non-numeric data such as characters or English. Therefore, in the embodiment of the present invention, the data information to be crawled is first converted into a base, and for decimal numerical data, the original data is not processed and retained, and for non-numeric data, ASCII code is used for conversion to obtain the corresponding decimal numerical data.

[0044] The decimal numerical data is stored in the numerical sequence in the order of its original data. Since each value in the matrix corresponds to a binary code and the matrix is ​​scanned in a traditional raster scanning method (row by row scanning), the original numerical sequence can be restored. Therefore, the numerical sequence is divided into m rows according to the numerical order of the numerical sequence, with n decimal values ​​as one row, to obtain a matrix of size It should be noted that there is no restriction on the values ​​of n and m, and they can be equal or unequal, depending on the implementation scenario.

[0045] It should be noted that, since the number of values ​​in the numerical sequence cannot be accurately divided into a complete two-dimensional matrix every time, and in order to maximize the division of the two-dimensional matrix, the data in the numerical sequence in the embodiment of the present invention needs to be divided into a two-dimensional matrix that is closer to a square as much as possible, that is, for When the number of data is an even number, a two-dimensional matrix corresponding to the minimum difference between n and m is constructed; when the number of data is an odd number, a minimum integer that makes it an even number can be added to the number of data, and then the data is processed in the same way as if the number of data is an even number. In order not to affect subsequent processing, the extra positions in the two-dimensional matrix are not filled with numbers.

[0046] For example: Refer to Figure 2 , which is a schematic diagram of a two-dimensional matrix. Figure 2 The 59 data obtained in need to construct a For a two-dimensional matrix The extra position in the two-dimensional matrix is ​​not filled with values.

[0047] At this point, the crawled data information is converted into a two-dimensional matrix.

[0048] Step S102, divide the two-dimensional matrix into at least one sub-matrix of a preset size, and obtain the data complexity of each sub-matrix according to the element difference between the two-dimensional matrix and each sub-matrix; according to the data complexity of each sub-matrix, perform an assimilation operation on the elements in the two-dimensional matrix to obtain a target two-dimensional matrix.

[0049] In most data processing tasks, there is correlation or similarity between adjacent data, that is, data in a local area is more likely to have similar characteristics or statistical distributions, which makes it simpler and more efficient to analyze the data in each small block, thereby simplifying the computational complexity and improving the computational efficiency. Therefore, in the embodiment of the present invention, the two-dimensional matrix is ​​divided into at least one sub-matrix of a preset size, and the specific process is as follows:

[0050] The data obtained in step S101 is The two-dimensional matrix is ​​divided into at least one dimension The submatrix of The uncertainty of the size of the two-dimensional matrix is ​​that the two-dimensional matrix may not be completely divided into multiple Therefore, for the remaining data after the sub-matrix division, the sub-matrix with the same amount of data is divided into groups of nine data and the order and direction of the raster scan are used as the priority. The remaining data after the sub-matrix is ​​divided is obtained in sequence according to the raster scanning order to obtain 9 data as a sub-matrix. As for the data that is not enough to be constructed into a sub-matrix, since its data volume is small, it will not affect the overall compression crawling efficiency. Therefore, the data that is not enough to be constructed into a sub-matrix is ​​not processed and the original data value is retained.

[0051] It should be noted that the embodiment of the present invention does not impose any requirements on the size of the sub-matrix, which can be set according to the implementation scenario.

[0052] After dividing the two-dimensional matrix into at least one sub-matrix of a preset size, in order to achieve a greater degree of run-length coding compression, the same data can be used to replace the data in the sub-matrix with low data complexity. Therefore, in an embodiment of the present invention, first, according to the element difference between the two-dimensional matrix and each sub-matrix, the data complexity of each sub-matrix is ​​obtained respectively. The lower the data complexity, the more likely the data in the sub-matrix is ​​to be assimilated. Then, according to the data complexity of each sub-matrix, the elements in the two-dimensional matrix are assimilated to obtain the target two-dimensional matrix for subsequent run-length coding to obtain compressed data.

[0053] Among them, according to the element difference between the two-dimensional matrix and each submatrix, the data complexity of each submatrix is ​​obtained respectively, including:

[0054] (1) For any submatrix, the discreteness of the submatrix is ​​obtained according to the difference in elements and the difference in the number of elements between the two-dimensional matrix and the submatrix.

[0055] Specifically, a first difference between a maximum element and a minimum element in the submatrix is ​​obtained, a second difference between a maximum element and a minimum element in the two-dimensional matrix is ​​obtained, and a difference ratio between the first difference and the second difference is calculated;

[0056] Counting the number of occurrences of each element in the submatrix to obtain a maximum number of occurrences, calculating a quantity ratio between the maximum number of occurrences and a total number of elements in the submatrix, and obtaining a difference between a constant 1 and the quantity ratio;

[0057] A weighted sum is performed on the difference ratio and the difference, and the corresponding weighted sum result is used as the discreteness of the submatrix.

[0058] In one embodiment, according to the step S101 obtained The two-dimensional matrix is ​​divided into K sub-matrices. Taking the k-th sub-matrix obtained by division as an example, get the maximum element in the k-th sub-matrix and the smallest element The first difference between the two, and at the same time, get the maximum element in the two-dimensional matrix and the smallest element The second difference between the first difference and the second difference is calculated by taking the first difference as the numerator and the second difference as the denominator; the maximum number of occurrences of different elements in the kth submatrix is ​​counted ,Will As the numerator, the total number of elements in the kth submatrix As the denominator, calculate and The quantity ratio between them is calculated, and then the discreteness of the k-th submatrix is ​​obtained according to the difference ratio and the quantity ratio. The expression of the discreteness of the k-th submatrix is:

[0059]

[0060] in, represents the discreteness of the kth submatrix, represents the largest element in the kth submatrix, represents the smallest element in the kth submatrix, represents the maximum element in a two-dimensional matrix, represents the smallest element in a two-dimensional matrix, is the maximum number of occurrences of each element in the kth submatrix, is the total number of elements contained in the kth submatrix, 1 represents a constant, represents the first weight, Represents the second weight.

[0061] It should be noted that It reflects the proportion of the same data in the kth submatrix. The larger the proportion, the more identical elements there are in the kth submatrix. The smaller the corresponding element type, the lower the discreteness of the elements in the kth submatrix. The larger the value of , the smaller the discreteness of the corresponding k-th submatrix; reflects the compactness of the elements in the kth submatrix, The larger the value of , the more element types there are in the k-th submatrix, the more discrete the elements are, and the larger the discreteness of the corresponding k-th submatrix is.

[0062] Preferably, the embodiment of the present invention sets the first weight and the second weight as empirical values, that is, There is no restriction on this, and it can be set according to the implementation scenario.

[0063] (2) Obtaining the degree of deviation of the submatrix according to the differences between the elements in the submatrix.

[0064] Specifically, the average deviation of all elements in the submatrix is ​​calculated, and the average deviation is negatively mapped using an exponential function with a natural constant as the base to obtain a corresponding mapping value, and the subtraction result between the constant 1 and the mapping value is obtained as the deviation degree of the submatrix.

[0065] In one implementation, taking the kth submatrix obtained by division as an example, the element mean of all elements in the kth submatrix is ​​calculated, which is recorded as ; According to all elements and element means in the kth submatrix The mean of the absolute values ​​of the deviations between them is calculated as the average deviation. The opposite of the average deviation is substituted into the exponential function with the natural constant as the base to obtain the corresponding exponential function value. The deviation degree of the k-th submatrix can be obtained according to the exponential function value. The expression of the deviation degree of the k-th submatrix is:

[0066]

[0067] in, Indicates the kth submatrix The value of the element, represents the mean of the kth sub-matrix data, represents the total number of elements contained in the kth submatrix. In the embodiment of the present invention, , Represents the i-th element and the mean of the elements in the k-th submatrix The absolute value of the difference, represents the deviation of the kth submatrix, exp() represents the exponential function with a natural constant as the base, || represents the absolute value sign, and 1 represents a constant.

[0068] It should be noted that It reflects the degree of deviation of each element in the k-th submatrix from its element mean. The larger the value of , the greater the difference between each element in the k-th matrix and the element mean, the greater the difference between the elements in the k-th submatrix, and the greater the deviation of the corresponding k-th submatrix.

[0069] (3) According to the discreteness and deviation of the submatrix, the data complexity of the submatrix is ​​obtained.

[0070] Specifically, a weighted sum is performed on the discreteness of the submatrix and the deviation of the submatrix, and the corresponding weighted sum result is used as the data complexity of the submatrix.

[0071] In one implementation, taking the kth submatrix obtained by division as an example, after obtaining the discreteness and deviation of the kth submatrix, the data complexity of the kth submatrix can be obtained according to the discreteness and deviation, and the expression of the data complexity of the kth submatrix is:

[0072]

[0073] in, represents the data complexity of the k-th matrix, represents the discreteness of the kth submatrix, represents the deviation degree of the kth submatrix, represents the discreteness weight, Represents the deviation weight.

[0074] It should be noted that reflects the discreteness of the elements in the kth submatrix, The larger the value of , the greater the variation range and discreteness of the elements in the k-th submatrix, and the greater the data complexity of the corresponding k-th submatrix; reflects the deviation of the elements in the kth submatrix, The larger the value of , the greater the difference between the elements in the k-th submatrix, and the greater the data complexity of the corresponding k-th submatrix.

[0075] Preferably, the embodiment of the present invention sets the dispersion weight and the deviation weight as empirical values, that is, There is no restriction on this, and it can be set according to the implementation scenario.

[0076] Since the higher the data complexity of the submatrix, the lower the degree of assimilation of the elements in the corresponding submatrix, and the lower the data complexity of the submatrix, the higher the degree of assimilation of the elements in the corresponding submatrix, in an embodiment of the present invention, an assimilation operation is performed on the elements in the two-dimensional matrix according to the data complexity of each submatrix to obtain a target two-dimensional matrix.

[0077] Among them, according to the data complexity of each sub-matrix, the elements in the two-dimensional matrix are assimilated to obtain the target two-dimensional matrix, including:

[0078] Obtain a preset data complexity threshold, and if the data complexity of any submatrix is ​​less than the data complexity threshold, use the submatrix as a target submatrix;

[0079] For any target submatrix, calculate the element mean of all elements in the target submatrix, replace all elements in the target submatrix with the element mean, and obtain the assimilated submatrix;

[0080] Each of the assimilated sub-matrices is used to replace the corresponding target sub-matrix in the two-dimensional matrix to obtain a target two-dimensional matrix.

[0081] In one embodiment, according to step S101, The two-dimensional matrix is ​​divided into K sub-matrices, and the data complexity of each sub-matrix is ​​recorded as ,Take the kth submatrix obtained by division as an example, first obtain the preset data complexity threshold. The data complexity threshold of this scheme is 0.3. If the data complexity of the kth submatrix is ​​less than 0.3, that is , then all elements in the k-th submatrix are assimilated, where the assimilation method is: calculate the element mean of all elements in the k-th submatrix , replace all elements of the kth submatrix with the element mean , corresponding to the k-th sub-matrix after assimilation, replace the k-th sub-matrix after assimilation The corresponding unassimilated kth submatrix in the two-dimensional matrix of . Similarly, the above operation is performed on each submatrix to finally obtain the target two-dimensional matrix.

[0082] It should be noted that in the embodiments of the present invention, there is no restriction on the preset data complexity threshold, which can be set according to the implementation scenario.

[0083] At this point, the target two-dimensional matrix is ​​obtained after assimilation of some elements in the two-dimensional matrix.

[0084] Step S103, for any scanning method among at least two preset scanning methods, use the scanning method to scan the elements in the target two-dimensional matrix to obtain a corresponding one-dimensional data sequence, and obtain the scanning optimality of the scanning method based on the continuous identical data in the one-dimensional data sequence; according to the scanning optimality of each scanning method, obtain the optimal scanning method.

[0085] After obtaining the target two-dimensional matrix, in order to facilitate run-length coding compression, the elements in the two-dimensional matrix can be expanded into one-dimensional sequence data through a preset scanning method. Therefore, the preset scanning method in the embodiment of the present invention includes but is not limited to raster scanning, Zigzag scanning, serpentine scanning, and Hilbert scanning, wherein raster scanning is performed in a row-by-row scanning manner, and Zigzag scanning is a zigzag scanning method from the upper left corner to the lower right corner. The other scanning methods are also well-known technologies and are not described here. Then, for any scanning method, the elements in the target two-dimensional matrix are first scanned using the scanning method to obtain the corresponding one-dimensional data sequence, and then the scanning optimality of the scanning method is obtained according to the continuous identical data in the one-dimensional data sequence. Finally, according to the scanning optimality of each scanning method, the optimal scanning method is obtained, so that the obtained optimal scanning method has the most identical data on its scanning path, thereby maximizing the compression degree of run-length coding.

[0086] Wherein, obtaining the scanning optimality of the scanning method according to the continuous identical data in the one-dimensional data sequence includes:

[0087] Counting the lengths of subsequences of consecutive identical data in the one-dimensional data sequence, obtaining a maximum subsequence length, counting the total sequence length of the one-dimensional data sequence, and calculating a first length ratio between the maximum subsequence length and the total sequence length;

[0088] Obtaining a preset length threshold, counting the number of subsequences whose subsequence lengths are greater than the length threshold, and calculating a second length ratio between the number of subsequences and the total sequence length;

[0089] A weighted sum is performed on the first length ratio and the second length ratio to obtain a corresponding weighted sum result as the scanning optimality of the scanning method.

[0090] In one embodiment, taking the mth scanning mode as an example, the target two-dimensional matrix is ​​scanned by the mth scanning mode to obtain the corresponding one-dimensional data sequence, and the subsequences composed of continuous identical data in the one-dimensional data sequence are counted, and the subsequence length of each subsequence is obtained to obtain the maximum subsequence length. , count the total sequence length of the one-dimensional data sequence , the maximum subsequence length As a numerator, the total sequence length As the denominator, calculate the first length ratio between the maximum subsequence length and the total sequence length; obtain a preset length threshold, which is 5 in this embodiment, and count the number of subsequences with a subsequence length greater than 5, taking the number of subsequences as the numerator, and the total sequence length As the denominator, the second length ratio between the number of subsequences and the total sequence length is calculated; the scanning optimality of the mth scanning mode is obtained according to the first length ratio and the second length ratio, and the expression of the scanning optimality is:

[0091]

[0092] in, represents the scanning optimality of the one-dimensional data sequence corresponding to the m-th scanning mode, Indicates the maximum subsequence length of the subsequence consisting of consecutive identical data in the one-dimensional data sequence corresponding to the m-th scanning mode, Represents the total sequence length of the one-dimensional data sequence corresponding to the m-th scanning mode, Indicates that the length of each subsequence in the one-dimensional data sequence corresponding to the m-th scanning mode is greater than the length threshold The number of subsequences, represents the subsequence length of the subsequence consisting of continuous identical data in the one-dimensional data sequence corresponding to the m-th scanning mode, a represents the weight of the first length ratio, b represents the weight of the second length ratio, Indicates the length threshold.

[0093] It should be noted that It reflects the maximum proportion of continuous identical data in the one-dimensional data sequence corresponding to the m-th scanning mode. The larger the proportion, the longer the maximum subsequence length of the subsequence composed of continuous identical data in the one-dimensional data sequence corresponding to the m-th scanning mode, and the higher the continuity of the one-dimensional data sequence corresponding to the m-th scanning mode. It reflects the proportion of consecutive subsequences of identical data in the one-dimensional data sequence corresponding to the m-th scanning mode. The larger it is, the more repeated values ​​there are in the one-dimensional data sequence, and the higher the continuity and consistency of the one-dimensional data sequence obtained using the m-th scanning method. The greater the scanning optimality of the m-th scanning method, the greater the probability of becoming the optimal scanning method.

[0094] Preferably, the embodiment of the present invention sets the length threshold, the weight of the first length ratio and the weight of the second length ratio as empirical values, that is, , a=3, b=7, there is no restriction on this and it can be set according to the implementation scenario.

[0095] Similarly, referring to the method for obtaining the scanning optimality of the above scanning method, the scanning optimality of each scanning method is obtained respectively, and then the optimal scanning method is obtained according to the scanning optimality of each scanning method. The specific method for obtaining the optimal scanning method is:

[0096] According to the scanning optimality of each scanning mode, the scanning mode corresponding to the maximum scanning optimality is obtained as the optimal scanning mode.

[0097] At this point, the optimal scanning method is obtained according to the scanning optimality obtained by different scanning methods.

[0098] Step S104, use the optimal scanning method to scan the elements in the target two-dimensional matrix to obtain the target data sequence, perform run-length encoding compression on the target data sequence to obtain corresponding compressed data, and decompress the crawled compressed data obtained by automatically crawling the compressed data to obtain the crawled data information in the website.

[0099] The elements in the target two-dimensional matrix are scanned by the optimal scanning method obtained in step S103 to obtain a target data sequence that can be compressed to the maximum extent, and the target data sequence is run-length encoded to obtain corresponding compressed data, wherein the run-length encoding belongs to the prior art and will not be described in detail here. It is worth noting that the run-length encoding compression method based on the optimal scanning method can reduce the loss of the data automatically crawled, increase the crawling efficiency, and increase the compression rate.

[0100] After obtaining the compressed data, the optimal scanning method corresponding to the compressed data and the construction parameters of the target two-dimensional matrix are stored together with the compressed data to facilitate subsequent restoration processing. Specifically, after crawling the compressed data, the inverse operation is performed according to the compression process of the compressed data, and the original crawled data information in the website can be decoded and restored, that is, the data information to be crawled obtained in step S101.

[0101] It should be noted that the focus of the present invention is on how to perform run-length encoding compression on the data information to be crawled in the website. Therefore, the process of decompressing and restoring the compressed data belongs to the prior art and will not be described in detail here.

[0102] In summary, the present invention obtains data information to be crawled in a website, performs base conversion on the data information to be crawled, obtains a corresponding numerical sequence, and constructs a two-dimensional matrix of the numerical sequence; divides the two-dimensional matrix into at least one sub-matrix of a preset size, obtains the data complexity of each sub-matrix according to the element difference between the two-dimensional matrix and each sub-matrix, performs assimilation operation on the elements in the two-dimensional matrix according to the data complexity of each sub-matrix, and obtains a target two-dimensional matrix; for any scanning method among at least two preset scanning methods, uses the scanning method to scan the elements in the target two-dimensional matrix to obtain a corresponding one-dimensional data sequence, obtains the scanning optimality of the scanning method according to the continuous identical data in the one-dimensional data sequence, and obtains the optimal scanning method according to the scanning optimality of each scanning method; uses the optimal scanning method to scan the elements in the target two-dimensional matrix to obtain a target data sequence, performs run-length coding compression on the target data sequence to obtain corresponding compressed data, and decompresses and restores the crawled compressed data obtained by automatically crawling the compressed data to obtain the crawled data information in the website. Among them, since there are fewer repeated values ​​in the data to be crawled in the network, and the lower the data complexity of the submatrix, the higher the degree of assimilation of the elements in the corresponding submatrix, some elements in the two-dimensional matrix are assimilated to obtain the target two-dimensional matrix, thereby achieving the purpose of simplifying complexity and improving computational efficiency; since the number and length of subsequences of continuous identical data in the one-dimensional data sequence corresponding to the target two-dimensional matrix obtained by different scanning methods are different, the scanning optimality of each scanning method is analyzed according to the continuous identical data in the one-dimensional data sequence obtained under each scanning method to obtain the optimal scanning method, and the scanning path of the optimal scanning method has the most identical data, so that the repeated values ​​when the target two-dimensional matrix is ​​run-length encoded using the optimal scanning method are the most, so that the data compression degree is maximized, thereby improving the efficiency of automated crawling of data information.

[0103] The above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features may be replaced by equivalents. Such modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention, and should be included in the protection scope of the present invention.

Claims

1. A method for automatically crawling website data based on the Internet of Things, characterized in that: The method comprises: Acquire data information to be crawled from the website, perform base conversion on the data information to be crawled, obtain a corresponding numerical sequence, and construct a two-dimensional matrix of the numerical sequence; Divide the two-dimensional matrix into at least one submatrix of a preset size, obtain the data complexity of each submatrix according to the element difference between the two-dimensional matrix and each submatrix, and perform an assimilation operation on the elements in the two-dimensional matrix according to the data complexity of each submatrix to obtain a target two-dimensional matrix; For any scanning mode among at least two preset scanning modes, use the scanning mode to scan the elements in the target two-dimensional matrix to obtain a corresponding one-dimensional data sequence, obtain the scanning optimality of the scanning mode according to the continuous identical data in the one-dimensional data sequence, and obtain the optimal scanning mode according to the scanning optimality of each scanning mode; Scanning the elements in the target two-dimensional matrix using the optimal scanning method to obtain a target data sequence, performing run-length encoding compression on the target data sequence to obtain corresponding compressed data, and decompressing and restoring the crawled compressed data obtained by automatically crawling the compressed data to obtain crawled data information in the website; The assimilation operation is performed on the elements in the two-dimensional matrix according to the data complexity of each of the sub-matrices to obtain a target two-dimensional matrix, including: Obtain a preset data complexity threshold, and if the data complexity of any submatrix is ​​less than the data complexity threshold, use the submatrix as a target submatrix; For any target submatrix, calculate the element mean of all elements in the target submatrix, replace all elements in the target submatrix with the element mean, and obtain the assimilated submatrix; Using each of the assimilated sub-matrices to replace the corresponding target sub-matrix in the two-dimensional matrix to obtain a target two-dimensional matrix; The step of obtaining the scanning optimality of the scanning method according to the continuous identical data in the one-dimensional data sequence includes: Counting the lengths of subsequences of consecutive identical data in the one-dimensional data sequence, obtaining a maximum subsequence length, counting the total sequence length of the one-dimensional data sequence, and calculating a first length ratio between the maximum subsequence length and the total sequence length; Obtaining a preset length threshold, counting the number of subsequences whose subsequence lengths are greater than the length threshold, and calculating a second length ratio between the number of subsequences and the total sequence length; A weighted sum is performed on the first length ratio and the second length ratio to obtain a corresponding weighted sum result as the scanning optimality of the scanning method.

2. According to the method of automatic crawling of website data based on the Internet of Things in claim 1, it is characterized in that: The obtaining the data complexity of each submatrix according to the element difference between the two-dimensional matrix and each submatrix includes: For any submatrix, according to the difference in elements and the difference in the number of elements between the two-dimensional matrix and the submatrix, obtain the discreteness of the submatrix; Obtaining the degree of deviation of the submatrix according to the difference between each element in the submatrix; The data complexity of the submatrix is ​​obtained according to the discreteness and the deviation of the submatrix.

3. According to the method of automatic crawling of website data based on the Internet of Things in claim 2, it is characterized in that: The obtaining the discreteness of the submatrix according to the element difference and the element quantity difference between the two-dimensional matrix and the submatrix comprises: Obtaining a first difference between a maximum element and a minimum element in the submatrix, obtaining a second difference between a maximum element and a minimum element in the two-dimensional matrix, and calculating a difference ratio between the first difference and the second difference; Counting the number of occurrences of each element in the submatrix to obtain a maximum number of occurrences, calculating a quantity ratio between the maximum number of occurrences and a total number of elements in the submatrix, and obtaining a difference between a constant 1 and the quantity ratio; A weighted sum is performed on the difference ratio and the difference, and the corresponding weighted sum result is used as the discreteness of the submatrix.

4. The method for automatically crawling website data based on the Internet of Things according to claim 2 is characterized in that: The step of obtaining the degree of deviation of the submatrix according to the difference between the elements in the submatrix comprises: The average deviation of all elements in the submatrix is ​​calculated, and the average deviation is negatively mapped using an exponential function with a natural constant as the base to obtain a corresponding mapping value, and a subtraction result between a constant 1 and the mapping value is obtained as the deviation degree of the submatrix.

5. The method for automatically crawling website data based on the Internet of Things according to claim 2 is characterized in that: The obtaining the data complexity of the submatrix according to the discreteness and the deviation of the submatrix comprises: A weighted sum is performed on the discreteness of the submatrix and the deviation of the submatrix, and the corresponding weighted sum result is used as the data complexity of the submatrix.

6. The method for automatically crawling website data based on the Internet of Things according to claim 1 is characterized in that: The step of obtaining the optimal scanning mode according to the scanning optimality of each scanning mode comprises: According to the scanning optimality of each scanning mode, the scanning mode corresponding to the maximum scanning optimality is obtained as the optimal scanning mode.

Images