Cross-platform data acquisition system and method

Abstract

The invention relates to the technical field of data analysis, and particularly discloses a cross-platform data acquisition system and method, and the system comprises a single platform crawling module which is used for carrying out data crawling on a single platform to obtain a single platform data crawling result; the data structure combing module is used for generating each cross-platform data structure tree; the data structure division module is used for dividing the complete data of each cross-platform according to the structure based on the data structure tree of each cross-platform to obtain all data sets of each structure level of each cross-platform; the correlation degree analysis module is used for performing correlation degree analysis on the single platform data crawling result and all data sets of all cross-platform structure hierarchies to obtain a correlation degree analysis result; the cross-platform crawling module is used for performing cross-platform data crawling in each piece of cross-platform complete data based on the correlation degree analysis result to obtain a cross-platform data crawling result; the method and the device are used for realizing high-precision cross-platform data acquisition.

Description

Technical Field

[0001] The present invention relates to the technical field of data analysis, and in particular to a cross-platform data acquisition system and method. Background Art

[0002] At present, cross-platform data extraction refers to the process of automatically collecting and extracting the required specific information from data sources on multiple different platforms. These platforms can include websites, APIs, Excel files, databases, etc. Cross-platform data extraction technology is widely used in many fields, such as business intelligence, marketing, finance, human resources, etc. The commonly used existing technologies for cross-platform data extraction mainly include the following aspects:

[0003] Web Scraping: A web crawler is a program that automatically crawls web page content. It can crawl text, pictures, links and other information from websites. Through web crawler technology, you can automatically obtain the required data on multiple platforms.

[0004] API calls (Application Programming Interface Calls): Many service providers provide APIs that allow developers to access and manipulate data using code on their platforms. Using APIs makes it easier to obtain data and is usually more efficient than using traditional web crawlers.

[0005] Natural Language Processing (NLP): Natural language processing technology is mainly used to process and parse human language text. In the cross-platform data extraction scenario, NLP technology can be used to analyze unstructured data (such as text messages, emails, etc.) to achieve cross-platform data extraction.

[0006] Data mapping and visualization tools: To better understand data sources and relationships, data mapping and visualization tools can be used to visualize the extracted data, such as charts, timelines, etc.

[0007] In addition, the technical background of cross-platform data extraction also includes: data mining, machine learning, artificial intelligence, etc. These technologies provide strong support for cross-platform data extraction, making data mining and utilization more intelligent and efficient.

[0008] However, cross-platform data extraction also has some defects, such as data crawling speed and quality issues: due to the complexity of the data source platform, cross-platform data extraction needs to compare the data of multiple platforms one by one to obtain cross-platform crawling results with a high correlation and meet the crawling requirements input by the user. This leads to the data quality of the data results obtained from cross-platform data may be limited by the accuracy of a large number of correlation calculation processes, and because the large amount of cross-platform data requires a large number of correlation calculation processes, the crawling efficiency is also relatively low.

[0009] Therefore, the present invention proposes a cross-platform data acquisition system and method. Summary of the invention

[0010] The present invention provides a cross-platform data acquisition system and method, which is used to sort out results based on multiple cross-platform data structures, analyze the degree of correlation between data results obtained by crawling a single platform and data sets of different data ranges on other cross-platforms, and realize high-precision cross-platform data acquisition based on the analyzed correlation analysis results, thereby improving the quality and efficiency of cross-platform data acquisition.

[0011] The present invention provides a cross-platform data acquisition system, comprising:

[0012] A single platform crawling module is used to crawl data on a single platform based on the user's data crawling requirements on a single platform and obtain the single platform data crawling results;

[0013] The data structure combing module is used to comb the structure of the complete data of each cross-platform of a single platform and generate a data structure tree for each cross-platform;

[0014] A data structure partitioning module is used to partition the complete data of each cross-platform by structure based on the data structure tree of each cross-platform, and obtain all data sets of each structural level of each cross-platform;

[0015] The correlation analysis module is used to perform correlation analysis on the data crawling results of a single platform and all data sets of all structural levels across all platforms to obtain correlation analysis results;

[0016] The cross-platform crawling module is used to perform cross-platform data crawling in the complete data of each cross-platform based on the correlation analysis results to obtain the cross-platform data crawling results.

[0017] Preferably, the data structure combing module includes:

[0018] A lowest-level determination submodule, used to obtain all lowest-level partition addresses of complete data of each cross-platform of a single platform;

[0019] The upper address summarization submodule is used to continuously perform upper summarization on each lowest partition address based on the hierarchical mark in each lowest partition address to obtain all upper addresses of each lowest partition address;

[0020] The data link generation submodule is used to sort and connect all the data stored in each lowest partition address and all the corresponding upper addresses according to the upper degree of the address from large to small, so as to obtain a unidirectional hierarchical data link of each lowest partition address;

[0021] The data link merging submodule is used to merge the data nodes with repeated addresses in the unidirectional hierarchical data links of all the lowest partition addresses of each cross-platform to obtain the data structure tree of each cross-platform.

[0022] Preferably, the data structure partitioning module includes:

[0023] A data node determination submodule is used to determine all data nodes in each cross-platform data structure tree;

[0024] The data set determination submodule is used to aggregate all data stored in the partition address corresponding to each data node as the data set of the corresponding data node, and to aggregate the data sets of all data nodes at each structural level of each cross-platform as all data sets of the corresponding structural level of the cross-platform.

[0025] Preferably, the correlation analysis module includes:

[0026] The correlation degree calculation submodule is used to calculate the correlation degree between the data crawling results of a single platform and each data set in the last layer of each cross-platform;

[0027] The correlation degree upper submodule is used to perform upper crawling calculation based on the correlation degree between each cross-platform data structure tree and the single platform data crawling result and all corresponding data sets in the last layer, and obtain the correlation degree between the single platform data crawling result and each data set of each structural level of each cross-platform;

[0028] The result summary submodule is used to take the correlation degree between the data crawling results of a single platform and all data sets of all structural levels across all platforms as the correlation analysis result.

[0029] Preferably, the correlation degree calculation submodule includes:

[0030] A form standardization processing unit is used to perform form standardization processing on all data in the data crawling results of a single platform to obtain first form standardized data, and at the same time, perform form standardization processing on each data set in the last layer of each cross-platform to obtain second form standardized data of each data set in the last layer of each cross-platform;

[0031] A first escape error rate calculation unit is used to take the ratio of the amount of data in each non-standardized form in all data in the single platform data crawling result to the amount of all data in the single platform data crawling result as the proportion of the amount of data in each non-standardized form in the single platform data crawling result, and take the sum of the products of the escape error rates of all non-standardized forms in the single platform data crawling result and the data volume proportion as the standardized escape error rate of the first form standardized data;

[0032] A second escape error rate calculation unit is used to take the ratio of the amount of data in each non-standardized form in all data in each data set in the last layer of each cross-platform to the amount of data in the corresponding data set as the proportion of the amount of data in each non-standardized form in the corresponding data set, and take the sum of the products of the escape error rates of all non-standardized forms of each data set in the last layer of each cross-platform and the proportion of the data amount as the standardized escape error rate of the corresponding second-form standardized data;

[0033] A first correlation degree calculation unit is used to calculate the correlation degree between the first form of standardized data and the second form of standardized data of each data set in the last layer of each cross-platform;

[0034] A second correlation degree calculation unit is used to calculate the correlation degree between the data crawling results of a single platform and each data set in the last layer of each cross-platform based on the correlation degree between the first-form standardized data and the second-form standardized data of each data set in the last layer of each cross-platform and the standardized escape error rate of the first-form standardized data and the standardized escape error rate of the second-form standardized data of each data set in the last layer of each cross-platform.

[0035] Preferably, the first correlation degree calculation unit includes:

[0036] The data segmentation subunit is used to segment the first form of standardized data to obtain a plurality of first segmented data, and at the same time, segment the second form of standardized data of each data set of the last layer of each cross-platform to obtain a plurality of second segmented data;

[0037] A relevance analysis subunit, used to analyze the data relevance between each first segmented data and each second segmented data based on the paragraph data relevance analysis model;

[0038] The correlation degree calculation subunit is used to regard the data correlation degree between each first segmented data in the first form standardized data and all the second segmented data in each data set of each cross-platform last layer as the available data correlation degree of each first segmented data in the first form standardized data, and to regard the average of all available data correlation degrees in the first form standardized data as the correlation degree between the first form standardized data and the second form standardized data of each data set of each cross-platform last layer.

[0039] Preferably, the second correlation degree calculation unit calculates the correlation degree between the single platform data crawling result and each data set in the last layer of each cross-platform based on the correlation degree between the first form standardized data and the second form standardized data of each data set in the last layer of each cross-platform and the standardized escape error rate of the first form standardized data and the standardized escape error rate of the second form standardized data of each data set in the last layer of each cross-platform, including:

[0040]

[0041] In the formula, ε ’ is the correlation degree between the data crawling results of a single platform and the currently calculated data set of the last layer of the cross-platform currently calculated, ε is the correlation degree between the first form standardized data and the second form standardized data of the currently calculated data set of the last layer of the cross-platform currently calculated, σ1 is the standardized escape error rate of the first form standardized data, σ2 is the standardized escape error rate of the second form standardized data of the currently calculated data set of the last layer of the cross-platform currently calculated, and min(ε+σ1-σ2,ε-σ1+σ2) is the minimum value of ε+σ1-σ2 and ε-σ1+σ2.

[0042] Preferably, the submodules of the degree of association include:

[0043] A node relationship combing unit, used to determine all hierarchical adjacent child nodes of each data node in each cross-platform data structure tree based on each cross-platform data structure tree;

[0044] A crawling weight determination unit, used to take the ratio between the amount of data of each level adjacent child node of each data node and the sum of the amount of data of all level adjacent child nodes of the corresponding data node as the upper crawling weight of each level adjacent child node of each data node;

[0045] The upper-level crawling calculation unit is used to perform upper-level crawling calculation based on the upper-level crawling weight of each adjacent child node of each level of each data node in each cross-platform data structure tree and the degree of association between the single platform data crawling results and all data sets of the corresponding last layer, so as to obtain the degree of association between the single platform data crawling results and each data set of each structural level of each cross-platform.

[0046] Preferably, the cross-platform crawling module includes:

[0047] The correlation degree calculation submodule is used to determine the correlation degree between the data crawling results of a single platform and all data sets of all structural levels across all platforms based on the correlation degree analysis results;

[0048] The cross-platform crawling submodule is used to screen out all data sets with a correlation degree not less than a correlation degree threshold from all data sets at all structural levels across all platforms as cross-platform data crawling results.

[0049] The present invention provides a cross-platform data acquisition method, which is used to execute the cross-platform data acquisition system described in any one of Embodiments 1 to 9, including:

[0050] S1: Based on the user's data crawling requirements on a single platform, data is crawled on a single platform to obtain data crawling results on a single platform;

[0051] S2: Structuring the complete data of each cross-platform on a single platform to generate a data structure tree for each cross-platform;

[0052] S3: Based on the data structure tree of each cross-platform, the complete data of each cross-platform is divided by structure to obtain all data sets of each structural level of each cross-platform;

[0053] S4: Perform correlation analysis on the data crawling results of a single platform and all data sets of all structural levels across all platforms to obtain correlation analysis results;

[0054] S5: Based on the correlation analysis results, cross-platform data crawling is performed in the complete data of each cross-platform to obtain the cross-platform data crawling results.

[0055] Compared with the prior art, the beneficial effects of the present invention are as follows: based on the results of combing data structures across multiple platforms, the degree of correlation between data results obtained by crawling on a single platform and data sets of different data ranges across other platforms is analyzed, and high-precision cross-platform data acquisition is achieved based on the analyzed correlation analysis results, thereby improving the quality and efficiency of cross-platform data acquisition.

[0056] Other features and advantages of the present invention will be described in the following description, and partly become apparent from the description, or understood by practicing the present invention. The purpose and other advantages of the present invention can be realized and obtained by the structures specifically pointed out in this application document.

[0057] The technical solution of the present invention is further described in detail below through the accompanying drawings and embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0058] The accompanying drawings are used to provide a further understanding of the present invention and constitute a part of the specification. Together with the embodiments of the present invention, they are used to explain the present invention and do not constitute a limitation of the present invention. In the accompanying drawings:

[0059] Figure 1 A schematic diagram of internal functional submodules of a cross-platform data acquisition system in an embodiment of the present invention;

[0060] Figure 2 The present invention is a flowchart of a cross-platform data acquisition method in an embodiment of the present invention. DETAILED DESCRIPTION

[0061] The preferred embodiments of the present invention are described below in conjunction with the accompanying drawings. It should be understood that the preferred embodiments described herein are only used to illustrate and explain the present invention, and are not used to limit the present invention.

[0062] Embodiment 1:

[0063] The present invention provides a cross-platform data acquisition system, referring to Figure 1 ,include:

[0064] A single platform crawling module is used to crawl data on a single platform based on the user's data crawling requirements on a single platform and obtain the single platform data crawling results;

[0065] The data structure combing module is used to comb the structure of the complete data of each cross-platform of a single platform and generate a data structure tree for each cross-platform;

[0066] A data structure partitioning module is used to partition the complete data of each cross-platform by structure based on the data structure tree of each cross-platform, and obtain all data sets of each structural level of each cross-platform;

[0067] The correlation analysis module is used to perform correlation analysis on the data crawling results of a single platform and all data sets of all structural levels across all platforms to obtain correlation analysis results;

[0068] The cross-platform crawling module is used to perform cross-platform data crawling in the complete data of each cross-platform based on the correlation analysis results to obtain the cross-platform data crawling results.

[0069] In this embodiment, the single platform is a data storage platform where users directly input data crawling requirements, such as a search engine website.

[0070] In this embodiment, the data crawling requirement is a data feature of the data that the user wants to obtain.

[0071] In this embodiment, based on the user's data crawling requirements on a single platform, data crawling is performed on the single platform, and obtaining the single platform data crawling results can be achieved through existing web crawler technology.

[0072] In this embodiment, the single platform data crawling results include all data results obtained from crawling the single platform that meet the data crawling requirements.

[0073] In this embodiment, cross-platform refers to a data storage platform other than a single platform, such as an Excel file.

[0074] In this embodiment, the complete data refers to all data stored in a single platform.

[0075] In this embodiment, the data structure tree is a tree structure including multiple data nodes belonging to different levels, wherein the data nodes contain data in different data ranges in the corresponding platform, and the higher the level to which the data node belongs, the wider the data range stored therein, and vice versa.

[0076] The beneficial effects of the above technology are: based on the results of combing data structures across multiple platforms, the degree of correlation between the data results obtained by crawling on a single platform and data sets of different data ranges across other platforms is analyzed, and high-precision cross-platform data acquisition is achieved based on the analyzed correlation analysis results, thereby improving the quality and efficiency of cross-platform data acquisition.

[0077] Embodiment 2:

[0078] Based on Example 1, the data structure combing module includes:

[0079] A lowest-level determination submodule, used to obtain all lowest-level partition addresses of complete data of each cross-platform of a single platform;

[0080] The upper address summarization submodule is used to continuously perform upper summarization on each lowest partition address based on the hierarchical mark in each lowest partition address to obtain all upper addresses of each lowest partition address;

[0081] The data link generation submodule is used to sort and connect all the data stored in each lowest partition address and all the corresponding upper addresses according to the upper degree of the address from large to small, so as to obtain a unidirectional hierarchical data link of each lowest partition address;

[0082] The data link merging submodule is used to merge the data nodes with repeated addresses in the unidirectional hierarchical data links of all the lowest partition addresses of each cross-platform to obtain the data structure tree of each cross-platform.

[0083] In this embodiment, the lowest partition address is the smallest unit storage address among all data storage addresses across platforms and is also the address containing the most hierarchical tags, that is, the data storage address without any lower and more detailed data storage address.

[0084] In this embodiment, the hierarchical mark is a mark in the storage address used to distinguish the storage address hierarchy, such as “ / ” in the URL.

[0085] In this embodiment, based on the level mark in each lowest partition address, each lowest partition address is continuously summarized to obtain all upper addresses of each lowest partition address, that is:

[0086] The last slash of each lowest partition address and part of the address information after the last slash are deleted, which is the first address summarization process, and an upper address of the lowest partition address is obtained;

[0087] Then, the last slash in the upper address obtained in the previous step and part of the address information after the last slash are deleted, which is the second sub-address summarization process, and the second upper address of the lowest partition address is obtained;

[0088] This process is repeated in this way until the latest upper address does not contain the hierarchy mark, then the address summarization process is stopped and all upper addresses of the lowest partition address are obtained.

[0089] In this embodiment, the upper address is a data storage address whose data range includes the data range corresponding to the lowest partition address.

[0090] In this embodiment, the larger the data range of the upper address is relative to the data range of the corresponding lowest partition address, the greater its upper degree, and vice versa.

[0091] The beneficial effects of the above technology are: by continuously summarizing all the lowest-level partition addresses of each cross-platform complete data of a single platform, all the upper-level addresses of each lowest-level partition address are obtained, and it is achieved by combing the data storage addresses across platforms, completing the efficient combing of the data structure of the data stored across platforms, and by generating a unidirectional hierarchical data chain and a data structure tree, a clear representation of the data structure of the data stored across platforms is achieved.

[0092] Embodiment 3:

[0093] Based on Example 1, the data structure division module includes:

[0094] A data node determination submodule is used to determine all data nodes in each cross-platform data structure tree;

[0095] The data set determination submodule is used to aggregate all data stored in the partition address corresponding to each data node as the data set of the corresponding data node, and to aggregate the data sets of all data nodes at each structural level of each cross-platform as all data sets of the corresponding structural level of the cross-platform.

[0096] In this embodiment, the data node is a node in the data structure tree used to represent a corresponding data range.

[0097] The beneficial effect of the above technology is: based on the structural hierarchy of the data nodes in each cross-platform data structure tree, a clear combing of all cross-platform data is achieved.

[0098] Embodiment 4:

[0099] Based on Example 1, the correlation analysis module includes:

[0100] The correlation degree calculation submodule is used to calculate the correlation degree between the data crawling results of a single platform and each data set in the last layer of each cross-platform;

[0101] The correlation degree upper submodule is used to perform upper crawling calculation based on the correlation degree between each cross-platform data structure tree and the single platform data crawling result and all corresponding data sets in the last layer, and obtain the correlation degree between the single platform data crawling result and each data set of each structural level of each cross-platform;

[0102] The result summary submodule is used to take the correlation degree between the data crawling results of a single platform and all data sets of all structural levels across all platforms as the correlation analysis result.

[0103] In this embodiment, the degree of association indicates the degree of association between the corresponding two groups of data in terms of data crawling requirements. For example, if the data crawling requirements limit the content of the crawled data, the degree of association indicates the degree of association between the data in terms of data content.

[0104] If the data crawling requirement restriction is the data form of the crawled data, then the correlation degree refers to the correlation degree between the data in terms of the data form.

[0105] The beneficial effect of the above technology is: by calculating the degree of correlation between the data crawling results of a single platform and each data set at the last layer of each cross-platform, and performing upper-level crawling calculations, the acquisition rate of the degree of correlation between the data crawling results of a single platform and each data set at each structural level of each cross-platform is greatly improved.

[0106] Embodiment 5:

[0107] Based on Example 4, the association degree calculation submodule includes:

[0108] A form standardization processing unit is used to perform form standardization processing on all data in the data crawling results of a single platform to obtain first form standardized data, and at the same time, perform form standardization processing on each data set in the last layer of each cross-platform to obtain second form standardized data of each data set in the last layer of each cross-platform;

[0109] A first escape error rate calculation unit is used to take the ratio of the amount of data in each non-standardized form in all data in the single platform data crawling result to the amount of all data in the single platform data crawling result as the proportion of the amount of data in each non-standardized form in the single platform data crawling result, and take the sum of the products of the escape error rates of all non-standardized forms in the single platform data crawling result and the data volume proportion as the standardized escape error rate of the first form standardized data;

[0110] A second escape error rate calculation unit is used to take the ratio of the amount of data in each non-standardized form in all data in each data set in the last layer of each cross-platform to the amount of data in the corresponding data set as the proportion of the amount of data in each non-standardized form in the corresponding data set, and take the sum of the products of the escape error rates of all non-standardized forms of each data set in the last layer of each cross-platform and the proportion of the data amount as the standardized escape error rate of the corresponding second-form standardized data;

[0111] A first correlation degree calculation unit is used to calculate the correlation degree between the first form of standardized data and the second form of standardized data of each data set in the last layer of each cross-platform;

[0112] A second correlation degree calculation unit is used to calculate the correlation degree between the data crawling results of a single platform and each data set in the last layer of each cross-platform based on the correlation degree between the first-form standardized data and the second-form standardized data of each data set in the last layer of each cross-platform and the standardized escape error rate of the first-form standardized data and the standardized escape error rate of the second-form standardized data of each data set in the last layer of each cross-platform.

[0113] In this embodiment, the format standardization processing is to convert the data format of all data within the corresponding data range into standard format data. The standard format data here is text format. This process can be completed by using the existing data format conversion method.

[0114] In this embodiment, the non-standardized format refers to other data formats other than the standard format, such as image format, voice format, video format, etc.

[0115] In this embodiment, the escape error rate is the error rate generated by the data content when the corresponding non-standard format data is converted into standard format data.

[0116] In this embodiment, the standardized escape error rate is the error rate generated by the data content when each data set in the last layer of each cross-platform is converted into the second form of standardized data.

[0117] The beneficial effect of the above technology is: by standardizing the form of all data in the data crawling results of a single platform and each data set in the last layer of each cross-platform, and calculating the escape error rate generated in the form conversion process, the accuracy of the correlation between the calculated single platform data crawling results and each data set in the last layer of each cross-platform is guaranteed.

[0118] Embodiment 6:

[0119] Based on Embodiment 5, the first association degree calculation unit includes:

[0120] The data segmentation subunit is used to segment the first form of standardized data to obtain a plurality of first segmented data, and at the same time, segment the second form of standardized data of each data set of the last layer of each cross-platform to obtain a plurality of second segmented data;

[0121] A relevance analysis subunit, used to analyze the data relevance between each first segmented data and each second segmented data based on the paragraph data relevance analysis model;

[0122] The correlation degree calculation subunit is used to regard the data correlation degree between each first segmented data in the first form standardized data and all the second segmented data in each data set of each cross-platform last layer as the available data correlation degree of each first segmented data in the first form standardized data, and to regard the average of all available data correlation degrees in the first form standardized data as the correlation degree between the first form standardized data and the second form standardized data of each data set of each cross-platform last layer.

[0123] In this embodiment, data segmentation can be performed on the data based on the timestamp of the data or a preset time interval.

[0124] In this embodiment, the paragraph data association analysis model is a pre-trained model that can evaluate the data association between two sets of input data;

[0125] The model is obtained by training with two sets of data whose data association is calculated by using existing technology as training samples.

[0126] In this embodiment, the data association degree indicates the association degree between the two groups of formal standardized data in terms of data crawling requirements. For example, if the data crawling requirements limit the content of crawled data, the association degree indicates the association degree between the data in terms of data content.

[0127] If the data crawling requirement restriction is the data form of the crawled data, then the correlation degree refers to the correlation degree between the data in terms of the data form.

[0128] The beneficial effect of the above technology is: by simultaneously segmenting the first form standardized data and the second form standardized data, and introducing a neural network model to accurately calculate the data correlation between the segmented data, it is possible to accurately calculate the degree of correlation between the first form standardized data and the second form standardized data of each data set in the last layer of each cross-platform.

[0129] Embodiment 7:

[0130] On the basis of Embodiment 5, the second correlation degree calculation unit calculates the correlation degree between the single platform data crawling result and each data set of the last layer of each cross-platform based on the correlation degree between the first form standardized data and the second form standardized data of each data set of the last layer of each cross-platform and the standardized escape error rate of the first form standardized data and the standardized escape error rate of the second form standardized data of each data set of the last layer of each cross-platform, including:

[0131]

[0132] In the formula, ε ’ is the correlation degree between the data crawling results of a single platform and the currently calculated data set of the last layer of the cross-platform currently calculated, ε is the correlation degree between the first form standardized data and the second form standardized data of the currently calculated data set of the last layer of the cross-platform currently calculated, σ1 is the standardized escape error rate of the first form standardized data, σ2 is the standardized escape error rate of the second form standardized data of the currently calculated data set of the last layer of the cross-platform currently calculated, and min(ε+σ1-σ2,ε-σ1+σ2) is the minimum value of ε+σ1-σ2 and ε-σ1+σ2.

[0133] The beneficial effect of the above technology is: by introducing the standardized escape error rate of the first form of standardized data and the standardized escape error rate of the second form of standardized data, the accurate calculation of the correlation degree between the data crawling results of a single platform and each data set of the last layer of each cross-platform is achieved.

[0134] Embodiment 8:

[0135] On the basis of Example 4, the sub-modules in terms of association degree include:

[0136] A node relationship combing unit, used to determine all hierarchical adjacent child nodes of each data node in each cross-platform data structure tree based on each cross-platform data structure tree;

[0137] A crawling weight determination unit, used to take the ratio between the amount of data of each level adjacent child node of each data node and the sum of the amount of data of all level adjacent child nodes of the corresponding data node as the upper crawling weight of each level adjacent child node of each data node;

[0138] The upper-level crawling calculation unit is used to perform upper-level crawling calculation based on the upper-level crawling weight of each adjacent child node of each level of each data node in each cross-platform data structure tree and the degree of association between the single platform data crawling results and all data sets of the corresponding last layer, so as to obtain the degree of association between the single platform data crawling results and each data set of each structural level of each cross-platform.

[0139] In this embodiment, all the hierarchical adjacent child nodes of a data node are the child nodes of the next next level of the level to which the data node belongs.

[0140] In this embodiment, the upper crawling weight represents the correlation degree between the adjacent child nodes of the level and the data crawling results of a single platform, and its proportion in the correlation degree value between the data node and the data crawling results of a single platform.

[0141] In this embodiment, an upper crawling calculation is performed based on the upper crawling weight of each adjacent sub-node of each level of each data node in each cross-platform data structure tree and the degree of association between the single platform data crawling result and all data sets of the corresponding last layer, and the degree of association between the single platform data crawling result and each data set of each structural level of each cross-platform is obtained, including:

[0142] The sum of the product of the degree of association between the data sets of all child nodes (which are also one or more data nodes in the last layer) of each data node belonging to the penultimate layer and the data crawling results of a single platform and the upper crawling weight of the corresponding child node is regarded as the degree of association between the data sets of the corresponding data nodes in the penultimate layer and the data crawling results of a single platform;

[0143] And in accordance with the above method, based on the correlation degree between the data sets of all child nodes of each data node belonging to the third-to-last layer and the single platform data crawling result and the upper crawling weight of the corresponding child node, continue to calculate the correlation degree between the data sets of the corresponding data nodes in the third-to-last layer and the single platform data crawling result;

[0144] And so on, until the degree of correlation between the data set of the top-level data node and the data crawling result of a single platform is determined, and the degree of correlation between the data crawling result of a single platform and each data set of each structural level of each cross-platform is obtained.

[0145] The beneficial effect of the above technology is: by introducing the upper-level crawling weights of the adjacent child nodes of each level of each data node, continuous upper-level crawling calculations are performed based on the degree of correlation between the data crawling results of a single platform and all data sets of the corresponding last layer. Compared with directly calculating the degree of correlation between the data crawling results of a single platform and each data set of each structural level of each cross-platform, the method in this embodiment has much less calculation amount, therefore, making the acquisition of the degree of correlation between the data crawling results of a single platform and each data set of each structural level of each cross-platform more efficient.

[0146] Embodiment 9:

[0147] Based on Example 1, the cross-platform crawling module includes:

[0148] The correlation degree calculation submodule is used to determine the correlation degree between the data crawling results of a single platform and all data sets of all structural levels across all platforms based on the correlation degree analysis results;

[0149] The cross-platform crawling submodule is used to screen out all data sets with a correlation degree not less than a correlation degree threshold from all data sets at all structural levels across all platforms as cross-platform data crawling results.

[0150] In this embodiment, the correlation degree threshold is a value that the correlation degree between all data sets in the cross-platform data crawling results and the single platform data crawling results cannot be less than.

[0151] The beneficial effect of the above technology is: based on the degree of correlation between the data crawling results of a single platform and all data sets of all structural levels across all platforms, cross-platform data that meets the data crawling requirements input by the user is screened out.

[0152] Embodiment 10:

[0153] The present invention provides a cross-platform data acquisition method for executing the cross-platform data acquisition system described in any one of embodiments 1 to 9, referring to Figure 2 ,include:

[0154] S1: Based on the user's data crawling requirements on a single platform, data is crawled on a single platform to obtain data crawling results on a single platform;

[0155] S2: Structuring the complete data of each cross-platform on a single platform to generate a data structure tree for each cross-platform;

[0156] S3: Based on the data structure tree of each cross-platform, the complete data of each cross-platform is divided by structure to obtain all data sets of each structural level of each cross-platform;

[0157] S4: Perform correlation analysis on the data crawling results of a single platform and all data sets of all structural levels across all platforms to obtain correlation analysis results;

[0158] S5: Based on the correlation analysis results, cross-platform data crawling is performed in the complete data of each cross-platform to obtain the cross-platform data crawling results.

[0159] The beneficial effects of the above technology are: based on the results of combing data structures across multiple platforms, the degree of correlation between the data results obtained by crawling on a single platform and data sets of different data ranges across other platforms is analyzed, and high-precision cross-platform data acquisition is achieved based on the analyzed correlation analysis results, thereby improving the quality and efficiency of cross-platform data acquisition.

[0160] Obviously, those skilled in the art can make various changes and modifications to the present invention without departing from the spirit and scope of the present invention. Thus, if these modifications and variations of the present invention fall within the scope of the claims of the present invention and their equivalents, the present invention is also intended to include these modifications and variations.

Claims

1. A cross-platform data acquisition system, characterized in that: include: A single platform crawling module is used to crawl data on a single platform based on the user's data crawling requirements on a single platform and obtain the single platform data crawling results; The data structure combing module is used to comb the structure of the complete data of each cross-platform of a single platform and generate a data structure tree for each cross-platform; A data structure partitioning module is used to partition the complete data of each cross-platform by structure based on the data structure tree of each cross-platform, and obtain all data sets of each structural level of each cross-platform; The correlation analysis module is used to perform correlation analysis on the data crawling results of a single platform and all data sets of all structural levels across all platforms to obtain correlation analysis results; The cross-platform crawling module is used to perform cross-platform data crawling in the complete data of each cross-platform based on the correlation analysis results to obtain the cross-platform data crawling results.

2. The cross-platform data acquisition system according to claim 1, characterized in that: Data structure combing module, including: A lowest-level determination submodule, used to obtain all lowest-level partition addresses of complete data of each cross-platform of a single platform; The upper address summarization submodule is used to continuously perform upper summarization on each lowest partition address based on the hierarchical mark in each lowest partition address to obtain all upper addresses of each lowest partition address; The data link generation submodule is used to sort and connect all the data stored in each lowest partition address and all the corresponding upper addresses according to the upper degree of the address from large to small, so as to obtain a unidirectional hierarchical data link of each lowest partition address; The data link merging submodule is used to merge the data nodes with repeated addresses in the unidirectional hierarchical data links of all the lowest partition addresses of each cross-platform to obtain the data structure tree of each cross-platform.

3. The cross-platform data acquisition system according to claim 1, characterized in that: Data structure partitioning module, including: A data node determination submodule is used to determine all data nodes in each cross-platform data structure tree; The data set determination submodule is used to aggregate all data stored in the partition address corresponding to each data node as the data set of the corresponding data node, and to aggregate the data sets of all data nodes at each structural level of each cross-platform as all data sets of the corresponding structural level of the cross-platform.

4. The cross-platform data acquisition system according to claim 1, characterized in that: Relevance analysis module, including: The correlation degree calculation submodule is used to calculate the correlation degree between the data crawling results of a single platform and each data set of the last layer of each cross-platform; The correlation degree upper submodule is used to perform upper crawling calculation based on the correlation degree between each cross-platform data structure tree and the single platform data crawling result and all corresponding data sets in the last layer, and obtain the correlation degree between the single platform data crawling result and each data set of each structural level of each cross-platform; The result summary submodule is used to take the correlation degree between the data crawling results of a single platform and all data sets of all structural levels across all platforms as the correlation analysis result.

5. The cross-platform data acquisition system according to claim 4, characterized in that: The correlation degree calculation submodule includes: A form standardization processing unit is used to perform form standardization processing on all data in the data crawling results of a single platform to obtain first form standardized data, and at the same time, perform form standardization processing on each data set in the last layer of each cross-platform to obtain second form standardized data of each data set in the last layer of each cross-platform; A first escape error rate calculation unit is used to take the ratio of the amount of data in each non-standardized form in all data in the single platform data crawling result to the amount of all data in the single platform data crawling result as the proportion of the amount of data in each non-standardized form in the single platform data crawling result, and take the sum of the products of the escape error rates of all non-standardized forms in the single platform data crawling result and the data volume proportion as the standardized escape error rate of the first form standardized data; A second escape error rate calculation unit is used to take the ratio of the amount of data in each non-standardized form in all data in each data set in the last layer of each cross-platform to the amount of data in the corresponding data set as the proportion of the amount of data in each non-standardized form in the corresponding data set, and take the sum of the products of the escape error rates of all non-standardized forms of each data set in the last layer of each cross-platform and the proportion of the data amount as the standardized escape error rate of the corresponding second-form standardized data; A first correlation degree calculation unit is used to calculate the correlation degree between the first form of standardized data and the second form of standardized data of each data set in the last layer of each cross-platform; A second correlation degree calculation unit is used to calculate the correlation degree between the data crawling results of a single platform and each data set in the last layer of each cross-platform based on the correlation degree between the first-form standardized data and the second-form standardized data of each data set in the last layer of each cross-platform and the standardized escape error rate of the first-form standardized data and the standardized escape error rate of the second-form standardized data of each data set in the last layer of each cross-platform.

6. The cross-platform data acquisition system according to claim 5, characterized in that: The first correlation degree calculation unit includes: The data segmentation subunit is used to segment the first form of standardized data to obtain a plurality of first segmented data, and at the same time, segment the second form of standardized data of each data set of the last layer of each cross-platform to obtain a plurality of second segmented data; A relevance analysis subunit, used to analyze the data relevance between each first segmented data and each second segmented data based on the paragraph data relevance analysis model; The correlation degree calculation subunit is used to regard the data correlation degree between each first segmented data in the first form standardized data and all the second segmented data in each data set of each cross-platform last layer as the available data correlation degree of each first segmented data in the first form standardized data, and to regard the average of all available data correlation degrees in the first form standardized data as the correlation degree between the first form standardized data and the second form standardized data of each data set of each cross-platform last layer.

7. The cross-platform data acquisition system according to claim 5, characterized in that: The second correlation degree calculation unit calculates the correlation degree between the single platform data crawling result and each data set of the last layer of each cross-platform based on the correlation degree between the first form standardized data and the second form standardized data of each data set of the last layer of each cross-platform, the standardized escape error rate of the first form standardized data, and the standardized escape error rate of the second form standardized data of each data set of the last layer of each cross-platform, including: In the formula, ε ’ is the correlation degree between the data crawling results of a single platform and the currently calculated data set of the last layer of the cross-platform currently calculated, ε is the correlation degree between the first form standardized data and the second form standardized data of the currently calculated data set of the last layer of the cross-platform currently calculated, σ1 is the standardized escape error rate of the first form standardized data, σ2 is the standardized escape error rate of the second form standardized data of the currently calculated data set of the last layer of the cross-platform currently calculated, and min(ε+σ1-σ2,ε-σ1+σ2) is the minimum value of ε+σ1-σ2 and ε-σ1+σ2.

8. The cross-platform data acquisition system according to claim 4, characterized in that: The submodules of the relevant level include: A node relationship combing unit, used to determine all hierarchical adjacent child nodes of each data node in each cross-platform data structure tree based on each cross-platform data structure tree; A crawling weight determination unit, used to take the ratio between the amount of data of each level adjacent child node of each data node and the sum of the amount of data of all level adjacent child nodes of the corresponding data node as the upper crawling weight of each level adjacent child node of each data node; The upper-level crawling calculation unit is used to perform upper-level crawling calculation based on the upper-level crawling weight of each adjacent child node of each level of each data node in each cross-platform data structure tree and the degree of association between the single platform data crawling results and all data sets of the corresponding last layer, so as to obtain the degree of association between the single platform data crawling results and each data set of each structural level of each cross-platform.

9. The cross-platform data acquisition system according to claim 1, characterized in that: Cross-platform crawling modules, including: The correlation degree calculation submodule is used to determine the correlation degree between the data crawling results of a single platform and all data sets of all structural levels across all platforms based on the correlation degree analysis results; The cross-platform crawling submodule is used to screen out all data sets with a correlation degree not less than a correlation degree threshold from all data sets at all structural levels across all platforms as cross-platform data crawling results.

10. A cross-platform data acquisition method, characterized in that: A cross-platform data acquisition system for executing any one of claims 1 to 9, comprising: S1: Based on the user's data crawling requirements on a single platform, data is crawled on a single platform to obtain data crawling results on a single platform; S2: Structuring the complete data of each cross-platform on a single platform to generate a data structure tree for each cross-platform; S3: Based on the data structure tree of each cross-platform, the complete data of each cross-platform is divided by structure to obtain all data sets of each structural level of each cross-platform; S4: Perform correlation analysis on the data crawling results of a single platform and all data sets of all structural levels across all platforms to obtain correlation analysis results; S5: Based on the correlation analysis results, cross-platform data crawling is performed in the complete data of each cross-platform to obtain the cross-platform data crawling results.

Images