A method and system for processing big data information based on cloud computing

Abstract

The present invention discloses a big data information processing method and system based on cloud computing, which belongs to the field of big data information processing technology, and includes a data acquisition module, a data channel module and a data cloud storage module, wherein the data channel module is bidirectionally electrically connected to the data acquisition module, the output end of the data acquisition module is electrically connected to the input end of the data cloud storage module, and the output end of the data cloud storage module is electrically connected to the input end of the cloud computing module. In the present invention, by setting the data acquisition module and the data channel module, the data channel module is respectively a plurality of different data acquisition ports, and the data acquisition module collects a large amount of different data according to different data acquisition ports, and transmits a large amount of data to the data cloud storage module for storage, and the acquisition direction change module can change which channel the data acquisition module collects data from, and at what time to increase or reduce the acquisition amount, which can effectively reduce the data acquisition repetition rate.

Description

Technical Field

[0001] The present invention belongs to the technical field of big data information processing, and in particular, relates to a big data information processing method and system based on cloud computing. Background Art

[0002] The strategic significance of big data technology lies not in mastering huge amounts of data information, but in professionally processing these meaningful data. In other words, if big data is compared to an industry, then the key to profitability of this industry lies in improving the "processing capability" of data and realizing the "value-added" of data through "processing". From a technical point of view, the relationship between big data and cloud computing is as inseparable as the front and back of a coin. Big data cannot be processed by a single computer and must adopt a distributed architecture. Its feature is distributed data mining of massive data. However, it must rely on distributed processing, distributed databases, cloud storage, and virtualization technologies of cloud computing. Big data requires special technologies to effectively process large amounts of data within a tolerable time. Technologies applicable to big data include massively parallel processing (MPP) databases, data mining, distributed file systems, distributed databases, cloud computing platforms, the Internet, and scalable storage systems.

[0003] Big data collection information is generally collected from multiple channels, which will inevitably lead to duplication of the collected data. The duplicate data will generally be screened and deleted, and the data cannot be effectively utilized, so it is impossible to improve the data collection efficiency. As a result, a lot of time is spent on collecting data with a high duplication rate, which affects the timely update and release of data. Summary of the invention

[0004] The purpose of the present invention is to solve the problem that big data collection information is generally collected from multiple channels, which will inevitably lead to duplication of the collected data, and the duplicate data will generally be screened and deleted, and the data cannot be effectively utilized, so it is impossible to improve the data collection efficiency, resulting in a high duplication rate of data collected at a high cost, affecting the timely update and publication of data, and a big data information processing method and system based on cloud computing are proposed.

[0005] In order to achieve the above object, the present invention adopts the following technical solutions:

[0006] A big data information processing method and system based on cloud computing, comprising a data acquisition module, a data channel module and a data cloud storage module, wherein the data channel module is bidirectionally electrically connected to the data acquisition module, the output end of the data acquisition module is electrically connected to the input end of the data cloud storage module, the output end of the data cloud storage module is electrically connected to the input end of the cloud computing module, the output end of the cloud computing module is electrically connected to the input end of a duplicate data screening module, the output end of the duplicate data screening module is electrically connected to the input end of a duplicate data deletion module, the output end of the duplicate data screening module is electrically connected to the input end of a duplicate data rate calculation module, the output end of the duplicate data rate calculation module is electrically connected to the input end of a data repetition rate table making module, and the output end of the data repetition rate table making module is electrically connected to the input end of a data publishing module.

[0007] As a further description of the above technical solution:

[0008] The output end of the data publishing module is electrically connected to the input end of the high repetition rate period detection module, the output end of the high repetition rate period detection module is electrically connected to the input end of the collection direction change module, and the output end of the collection direction change module is electrically connected to the input end of the data collection module.

[0009] As a further description of the above technical solution:

[0010] The cloud computing module includes a data collection classification module, a multi-channel transmission module, a cloud storage space segmentation module and a data backup module.

[0011] As a further description of the above technical solution:

[0012] The output end of the data collection classification module is electrically connected to the input end of the multi-channel transmission module, and the output end of the multi-channel transmission module is electrically connected to the input end of the cloud storage space separation module.

[0013] As a further description of the above technical solution:

[0014] The output end of the data backup module is electrically connected to the input end of the cloud storage space partition module.

[0015] As a further description of the above technical solution:

[0016] The data publishing module includes a market published data collection module, a data comparison module, a duplicate data information uploading module, a total data ratio calculation module and a normal data information uploading module.

[0017] As a further description of the above technical solution:

[0018] The output end of the market published data collection module is electrically connected to the input end of the data comparison module, and the output end of the duplicate data information uploading module is electrically connected to the input end of the data comparison module.

[0019] As a further description of the above technical solution:

[0020] The output end of the data comparison module is electrically connected to the input end of the data total amount ratio calculation module, and the output end of the normal data information uploading module is electrically connected to the input end of the data total amount ratio calculation module.

[0021] In summary, due to the adoption of the above technical solution, the beneficial effects of the present invention are:

[0022] 1. In the present invention, by setting a data acquisition module and a data channel module, the data channel modules are respectively a plurality of different data acquisition ports, and the data acquisition module collects a large amount of different data according to different data acquisition ports, and transmits a large amount of data to the data cloud storage module for storage, and the data cloud storage module can send the collected data together to the cloud computing module, and the cloud computing module uses the duplicate data screening module to screen out the duplicate data after data comparison, and finally uses the duplicate data deletion to delete the duplicate data, and the duplicate data rate calculation module calculates the number of times the same data appears, and passes the information to the data repetition rate table making module, and the data repetition rate table making module can make a statistical table, and finally the data publishing module publishes the data, and the data publishing module passes the data to the high repetition rate period detection module, the high repetition rate period detection module detects from which time period and which channel the multiple duplicate data are collected, and sends the information to the collection direction change module, and the collection direction change module can change from which channel the data acquisition module collects data, and at what time to increase or reduce the collection amount, which can effectively reduce the data collection repetition rate.

[0023] 2. In the present invention, a cloud computing module is set up, in which a data collection classification module can classify all collected data, and transmit the data from multiple different channels to multiple different cloud storage subspaces through a multi-channel transmission module, and a cloud storage space separation module divides the cloud space into multiple different subspaces, and uses a data backup module to back up the same data in multiple different subspaces. This can not only greatly improve the data transmission efficiency and the data calculation speed, but also the collected data entering the cloud storage subspace can filter out duplicate data in the shortest time.

[0024] 3. In the present invention, a data publishing module is set up, wherein the market published data collection module collects data that has been published in the market, and the duplicate data information uploading module uploads the duplicate data to the data publishing module, the data comparison module compares the two data from the probability of duplication, and the total data proportion calculation module calculates the proportion of the duplicate data information uploading module to the data that has been published in the market. When the proportion of duplicate data is large, the normal data information uploading module uploads a large amount of non-duplicate data, and when the proportion of duplicate data is small, the normal data information uploading module uploads a small amount of non-duplicate data. By updating the data, it is ensured that the data that has been published in the market has more updated information, thereby attracting people who need data. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] Figure 1 This is a schematic diagram of the structure of a big data information processing method and system based on cloud computing proposed by the present invention;

[0026] Figure 2 A schematic diagram of the submodule structure of a cloud computing module in a big data information processing method based on cloud computing proposed by the present invention;

[0027] Figure 3 This is a schematic diagram of the sub-module structure of a big data information processing method based on cloud computing and a data publishing module in the system proposed by the present invention. DETAILED DESCRIPTION

[0028] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0029] See also Figure 1-3 The present invention provides a technical solution: a big data information processing method and system based on cloud computing, comprising a data acquisition module, a data channel module and a data cloud storage module, wherein the data channel module is bidirectionally electrically connected to the data acquisition module, the output end of the data acquisition module is electrically connected to the input end of the data cloud storage module, the output end of the data cloud storage module is electrically connected to the input end of the cloud computing module, the cloud computing module comprises a data acquisition classification module, a multi-channel transmission module, a cloud storage space partition module and a data backup module, the output end of the data acquisition classification module is electrically connected to the input end of the multi-channel transmission module, the output end of the multi-channel transmission module is electrically connected to the input end of the cloud storage space partition module, and the output end of the data backup module is electrically connected to the input end of the cloud storage space partition module;

[0030] The specific implementation method is as follows: by setting a data acquisition module and a data channel module, the data channel modules are respectively a plurality of different data acquisition ports, and the data acquisition module collects a large amount of different data according to different data acquisition ports, and transmits a large amount of data to the data cloud storage module for storage, and the data cloud storage module can send the collected data together to the cloud computing module, and the cloud computing module uses the duplicate data screening module to screen out the duplicate data after data comparison, and finally uses the duplicate data deletion to delete the duplicate data, and the duplicate data rate calculation module calculates the number of times the same data appears, and passes the information to the data repetition rate table making module, and the data repetition rate table making module can make a statistical table, and finally the data publishing module publishes the data.

[0031] The output end of the cloud computing module is electrically connected to the input end of the duplicate data screening module, the output end of the duplicate data screening module is electrically connected to the input end of the duplicate data deleting module, the output end of the duplicate data screening module is electrically connected to the input end of the duplicate data rate calculation module, and the output end of the duplicate data rate calculation module is electrically connected to the input end of the data repetition rate table making module;

[0032] The specific implementation method is as follows: by setting up a cloud computing module, in which the data collection and classification module can classify all collected data, and transmit the data from multiple different channels to multiple different cloud storage subspaces through a multi-channel transmission module, and the cloud storage space separation module divides the cloud space into multiple different subspaces, and uses the data backup module to back up the same data in multiple different subspaces. This can not only greatly improve the data transmission efficiency and the speed of data calculation, but also the collected data entering the cloud storage subspace can filter out duplicate data in the shortest time.

[0033] The output end of the data repetition rate table making module is electrically connected to the input end of the data publishing module, and the data publishing module includes a market published data collection module, a data comparison module, a duplicate data information uploading module, a total data proportion calculation module and a normal data information uploading module;

[0034] The specific implementation method is: by setting up a data publishing module, wherein the market published data collection module collects data that has been published in the market, and the duplicate data information upload module uploads the duplicate data to the data publishing module, the data comparison module compares the two data from the probability of duplication, and the total data proportion calculation module calculates the proportion of the duplicate data information upload module to the data that has been published in the market. When the proportion of duplicate data is large, the normal data information upload module uploads a large amount of non-duplicate data. When the proportion of duplicate data is small, the normal data information upload module uploads a small amount of non-duplicate data. By updating the data, it is guaranteed to have more updated information than the data already published in the market, thereby attracting people who need data.

[0035] The output end of the market data collection module is electrically connected to the input end of the data comparison module, the output end of the repeated data information upload module is electrically connected to the input end of the data comparison module, the output end of the data comparison module is electrically connected to the input end of the data total proportion calculation module, the output end of the normal data information upload module is electrically connected to the input end of the data total proportion calculation module, the output end of the data publication module is electrically connected to the input end of the high repetition rate period detection module, the output end of the high repetition rate period detection module is electrically connected to the input end of the collection direction change module, and the output end of the collection direction change module is electrically connected to the input end of the data collection module;

[0036] The specific implementation method is as follows: the data publishing module passes the data to the high repetition rate period detection module, the high repetition rate period detection module detects the time period and channel from which the multiple repeated data are collected, and sends the information to the collection direction change module. The collection direction change module can change the channel from which the data collection module collects data, and at what time to increase or decrease the collection amount, which can effectively reduce the data collection repetition rate.

[0037] Working principle: When in use, the data channel modules are respectively multiple different data acquisition ports, and the data acquisition module collects a large amount of different data according to different data acquisition ports, and transmits a large amount of data to the data cloud storage module for storage, and the data cloud storage module can send the collected data together to the cloud computing module, wherein the collected data classification module can classify all the collected data, and transmit the data from multiple different channels to multiple different cloud storage subspaces through the multi-channel transmission module, and the cloud storage space separation module divides the cloud space into multiple different subspaces, and uses the data backup module to back up the same data in multiple different subspaces, which can not only greatly improve the data transmission efficiency and the data calculation speed, but also the collected data enters the cloud storage subspace, and can filter out duplicate data in the shortest time;

[0038] After data comparison, the cloud computing module uses the duplicate data screening module to screen out duplicate data, and finally uses duplicate data deletion to delete duplicate data, while the duplicate data rate calculation module calculates the number of times the same data appears, and passes the information to the data duplication rate table making module, and the data duplication rate table making module can make a statistical table, and finally the data publishing module publishes the data, among which the market published data collection module collects the data that has been published by the market, and the duplicate data information upload module uploads the duplicate data to the data publishing module, the data comparison module compares the two data from the probability of duplication, and the total data proportion calculation module calculates the proportion of the duplicate data information upload module to the data that has been published by the market. When the proportion of duplicate data is large, the normal data information upload module uploads a large amount of non-duplicate data, and when the proportion of duplicate data is small, the normal data information upload module uploads a small amount of non-duplicate data. By updating the data, it is ensured that the data that has been published in the market has more updated information, so as to attract people who need data;

[0039] The data publishing module passes the data to the high-repetition-rate period detection module. The high-repetition-rate period detection module detects the time period and channel from which multiple repeated data are collected, and sends the information to the collection direction change module. The collection direction change module can change the channel from which the data collection module collects data, and the time to increase or decrease the collection volume, which can effectively reduce the data collection repetition rate.

[0040] The above description is only a preferred specific implementation manner of the present invention, but the protection scope of the present invention is not limited thereto. Any technician familiar with the technical field can make equivalent replacements or changes according to the technical scheme and inventive concept of the present invention within the technical scope disclosed by the present invention, which should be covered by the protection scope of the present invention.

Claims

1. A big data information processing system based on cloud computing, comprising a data acquisition module, a data channel module and a data cloud storage module, characterized in that: The data channel module is bidirectionally electrically connected to the data acquisition module, the output end of the data acquisition module is electrically connected to the input end of the data cloud storage module, the output end of the data cloud storage module is electrically connected to the input end of the cloud computing module, the output end of the cloud computing module is electrically connected to the input end of the duplicate data screening module, the output end of the duplicate data screening module is electrically connected to the input end of the duplicate data deleting module, the output end of the duplicate data screening module is electrically connected to the input end of the duplicate data rate calculation module, the output end of the duplicate data rate calculation module is electrically connected to the input end of the data repetition rate table making module, and the output end of the data repetition rate table making module is electrically connected to the input end of the data publishing module; By setting a data acquisition module and a data channel module, the data channel module is respectively a plurality of different data acquisition ports, and the data acquisition module collects a large amount of different data according to different data acquisition ports, and transmits a large amount of data to the data cloud storage module for storage, and the data cloud storage module can send the collected data together to the cloud computing module, and the cloud computing module uses the duplicate data screening module to screen out the duplicate data after data comparison, and finally uses the duplicate data deletion to delete the duplicate data, and the duplicate data rate calculation module calculates the number of times the same data appears, and transmits the information to the data repetition rate table making module, and the data repetition rate table making module can make a statistical table, and finally the data publishing module publishes the data; The data publication module includes a market publication data collection module, a data comparison module, a duplicate data information upload module, a data total proportion calculation module and a normal data information upload module. The output end of the market publication data collection module is electrically connected to the input end of the data comparison module, the output end of the duplicate data information upload module is electrically connected to the input end of the data comparison module, the output end of the data comparison module is electrically connected to the input end of the data total proportion calculation module, the output end of the normal data information upload module is electrically connected to the input end of the data total proportion calculation module, the output end of the data publication module is electrically connected to the input end of the high repetition rate period detection module, the output end of the high repetition rate period detection module is electrically connected to the input end of the collection direction change module, the output end of the collection direction change module is electrically connected to the input end of the data collection module, the data publication module transmits the data to the high repetition rate period detection module, the high repetition rate period detection module detects from which time period and which channel multiple repeated data are collected, and sends the information to the collection direction change module, and the collection direction change module can change from which channel the data collection module collects data, and at which time to increase or reduce the collection amount, which can effectively reduce the data collection repetition rate.

2. The big data information processing system based on cloud computing according to claim 1, characterized in that: The cloud computing module includes a data collection classification module, a multi-channel transmission module, a cloud storage space segmentation module and a data backup module.

3. The big data information processing system based on cloud computing according to claim 2 is characterized in that: The output end of the data collection classification module is electrically connected to the input end of the multi-channel transmission module, and the output end of the multi-channel transmission module is electrically connected to the input end of the cloud storage space separation module.

4. The big data information processing system based on cloud computing according to claim 3 is characterized in that: The output end of the data backup module is electrically connected to the input end of the cloud storage space partition module.

Images