Distributed data acquisition and object visualization method

Abstract

The invention discloses a distributed data acquisition and object visualization method. The method comprises the following steps: S1, arranging an acquisition module; s2, regularly uploading the data; s3, sorting and integrating the data; and S4, automatically converting the data. According to the distributed data collection and object visualization method, related data are synchronously collected through cooperation of multiple types of collection devices, namely a camera and a sensor and a GPS, and meanwhile, related data information uploaded to a webpage by others is crawled through a crawler to be collected, so that the accuracy of the data is ensured; the method comprises the following steps: firstly, collecting multiple types of data, then uploading the collected multiple types of data to a cloud for backup in the S1 stage, then, carrying out concentration and classification integration on the uploaded information through a naive Bayesian algorithm in the S3 stage, and then, in the S4 stage, inputting the integrated data into a value JSON, so that the text is converted into a chart in a set form, and finally, in the S5 stage, the converted chart, the collected video picture and related text information jointly form a display result.

Description

Technical Field

[0001] The present invention relates to the technical field of data collection, and in particular to a distributed data collection and object visualization method. Background Art

[0002] Data collection, also known as data acquisition, is the use of a device to collect data from outside the system and input it into an interface inside the system. Visualization is used as a PPT to complete visualization development. After searching, it is found that the typical data collection method in the prior art is as follows: Publication No. CN108769115A A distributed RSS data collection system, including: at least one collection node module distributed in the network, used to collect RSS data on the network; a registration center, used to save all information of the collection node modules, and provide the information to the scheduling module; a scheduling module, used to assign the collection task to the collection node module according to the information. The distributed RSS data collection method includes: collecting information of the collection node modules distributed in the network through the registration center; pulling the collection task through the scheduling module; pulling the information through the scheduling module; assigning the collection task to the collection node module according to the information; collecting RSS data on the network, and its main features are high stability, high parallelism, and distributed collection of RSS information.

[0003] In summary, after the data collection is completed, the existing data collection method mainly outputs pure text data in the form of text, which requires human thinking and comparison. The data comparison results are not intuitive enough. In view of the above problems, the existing equipment needs to be improved. Summary of the invention

[0004] The purpose of the present invention is to provide a distributed data collection and object visualization method to solve the problem that the existing data collection method proposed in the above background technology mainly outputs pure text data in the form of text after the collection is completed, that is, it requires human thinking and comparison, and the data comparison result is not intuitive enough.

[0005] To achieve the above object, the present invention provides the following technical solution: a distributed data collection and object visualization method, comprising:

[0006] S1. Collection module layout: According to the actual collection requirements, the sensor modules and camera components are selectively arranged to collect video data and parameter data. At the same time, GPS is set around the sensor modules and camera components;

[0007] S2, data upload at regular intervals: The data collected in the S1 stage is uploaded to the cloud for storage at regular intervals with GPS;

[0008] S3, data sorting and integration: The data uploaded to the cloud in the S1 stage is condensed, integrated and classified through the Naive Bayes algorithm, and the integrated and classified data is uploaded to the cloud for storage;

[0009] S4. Automatic data conversion: visualize the integrated data through JSON objects and convert them into charts of set formats;

[0010] S5. Visual processing of objects: Combine the converted charts with the key words and phrases extracted after concentration to present the visualization of IoT data.

[0011] Preferably, the sensor module includes a first sensor, a second sensor and an Nth sensor, and the number of sensors is determined according to actual conditions. The sensors include a distance sensor, a temperature sensor, a humidity sensor and a pressure sensor.

[0012] Preferably, the camera assembly includes a first camera, a second camera and an Nth camera, and the number of cameras is determined according to actual conditions. At the same time, the videos and information collected by the cameras and sensors are uploaded via GPS.

[0013] Preferably, the GPS adds a timestamp to the data collected by different cameras and sensors, and the timestamp content includes location information and time information.

[0014] Preferably, the data collection in the S1 stage also includes a crawler program, that is, using a crawler to copy the relevant data information uploaded to the web page.

[0015] Preferably, the cloud is provided with a first storage area and a second storage area, and the first storage area is used to store the source data uploaded in the S2 stage, while the second storage area is used to store the data after integration and classification in the S3 stage.

[0016] Preferably, in the S3 stage, the data is extracted and concentrated by a keyword extraction algorithm, and the classification integration formula of the naive Bayes algorithm in the S3 stage is:

[0017]

[0018] Among them, P(Y|X) is the probability of event Y, assuming that X has occurred, P(X) is the probability of event X, P(Y) is the probability of event Y, P(X|Y) is the possibility of event X given a fixed value of Y, and X represents a feature variable and Y represents a target variable.

[0019] Preferably, the The search logic is: evaluate the probability of an event being needed based on the degree of association that may be related to the event.

[0020] Preferably, when there are two feature variables X1 and X2 in the naive Bayes algorithm, the data classification integration formula can be transformed into:

[0021]

[0022] Among them, X1 and X2 are independent of each other.

[0023] Preferably, the JSON conversion chart types in the S4 stage include bar charts, line charts and pie charts.

[0024] Compared with the prior art, the invention has the following beneficial effects: the distributed data collection and object visualization method,

[0025] First, in the S1 stage, the application uses various types of collection devices, namely cameras and sensors, in conjunction with GPS to synchronously collect relevant data, and at the same time uses crawlers to crawl relevant data information uploaded to the web page by others to ensure the accuracy of the data. Then, in the S1 stage, the collected multi-type data is uploaded to the cloud for backup, and then in the S3 stage, the uploaded information is condensed and classified and integrated through the Naive Bayes algorithm to facilitate subsequent conversion. Then, in the S4 stage, the integrated data input value JSON is converted into a chart of a set form. Finally, in the S5 stage, the converted chart is combined with the collected video images and related text information to form a display result, thereby improving the intuitiveness of the result, and thereby solving the problem that the existing data collection method mainly outputs pure text data in the form of text after the collection is completed, that is, it requires manual thinking and comparison, and the data comparison result is not intuitive enough. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] Figure 1 It is a schematic diagram of the overall process of the present invention;

[0027] Figure 2 This is a schematic diagram of the structure of the acquisition module of the present invention;

[0028] Figure 3 Schematic diagram of polynomial naive Bayes of the present invention;

[0029] Figure 4 This is a schematic diagram of the Bernoulli Naive Bayes of the present invention. DETAILED DESCRIPTION

[0030] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0031] See also Figure 1-4 The present invention provides a technical solution: a distributed data collection and object visualization method, comprising:

[0032] S1. Collection module layout: According to the actual collection requirements, the sensor modules and camera components are selectively arranged to collect video data and parameter data. At the same time, GPS is set around the sensor modules and camera components;

[0033] S2, data upload at regular intervals: The data collected in the S1 stage is uploaded to the cloud for storage at regular intervals with GPS;

[0034] S3, data sorting and integration: The data uploaded to the cloud in the S1 stage is condensed, integrated and classified through the Naive Bayes algorithm, and the integrated and classified data is uploaded to the cloud for storage;

[0035] S4. Automatic data conversion: visualize the integrated data through JSON objects and convert them into charts of set formats;

[0036] S5. Visual processing of objects: Combine the converted charts with the key words and phrases extracted after concentration to present the visualization of IoT data.

[0037] The sensor module includes a first sensor, a second sensor and an Nth sensor, and the number of sensors is determined according to actual conditions. The sensors include a distance sensor, a temperature sensor, a humidity sensor and a pressure sensor.

[0038] The camera assembly includes a first camera, a second camera and an Nth camera, and the number of cameras is determined according to actual conditions. At the same time, the videos and information collected by the cameras and sensors are uploaded via GPS.

[0039] GPS adds a timestamp to the data collected by different cameras and sensors, and the timestamp content includes location information and time information.

[0040] Data collection in the S1 stage also includes crawler programs, which use crawlers to copy relevant data information uploaded to the web page.

[0041] The cloud is provided with a first storage area and a second storage area, and the first storage area is used to store the source data uploaded in the S2 stage, while the second storage area is used to store the data after integration and classification in the S3 stage.

[0042] In the S3 stage, the data is extracted and concentrated through the keyword extraction algorithm, and the classification integration formula of the naive Bayes algorithm in the S3 stage is:

[0043]

[0044] Among them, P(Y|X) is the probability of event Y, assuming that X has occurred, P(X) is the probability of event X, P(Y) is the probability of event Y, P(X|Y) is the possibility of event X given a fixed value of Y, and X represents a feature variable and Y represents a target variable.

[0045] The search logic is: evaluate the probability of an event being needed based on the degree of association that may be related to the event.

[0046] When there are two feature variables X1 and X2 in the Naive Bayes algorithm, the data classification integration formula can be transformed into:

[0047]

[0048] Among them, X1 and X2 are independent of each other.

[0049] The types of JSON conversion charts in the S4 stage include bar charts, line charts, and pie charts.

[0050] The data method in the S3 stage can also use multinomial naive Bayes, which is based on Bayesian theory, but assumes that the probability distribution obeys a simple multinomial distribution. The multinomial distribution comes from the multinomial experiment in statistics. This experiment can be specifically explained as follows: the experiment includes n repeated experiments, each with different possible results. In any given experiment, the probability of a specific result is constant; the characteristics are as follows:

[0051] 1) Multinomial distribution is good at dealing with categorical variables. In its principle assumption, the probability of is discrete and independent of each other under different conditions. Although the multinomial distribution in sklearn can also handle continuous variables, in reality, if we really want to handle continuous variables, we should use Gaussian Naive Bayes.

[0052] 2) The experimental results in polynomial experiments are very specific. The features involved are often concepts such as number, frequency, count, and occurrence. These concepts are all discrete positive integers. Therefore, the multinomial naive Bayes in sklearn does not accept negative input.

[0053] In addition, the features of multinomial naive Bayes are mostly sparse matrices, but of course they are not always sparse matrices.

[0054] You can use the famous TF-IDF vector technology, or you can use the common and simple word count vector method in conjunction with Bayes. Both of the above methods are common text feature extraction methods, which can be easily implemented through sklearn. At the same time, the formula of TF-IDF vector technology is: TF-IDF = TF*IDF;

[0055] Among them, TF is the word frequency, which refers to the number of times a word appears in an article / the total number of words in the article, and IDF is the inverse document frequency, which refers to log(the total number of documents in the corpus / (the number of documents containing the word + 1));

[0056] The extraction logic of the formula TF-IDF=TF*IDF is: first, perform data preprocessing operations such as word segmentation, part-of-speech screening, deduplication and stop word removal on the text of the keywords to be extracted to obtain candidate keywords; secondly, calculate the TF-IDF value of all candidate keywords and take the keywords with the largest TF-IDF value TopN among all candidate keywords as the final keywords of the document;

[0057] The extraction algorithm of TF-IDF=TF*IDF will automatically prioritize the importance of words in the global document, and can be manually modified after extraction.

[0058] In sklearn, the class MultinomialNB used to perform multinomial naive Bayes contains the following parameters and attributes:

[0059] sklearn.naive_bayes.MultinomialNB(alpha=1.0

[0060] ,fit_prior=True

[0061] ,class_prior=None

[0062] ).

[0063] alpha is a floating point number, which can be left blank (default is 1.0). It is the parameter for Laplace or Lidstone smoothing. If it is set to 0, it means no smoothing option at all. However, it should be noted that smoothing is equivalent to adding some noise to the probability artificially. Therefore, the larger the setting, the lower the accuracy of multinomial naive Bayes will be (although the impact is not very large), and the Brill score will gradually increase;

[0064] fit_prior, Boolean value, optional (default is True) Whether to learn the prior probability P(Y=c). If set to false, the prior probability is not used, but the uniform prior probability (uniformprior), that is, the probability of each label class appearing is considered to be 1 / n_classes.

[0065] class_prior is an array-like structure (n_classes,). It can be left blank (default is None). The class prior probability P(Y=c) is automatically calculated based on the data if no specific prior probability is given. Figure 3 shown.

[0066] The data method of the S3 stage can also use Bernoulli Naive Bayes. In simple terms, multivariate Bernoulli distribution means that there can be multiple features in the data set, but each feature is binary and can be represented by a Boolean variable or any binary combination such as {0, 1} or {-1, 1}. Therefore, this class requires that the sample be converted into a binary feature vector. If the data itself is not binary, the parameter binarize in the class specifically used for binarization can be used to change the data.

[0067] In sklearn, the description of the Bernoulli Naive Bayes class is:

[0068] sklearn.naive_bayes.BernoulliNB(alpha=1.0

[0069] ,binarize=0.0

[0070] ,fit_prior=True

[0071] ,class_prior=None )

[0073] alpha is a floating point number, which can be left blank (default is 1.0). It is the parameter for Laplace or Lidstone smoothing. If it is set to 0, it means no smoothing option at all. However, it should be noted that smoothing is equivalent to adding some noise to the probability artificially. Therefore, the larger the setting, the lower the accuracy of multinomial naive Bayes will be (although the impact is not very large), and the Brill score will gradually increase;

[0074] Binarize, floating point number or None, optional, defaults to 0, the threshold for binarizing features. If None is set, it is assumed that the feature has been binarized.

[0075] fit_prior, Boolean value, optional (default is True) whether to learn the prior probability P(Y=c). If set to false, the prior probability is not used, but the uniform prior probability (uniformprior), that is, the probability of each label class appearing is 1 / n_classes;

[0076] class_prior is an array-like structure (n_classes,). It can be left blank (default is None). The class prior probability P(Y=c) is automatically calculated based on the data if no specific prior probability is given. Figure 4 shown.

[0077] Although the present invention has been described in detail with reference to the aforementioned embodiments, it is still possible for those skilled in the art to modify the technical solutions described in the aforementioned embodiments, or to make equivalent substitutions for some of the technical features therein. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A distributed data collection and object visualization method, characterized in that ,include: S1. Collection module layout: According to the actual collection requirements, the sensor modules and camera components are selectively arranged to collect video data and parameter data. At the same time, GPS is set around the sensor modules and camera components; S2, data upload at regular intervals: The data collected in the S1 stage is uploaded to the cloud for storage at regular intervals with GPS; S3, data sorting and integration: The data uploaded to the cloud in the S1 stage is condensed, integrated and classified through the Naive Bayes algorithm, and the integrated and classified data is uploaded to the cloud for storage; S4. Automatic data conversion: visualize the integrated data through JSON objects and convert them into charts of set formats; S5. Visual processing of objects: Combine the converted charts with the key words and phrases extracted after concentration to present the visualization of IoT data.

2. A distributed data collection and object visualization method according to claim 1, characterized in that: The sensor module includes a first sensor, a second sensor and an Nth sensor, and the number of sensors is determined according to actual conditions. The sensors include a distance sensor, a temperature sensor, a humidity sensor and a pressure sensor.

3. A distributed data collection and object visualization method according to claim 1, characterized in that: The camera assembly includes a first camera, a second camera and an Nth camera, and the number of cameras is determined according to actual conditions. At the same time, the videos and information collected by the cameras and sensors are uploaded via GPS.

4. A distributed data collection and object visualization method according to claim 1, characterized in that: The GPS adds a timestamp to the data collected by different cameras and sensors, and the timestamp content includes location information and time information.

5. A distributed data collection and object visualization method according to claim 1, characterized in that: The data collection in the S1 stage also includes a crawler program, that is, using a crawler to copy the relevant data information uploaded to the web page.

6. A distributed data collection and object visualization method according to claim 1, characterized in that: The cloud is provided with a first storage area and a second storage area, and the first storage area is used to store the source data uploaded in the S2 stage, while the second storage area is used to store the data after integration and classification in the S3 stage.

7. A distributed data collection and object visualization method according to claim 1, characterized in that: In the S3 stage, the data is extracted and concentrated by the keyword extraction algorithm, and the classification integration formula of the naive Bayes algorithm in the S3 stage is: Among them, P(Y|X) is the probability of event Y, assuming that X has occurred, P(X) is the probability of event X, P(Y) is the probability of event Y, P(X|Y) is the possibility of event X given a fixed value of Y, and X represents a feature variable and Y represents a target variable.

8. A distributed data collection and object visualization method according to claim 1, characterized in that: Said The search logic is: evaluate the probability of an event being needed based on the degree of association that may be related to the event.

9. A distributed data collection and object visualization method according to claim 1, characterized in that: When there are two feature variables X1 and X2 in the naive Bayes algorithm, the data classification integration formula can be transformed into: Among them, X1 and X2 are independent of each other.

10. A distributed data collection and object visualization method according to claim 1, characterized in that: The types of JSON conversion charts in the S4 stage include bar charts, line charts, and pie charts.

Images