Data mining method and device based on data model, computer equipment and medium
By using GPU and advanced word segmentation and vectorization technology in big data processing, combined with Apache Kafka and big data processing platform, the problem of inefficiency in deep learning models in big data processing is solved, and more efficient data analysis and characteristics are achieved Extraction enhances the comprehensive ability and interactivity of data analysis.
Patent Information
- Application Number
- CN202410913429.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-09
- Publication Date
- 2024-11-01
- Estimated Expiration
- Not applicable · inactive patent
AI Technical Summary
Existing deep learning models are inefficient in big data processing, especially in the data preprocessing and model training stages, and it is difficult to effectively cope with the diversity and complexity of big data, resulting in insufficient processing efficiency and accuracy.
GPU is used to normalize image data, use GPT for word segmentation of text data, and vectorized through TF-IDF or Word2Vec, combined with Apache Kafka and big data processing platform for data analysis, and finally use intelligent text mining module Perform visual chart analysis to improve the efficiency and accuracy of data mining.
By accelerating data preprocessing and feature extraction, the speed and quality of data analysis are improved, the comprehensive capabilities and interactivity of data analysis are enhanced, and the growing and diversified data needs are better able to respond to the growing and diversified data needs.
Smart Images

Figure CN118885643A_ABST
Abstract
Description
Technical Field
[0001] The embodiments of the present invention relate to the technical field of big data analysis and mining, and specifically, to a data mining method, device, computer equipment and medium based on a data model. Background Art
[0002] Big data refers to a collection of data that is huge in size, diverse in types, and complex to process. It usually has three characteristics: large volume, high speed, and diversity. These data sets usually come from a variety of sources, including but not limited to social media, sensors, communication records, etc. Due to its huge volume, traditional data processing methods can no longer meet the needs of effective management and analysis.
[0003] In the related technologies of mining and analyzing big data, deep learning models have become a powerful tool. Through a multi-level neural network structure, deep learning models can learn and extract high-level abstract features from large-scale data, and have strong adaptability and generalization capabilities. In the field of data mining, deep learning models are widely used in tasks such as image recognition, natural language processing, and time series data analysis, and have achieved remarkable results.
[0004] However, despite the excellent performance of deep learning models in big data mining and analysis, there are still some inefficiencies. One of them is the slow processing speed for big data, especially in the data preprocessing and model training stages. Due to the huge size of big data collections, traditional deep learning models often consume a lot of computing resources and time, resulting in low processing efficiency. In addition, the diversity and complexity of big data also bring challenges to the design and optimization of deep learning models, and more efficient and flexible methods are needed to cope with the feature extraction and pattern recognition of different types of data. Therefore, deep learning models for big data mining and analysis still need further research and optimization to improve processing efficiency and accuracy to cope with the growing and diverse data needs. Summary of the invention
[0005] The embodiments of the present invention provide a data mining method, apparatus, computer equipment and medium based on a data model, so as to at least solve the problem in the related art of improving processing efficiency and accuracy to cope with the ever-increasing and diversified data needs.
[0006] According to one embodiment of the present invention, a data mining method based on a data model is provided, including: identifying input data, determining that the input data is image data and / or text data based on Apache Kafka and a multimodal data classification component; normalizing the image data based on a GPU, performing word segmentation processing on the text data based on GPT and vectorizing processing based on TF-IDF or Word2Vec to obtain preprocessed data; analyzing the preprocessed data based on a big data processing platform module to obtain analysis results; converting the analysis results into visual charts; and analyzing the contents of the visual charts based on an intelligent text mining module to obtain mining results.
[0007] According to another embodiment of the present invention, a data mining device based on a data model is provided, comprising: a multimodal data classification module, used to identify input data, and determine whether the input data is image data and / or text data based on Apache Kafka and a multimodal data classification component; and normalize the image data based on a GPU, and perform word segmentation processing on the text data based on GPT and vectorization processing based on TF-IDF or Word2Vec to obtain preprocessed data; a big data processing platform module, used to analyze the preprocessed data to obtain analysis results; and, The analysis results are converted into visual charts; and an intelligent text mining module is used to analyze the contents of the visual charts to obtain mining results.
[0008] According to yet another embodiment of the present invention, a computer-readable storage medium is provided, in which a computer program is stored, wherein the computer program is configured to execute the steps of any one of the above method embodiments when run.
[0009] According to another embodiment of the present invention, a computer device is provided, including a memory and a processor, wherein the memory stores a computer program, and the processor is configured to run the computer program to execute the steps in any one of the above method embodiments.
[0010] Through the present invention, firstly, by utilizing GPU and advanced word segmentation and vectorization technology, the preprocessing speed of image and text data is accelerated and the quality of preprocessed data is improved. Secondly, by adopting lightweight and efficient models and distributed computing frameworks, the speed and efficiency of feature extraction are improved while ensuring high-quality feature representation. Finally, by using multimodal feature fusion and dynamic visualization tools, the comprehensive ability and interactivity of data analysis are enhanced, allowing users to understand and utilize analysis results more comprehensively. In summary, these improvements enable the present invention to have stronger adaptability and competitive advantages in the face of growing and diversified data demands. Therefore, the problem of improving processing efficiency and accuracy in order to cope with growing and diversified data demands in the related art is solved. BRIEF DESCRIPTION OF THE DRAWINGS
[0011] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the related technologies, the drawings required for use in the embodiments or the related technical descriptions are briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.
[0012] Figure 1 is a flow chart of a data mining method based on a data model according to an embodiment of the present invention; Figure 2 is a flowchart of a method for determining whether input data is image data and / or text data based on Apache Kafka and a multimodal data classification component according to an embodiment of the present invention; Figure 3 is a flowchart of a method for obtaining input data based on Apache Kafka according to an embodiment of the present invention; Figure 4 is a flow chart of a method for analyzing pre-processed data based on a big data processing platform module to obtain analysis results according to an embodiment of the present invention; Figure 5 is a flow chart of a method for performing parallel data processing on preprocessed data using Apache Spark to obtain analysis results according to an embodiment of the present invention; Figure 6 is a flow chart of a method for converting analysis results into a visual chart according to an embodiment of the present invention; Figure 7 is a flow chart of a method for analyzing the content of a visual chart based on an intelligent text mining module to obtain mining results according to an embodiment of the present invention; Figure 8 It is a structural schematic diagram of a data mining device based on a data model according to an embodiment of the present invention. DETAILED DESCRIPTION
[0013] Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings and in combination with the embodiments.
[0014] It should be noted that the terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence.
[0015] In this embodiment, a data mining method based on a data model is provided. Figure 1 is a flow chart of a data mining method based on a data model according to an embodiment of the present invention. Figure 1 As shown, the process includes the following steps: Step S101, identifying input data, and determining whether the input data is image data and / or text data based on Apache Kafka and a multimodal data classification component; Among them, Apache Kafka is a distributed stream processing platform for high-throughput data pipelines and stream processing. In this step, Kafka can be used to receive and transmit data. Multimodal data classification component: This component is responsible for analyzing the incoming data stream and determining the type of data. If it is image data, it can use image recognition technology; if it is text data, it will perform corresponding text analysis. Through the combination of these two technologies, the system is able to identify and classify the input data and prepare for subsequent processing.
[0016] Step S102, normalizing the image data based on the GPU, performing word segmentation processing on the text data based on the GPT and performing vectorization processing based on TF-IDF or Word2Vec to obtain preprocessed data; Among them, GPU: Graphics Processing Unit, used to process large amounts of data in parallel, is very suitable for image processing tasks. Normalization: Scale the pixel values of image data to a specific range (usually 0 to 1) to facilitate subsequent machine learning model processing. Word segmentation: Use GPT (a pre-trained language model) to segment text, that is, split the text into processable units such as words or phrases. Vectorization: Convert the segmented text into a numerical vector, which can be achieved using algorithms such as TF-IDF or Word2Vec. These algorithms can convert text into a format that can be understood by machine learning models.
[0017] Specifically, TF-IDF (TermFrequency-InverseDocumentFrequency) is a weighted technology used for information retrieval and text mining. It is a statistical method used to evaluate the importance of a word to a document set or one of the documents in a corpus. Its main idea is: if a word appears frequently in an article (TermFrequency, TF) and rarely appears in other articles (InverseDocumentFrequency, IDF), then it is considered that this word has a good ability to distinguish categories and has a good indication of the content of the article.
[0018] The calculation of TF-IDF is divided into two parts: Term Frequency (TF): Indicates how often a term (keyword) appears in a document. This number is usually normalized (usually the term frequency divided by the total number of terms in the document) to prevent it from being biased towards long documents. The formula is: TF = .
[0019] Inverse Document Frequency (IDF): Indicates the general importance of a term. The fewer documents contain a term, the greater the IDF, indicating that the term has a good ability to distinguish categories. The formula is: IDF = log_ n , where n is greater than 0 and not equal to 1.
[0020] By multiplying TF and IDF, we can get the importance score of a term in a document: TF-IDF=TF×IDF.
[0021] This score is used to evaluate the importance of a term in a document. For a term t in a document d, its TF-IDF can be calculated using the following formula: TF-IDF = ×log_ n , where n is greater than 0 and not equal to 1.
[0022] TF-IDF vectors can be used in tasks such as information retrieval, text clustering, and document classification to help machine learning algorithms better understand text data.
[0023] Word2Vec is an algorithm for generating word embeddings, which learns the vector representation of words by training the model. The Word2Vec algorithm can capture the semantic relationship between words, so that semantically similar words are also close in distance in the vector space.
[0024] There are two main architectures of Word2Vec: Continuous Bag of Words (CBOW) and Skip-Gram.
[0025] Among them, continuous bag of words (CBOW): The goal is to predict the target word based on the context words.
[0026] Given a target word, the CBOW model considers the context words around the target word and tries to predict the target word.
[0027] The input of the model is the vector representation of the context word, and the output is the vector representation of the target word.
[0028] Skip-Gram: The goal is to predict context words based on the target word.
[0029] In contrast to CBOW, the Skip-Gram model is given a target word and then predicts the context words around the target word.
[0030] This approach is usually more effective in capturing the semantic relationships of rare words.
[0031] The Word2Vec training process includes the following steps: Initialization: Randomly initialize the vector representation of words.
[0032] Training: The model is trained using a large amount of text data. For each word, the model predicts the context or is predicted by the context based on the selected architecture (CBOW or Skip-Gram).
[0033] Optimization: Adjust the word vectors through optimization algorithms such as gradient descent to minimize the prediction error.
[0034] Learning: The word vectors finally learned by the model can be used for various downstream tasks, such as text classification, sentiment analysis, machine translation, etc.
[0035] The word vectors generated by Word2Vec can capture various semantic relationships between words, such as synonyms, antonyms, hyponyms, etc. These word vectors can be used as input features of other machine learning models to improve the performance of the model in natural language processing tasks.
[0036] Step S103, analyzing the pre-processed data based on the big data processing platform module to obtain analysis results; Among them, big data processing platforms such as Apache Hadoop or Apache Spark are used to process and analyze the preprocessed data. These platforms are capable of handling large-scale data sets and provide the necessary tools to perform complex data analysis tasks.
[0037] Step S104, converting the analysis results into a visual chart; The analysis results (which can be statistical data, pattern recognition results, etc.) are converted into visual charts. Data visualization tools (such as Tableau, Power BI, or customized visualization libraries) are used to create charts to make the results easier to understand and analyze.
[0038] Step S105, analyzing the content of the visualization chart based on the intelligent text mining module to obtain mining results.
[0039] Among them, intelligent text mining technology, such as natural language processing (NLP) algorithms, is used to conduct in-depth analysis of the content in the visualization chart. This can include sentiment analysis, topic modeling, keyword extraction, etc., to further mine valuable information in the data.
[0040] Through the present invention, firstly, by utilizing GPU and advanced word segmentation and vectorization technology, the preprocessing speed of image and text data is accelerated and the quality of preprocessed data is improved. Secondly, by adopting lightweight and efficient models and distributed computing frameworks, the speed and efficiency of feature extraction are improved while ensuring high-quality feature representation. Finally, by using multimodal feature fusion and dynamic visualization tools, the comprehensive ability and interactivity of data analysis are enhanced, allowing users to understand and utilize analysis results more comprehensively. In summary, these improvements enable the present invention to have stronger adaptability and competitive advantages in the face of growing and diversified data demands. Therefore, the problem of improving processing efficiency and accuracy in order to cope with growing and diversified data demands in the related art is solved.
[0041] Figure 2 is a flowchart of a method for determining whether input data is image data and / or text data based on Apache Kafka and a multimodal data classification component according to an embodiment of the present invention. In one implementation, Figure 2 As shown, based on Apache Kafka and the multimodal data classification component, determining that the input data is image data and / or text data includes: Step S201, creating a corresponding topic based on input data through Apache Kafka; Among them, step S201 can be used to create a topic in Apache Kafka for storing and transmitting different types of data. In an exemplary implementation, it can be implemented in the following way: 1. Create topics: Create one or more topics in the Kafka cluster. Each topic can represent one type of data (for example, one topic for image data and another for text data).
[0042] 2. Configure the topic: Set the configuration parameters of the topic, such as the number of partitions, replication factor, etc., to meet the requirements of data throughput and reliability.
[0043] Step S202, routing different information of the input data to different topics, so that the multimodal data classification component obtains data from different topics to obtain classification results; Among them, step S202 can be used to route the input data to the corresponding Kafka topic according to the type for classification processing. In an exemplary implementation, it can be implemented in the following way: 1. Data routing: Develop or use existing producer clients based on preset parameters to send data to the corresponding Kafka topic based on the type of input data (image or text).
[0044] 2. Multimodal data classification component: This component can be a service or application that reads data from Kafka topics by subscribing to them and performs preliminary classification.
[0045] Step S203, sending the classification result to the classified-data topic, wherein the classification result includes image data and text data, and the topic includes the classified-data topic.
[0046] Among them, the output of the classification component, that is, the classification result, can be sent to a dedicated Kafka topic in step S203 for further processing. In an exemplary implementation, it can be implemented in the following manner: 1. Classification result processing: After completing data classification, the multimodal data classification component generates classification results, which include the identification of image data and text data.
[0047] 2. Send to the classified-data topic: Use the Kafka producer client to send the classification results to the topic called classified-data. This topic will serve as the data source for subsequent steps for further analysis and processing by other system components or services.
[0048] From step S201 to step S203, Apache Kafka acts as a central data pipeline, responsible for receiving, storing and transmitting data. The multimodal data classification component, as the starting point of data processing, is responsible for identifying and classifying input data to ensure that the data can be correctly routed and processed. In this way, the system can effectively process large-scale multimodal data and provide structured and classified data input for subsequent data mining steps.
[0049] Figure 3 is a flowchart of a method for obtaining input data based on Apache Kafka according to an embodiment of the present invention. In one implementation, Figure 3 As shown, before creating a topic based on input data through Apache Kafka, include: Step S301, acquiring input data from different sources and in different formats based on an input adapter; Wherein, step S301 can be used to obtain data from various data sources and prepare to send it to Kafka. In an exemplary implementation, it can be implemented in the following manner: 1. Input adapters: Develop or use existing input adapters based on preset parameters. These adapters can interact with different data sources (such as databases, file systems, APIs, etc.) and can read data in different formats (such as JSON, CSV, XML, etc.).
[0050] 2. Data acquisition: Extract data from the data source through the adapter according to the configured or predetermined rules.
[0051] Step S302, standardizing the input data based on the formatting engine to obtain data in a unified format; Among them, step S302 can be used to convert data in different formats into a unified format for subsequent processing. In an exemplary implementation, it can be implemented in the following manner: 1. Formatting engine: Design or use a formatting engine that can parse various data formats and convert them into a unified internal data model or format.
[0052] 2. Data conversion: The engine parses and converts the input data to ensure that all data follows the same structure and standards, such as unified field names, data types, and encodings.
[0053] Step S303: Send the data in a unified format to Apache Kafka based on the output adapter.
[0054] Among them, the standardized data can be sent to Kafka in step S303 for further processing and analysis. In an exemplary implementation, it can be implemented in the following manner: 1. Output Adapter: Develop or use an output adapter based on preset parameters, which is compatible with the Apache Kafka interface and sends data to the Kafka cluster.
[0055] 2. Data sending: The adapter encapsulates the standardized data into a message format that Kafka can process, and then sends these messages to Kafka according to the predetermined topic and configuration.
[0056] 3. Kafka producer: Use the output adapter to implement the Kafka producer API, which is responsible for maintaining the connection with the Kafka cluster and ensuring reliable data transmission.
[0057] Through steps S301 to S303, the original and diverse input data can be converted and adapted into a unified format suitable for Kafka processing, and then sent to the Kafka topic, laying the foundation for the subsequent data mining process. These steps are key links to ensure data quality and consistency, and are crucial to the success of the entire data mining method. Data transmitted from multiple platforms are usually data in different formats. By converting and adapting the input data into a unified format suitable for Kafka processing, the data can be processed across platforms, so as to facilitate multi-platform collaborative processing, so as to improve data processing efficiency and adaptability.
[0058] Figure 4 is a flow chart of a method for analyzing preprocessed data based on a big data processing platform module to obtain analysis results according to an embodiment of the present invention. In one implementation, Figure 4 As shown, the pre-processed data is analyzed based on the big data processing platform module to obtain analysis results, including: Step S401, using Apache Spark to perform parallel data processing on the preprocessed data to obtain analysis results; Wherein, step S401 can utilize the parallel processing capability of Apache Spark to efficiently process large-scale data sets. In an exemplary implementation, it can be implemented in the following manner: 1. Apache Spark: It is an open source distributed computing system that provides a fast, general and scalable big data processing platform.
[0059] 2. Data loading: First, load the preprocessed data into the Spark cluster. This can be achieved through Spark APIs, such as using SparkContext's textFile or wholeTextFiles method to read data files.
[0060] 3. Parallel processing: Use Spark's parallel data processing capabilities, such as map, reduce, filter and other transformation operations, as well as reduceByKey, groupByKey and other action operations, to process data. These operations can be executed in parallel on multiple nodes, thereby speeding up processing.
[0061] 4. Data Analysis: It can perform more complex data analysis tasks, such as machine learning algorithms, graph processing, stream processing, etc. Spark provides a rich library to support these advanced functions.
[0062] Specifically, in Apache Spark, SparkContext is the main entry point for applications to interact with the Spark cluster. It is the core of the Spark application and is responsible for initializing the connection to the cluster, allocating tasks, and coordinating data in the cluster.
[0063] Here are some key features and uses of SparkContext: Initialization: SparkContext is the first step to start a Spark application. When creating a SparkContext instance, you need to specify the configuration information of the application, such as the cluster manager type, resource allocation, etc.
[0064] Cluster connection: SparkContext is responsible for establishing a connection to a Spark cluster, whether in local mode, standalone deployment mode, or a cluster manager such as YARN or Mesos.
[0065] Resource allocation: Through SparkContext, you can control the allocation of resources, such as memory, number of CPU cores, etc.
[0066] Task Scheduling: SparkContext is responsible for converting the operations in the application into tasks and scheduling these tasks to be executed on various nodes in the cluster.
[0067] Data caching: SparkContext provides persist and unpersist operations, allowing developers to persist RDD (Resilient Distributed Dataset) or DataFrame in memory to speed up iterative computing.
[0068] Broadcast variables: SparkContext supports broadcast variables, an optimization that allows large datasets to be distributed to all worker nodes instead of being retransmitted with each task.
[0069] Accumulators: SparkContext allows the creation of accumulators, which are variables that can only be aggregated from various nodes in the cluster.
[0070] Close: When the application is finished executing, you can close the SparkContext by calling SparkContext.stop() to release cluster resources.
[0071] In Apache Spark, RDD (Resilient Distributed Dataset) is a basic data structure used to process and store a distributed collection of objects, which can be key-value pairs or just values. RDD provides fault tolerance and can process data in a parallel and distributed manner.
[0072] Following are some key features of RDDs: Distributed: RDDs are distributed across multiple nodes in the Spark cluster, with each node storing a subset of the RDD.
[0073] Resilience: RDD is fault-tolerant. If a node fails, the RDD part it stores can be recovered from other nodes.
[0074] Immutability: RDD is immutable and cannot be modified once created. All RDD transformation operations will generate a new RDD.
[0075] Parallel operations: RDD supports parallel operations and can be executed efficiently on a cluster.
[0076] Lazy loading: RDD calculations are lazy, meaning that calculations are performed only when needed (action operations).
[0077] Transformation operations: RDD supports a variety of transformation operations, such as map, filter, reduce, groupByKey, etc.
[0078] Action operations: Action operations (such as count, collect, saveAsTextFile, etc.) can trigger RDD calculations.
[0079] Dependencies: Transformation operations between RDDs can form dependencies, and Spark will optimize based on these dependencies.
[0080] In-memory computing: RDD supports in-memory computing. When possible, Spark caches RDD in memory to speed up iterative algorithms.
[0081] Data structure: RDD can contain a variety of data types, including basic data types, complex data types, and even user-defined data types.
[0082] In Apache Spark, textFile is a method that belongs to the SparkContext class that reads a text file and returns it as an RDD (Resilient Distributed Dataset). Each file is split into multiple chunks and then loaded in parallel to different nodes in the cluster. Each chunk is converted into a partition of an RDD.
[0083] Here are some key points about the textFile method: Reading Text Files: The textFile method can read text files from the local file system, HDFS, or any other file system supported by Hadoop.
[0084] Return RDD: The textFile method returns an RDD consisting of text lines, where each line is a record in the RDD.
[0085] Parallel processing: Since RDD is distributed, the textFile method allows parallel processing of large-scale text data.
[0086] Flexible file formats: textFile can read files in any text format, including plain text files, log files, etc.
[0087] Optional minimum number of partitions parameter: The textFile method can accept an optional second parameter that specifies the minimum number of partitions of the RDD. This helps avoid too many small tasks when reading small files.
[0088] In Apache Spark, wholeTextFiles method is similar to textFile method in that both are used to read text files, but they differ in the way they process files. wholeTextFiles method belongs to SparkContext class, and it is used to return the entire file as a key-value pair, where the key is the path to the file and the value is the content of the file.
[0089] Here are some key features of the wholeTextFiles method: Return key-value pairs: Unlike textFile, which returns an RDD consisting of lines, wholeTextFiles returns an RDD consisting of (key, value) pairs, where the key is the path to the file and the value is the contents of the entire file.
[0090] Good for small files: wholeTextFiles is particularly suitable for processing small files because it loads the entire file content into memory as a string. For large files, this approach may cause memory problems.
[0091] File path as key: Since each RDD element contains the file path, this makes wholeTextFiles suitable for scenarios that require file metadata or need to operate based on the file name.
[0092] File contents as values: The entire contents of the file are read as a string and can be processed as the value of an RDD element.
[0093] In Apache Spark, reduceByKey is a method used to transform and aggregate key-value pair RDDs. It belongs to the PairRDDFunctions class, which is an extension of the RDD class and provides additional operations for key-value pair RDDs.
[0094] The main function of the reduceByKey method is to aggregate all values with the same key and apply a given binary reduce function to these values, which is usually used to merge two values. This is different from groupByKey, which collects all values with the same key into an iterator, while reduceByKey merges these values into a single value.
[0095] Here are some key features of reduceByKey: Key-value aggregation: reduceByKey aggregates all values with the same key and merges them using the specified reduce function.
[0096] Parallelism: The reduceByKey operation is executed in parallel, which means it can take advantage of multiple nodes in the cluster to speed up the aggregation process.
[0097] Flexibility: The reduce function can be any binary function, allowing developers to customize the merge logic.
[0098] Result: reduceByKey returns a new RDD of key-value pairs where each key is associated with the result of applying the reduce function to all associated values.
[0099] groupByKey is a method in Apache Spark that performs grouping operations on key-value pairs RDD (Resilient Distributed Dataset). This method aggregates all values based on the key and returns a new RDD where each element is a pair of a key and a list of values corresponding to that key.
[0100] Following are some key features of the groupByKey method: Grouping: groupByKey groups together all values with the same key.
[0101] Return type: The RDD element returned by groupByKey is a key and an iterator containing all corresponding values.
[0102] Performance considerations: Because groupByKey may produce a large amount of data in a single partition, it may cause a lot of data movement (transfer over the network) and memory usage, especially when processing large datasets.
[0103] Usage scenarios: groupByKey is suitable for scenarios where data needs to be aggregated based on keys, for example, to calculate the sum, average, or any other aggregation operation for each key.
[0104] Scalability: groupByKey is executed in a distributed manner and can utilize multiple nodes in the cluster to process data in parallel.
[0105] Step S402, using RDD of Apache Spark to perform in-memory computing during parallel data processing.
[0106] Among them, in step S402, a resilient distributed dataset (RDD) can be used to optimize memory computing and improve data processing performance. In an exemplary implementation, it can be implemented in the following manner: 1. Create RDD: In Spark, RDD is a basic data abstraction that represents an immutable, distributed data set. You can create RDD by operating on a data set, such as using the parallelize method to convert a collection into an RDD.
[0107] 2. Memory computing: RDD provides a fault-tolerant memory computing mechanism. By dividing data among nodes for parallel processing, computing tasks can be performed quickly in memory, which is much faster than data operations stored on disk.
[0108] 3. Transformation and Action: Applying transformation operations (such as map and filter) to RDD will generate new RDD, while action operations (such as count and collect) will trigger actual calculations and return results.
[0109] 4. Caching and persistence: To further optimize performance, you can use the RDD cache or persist method to keep the data in memory so that subsequent calculations can avoid reading the data repeatedly.
[0110] Through steps S401 and S402, Apache Spark can efficiently process and analyze large-scale pre-processed data sets, providing powerful computing support for data mining methods, as a key link in achieving fast and scalable big data processing.
[0111] Figure 5 is a flow chart of a method for performing parallel data processing on preprocessed data using Apache Spark to obtain analysis results according to an embodiment of the present invention. In one implementation, Figure 5 As shown, Apache Spark is used to perform parallel data processing on the preprocessed data to obtain analysis results, including: Step S501, mapping, filtering, and aggregating the pre-processed data to obtain converted data; Wherein, step S501 can be used to read data from cloud storage (such as Amazon S3, Google Cloud Storage, etc.) and create RDD. In an exemplary implementation, for example, using Spark's SparkContext object, calling methods such as textFile or wholeTextFiles to read data files stored in the cloud, thereby creating RDD.
[0112] Step S502, using Spark SQL, DataFrames, and Datasets API to batch process the converted data to obtain batch processed data; In step S502, a conversion operation such as map, filter, flatMap, etc. may be applied to the RDD to convert the data. For example, the data may be processed by the RDD conversion operation, such as using the map function to apply a custom function, or using the filter to select data that meets a specific condition.
[0113] Step S503, performing a data aggregation operation on the batch processed data to obtain aggregated data; In step S503, actual calculation on the RDD is triggered and the result is returned. For example, calculation is performed through action operations such as count, collect, take, etc. These operations will trigger the conversion operation of the previous chain call and calculate the result in memory.
[0114] Step S504, using Spark's MLlib library to perform classification, regression, and clustering operations on the aggregated data to obtain analysis results; In particular, step S504 can utilize Spark SQL and advanced APIs for more structured and optimized data processing, such as using Spark SQL to execute SQL queries or using DataFrames and Datasets APIs for structured data processing, which provide more advanced abstraction and optimization.
[0115] Specifically, MLlib is a machine learning library for Apache Spark that provides a range of machine learning algorithms and tools for creating and tuning machine learning models on large datasets. MLlib supports many types of machine learning tasks, including classification, regression, clustering, collaborative filtering, dimensionality reduction, etc.
[0116] Here are some key features of MLlib: Scalability: MLlib is designed to run on large datasets, and it scales well to clusters of hundreds of nodes.
[0117] Ease of use: MLlib provides a concise API that makes it easy to implement machine learning tasks.
[0118] Diversity: MLlib includes a variety of algorithms, including linear regression, logistic regression, decision tree, random forest, support vector machine, naive Bayes, K-means clustering, principal component analysis (PCA), etc.
[0119] Pipeline: MLlib supports the Pipeline API, which allows users to build machine learning workflows consisting of multiple stages, each of which can be a feature transformation or a learning algorithm.
[0120] Feature extraction: MLlib provides a wealth of feature extraction tools, including text vectorization, feature hashing, normalization, PCA, etc.
[0121] Model evaluation: MLlib provides model evaluation tools, such as cross-validation and metric statistics, to help users select the best model parameters.
[0122] Model persistence: MLlib supports model persistence, which allows you to save trained models to disk and load them later.
[0123] Ensemble learning: MLlib provides ensemble learning algorithms such as gradient boosted trees (GBM) and stochastic gradient descent (SGD).
[0124] Collaborative filtering: MLlib provides collaborative filtering algorithms for recommender systems, such as model-based methods and memory-based methods.
[0125] Optimization: MLlib's algorithms take advantage of Spark's optimized execution engine to improve computational efficiency.
[0126] Spark SQL is a module of Apache Spark that provides a programming interface for processing structured and semi-structured data. Spark SQL allows users to perform data queries using SQL queries or Apache Hive's query language (HiveQL), and can be easily integrated with DataFrames and Datasets APIs.
[0127] Following are some key features of Spark SQL: Unified data processing interface: Spark SQL provides a unified interface to process various data sources, including HDFS, S3, local file systems, Hive tables, and NoSQL databases.
[0128] Support for multiple data formats: Spark SQL supports multiple data formats, such as CSV, JSON, Parquet, ORC, Avro, and Hive's RCFile.
[0129] Integration of DataFrames and Datasets: Spark SQL can be seamlessly integrated with DataFrames and Datasets APIs, allowing users to execute SQL queries on top of these high-level abstractions.
[0130] Performance optimization: Spark SQL uses the Catalyst optimizer and the Tungsten execution engine to optimize query plans and improve performance.
[0131] Hive compatibility: Spark SQL supports Hive metadata, UDF (user-defined functions), HiveQL syntax, and Hive serde (serialization / deserialization).
[0132] Data exploration: Spark SQL provides data exploration functions, such as DESCRIBE, SHOW TABLES, SHOW DATABASES and other commands.
[0133] Data Transformation: Spark SQL allows users to transform data through SQL queries, such as data projection, filtering, and aggregation using SELECT statements.
[0134] Data aggregation: Users can use SQL statements to perform complex data aggregation operations, such as GROUP BY, ORDER BY, and window functions.
[0135] Temporary Views: Spark SQL allows users to create temporary views, which can be used in SQL queries like tables.
[0136] Extensibility: The design of Spark SQL allows it to be integrated with other Spark modules such as Spark Streaming, MLlib, and GraphX to provide an end-to-end data processing solution.
[0137] In Apache Spark, DataFrame is a distributed object that stores data in the form of a structured dataset. DataFrame is the core abstraction of the Spark SQL module, which provides an efficient and easy-to-use API for processing structured data.
[0138] Following are some key features of DataFrame: Structured: A DataFrame has a well-defined schema, i.e. the structure of the data, including column names and data types.
[0139] Optimized execution: DataFrame operations take advantage of Spark’s optimized execution engine, Catalyst, and the Tungsten physical execution engine, which can significantly improve query performance.
[0140] Ease of use: The DataFrame API provides a high-level abstraction that makes data transformation and analysis tasks easier.
[0141] Multiple language support: DataFrame API is available in multiple programming languages such as Scala, Java, Python, and R.
[0142] Transformation operations: DataFrame supports a variety of transformation operations, such as select, filter, groupBy, orderBy, etc.
[0143] Action operations: DataFrame supports a variety of action operations, such as collect, count, show, saveAsTextFile, etc.
[0144] Interoperability with RDD: DataFrame can be converted to and from RDD, making it possible to leverage RDD’s low-level API for more complex operations when needed.
[0145] Integration of multiple data sources: DataFrame can directly read and write multiple data sources, such as HDFS, S3, Parquet, JSON, CSV, etc.
[0146] Support SQL query: DataFrame supports SQL query and can execute SQL statements through Spark SQL.
[0147] Support for UDF: Users can define their own functions (UDF) to extend the functionality of DataFrame.
[0148] In Apache Spark, the Dataset API is a new programming abstraction that combines the advantages of RDD's strong typing and DataFrame's declarative operations. The Dataset API was introduced in Spark 1.6 and aims to provide a type-safe and easy-to-use programming model while maintaining high performance.
[0149] Key features of the Dataset API include: Strong typing: The Dataset API allows developers to operate on data in a strongly typed manner, which means that type errors can be caught at compile time rather than at runtime.
[0150] Compile-time type checking: Since Dataset is strongly typed, the compiler can check type errors at compile time, thus reducing runtime errors.
[0151] Optimized execution: The Dataset API leverages Spark’s optimized execution engine Catalyst and Tungsten physical execution engine to provide efficient data processing capabilities.
[0152] Transformation operations: Dataset supports the same transformation operations as DataFrame, such as select, filter, groupBy, orderBy, etc.
[0153] Action operations: Dataset also supports action operations, such as collect, count, show, etc.
[0154] Interoperation with RDD: Dataset can be converted to RDD and vice versa through the Dataset as method.
[0155] Support for multiple data sources: Dataset can directly read and write multiple data sources, the same as DataFrame.
[0156] Support SQL query: Dataset supports SQL query and can execute SQL statements through Spark SQL.
[0157] Support UDF: Users can define their own functions (UDF) to extend the functionality of the Dataset.
[0158] Hassle-free migration: Migrating from DataFrame to Dataset is relatively easy, just add type information.
[0159] Among them, Apache Spark's RDD is used in the process of parallel data processing to perform memory computing, including: Step S505, creating an RDD based on cloud storage resources; Wherein, step S505 may be used to aggregate the data, such as summing, averaging, maximum / minimum value, etc. For example, operations such as reduce, aggregate, or groupBy are used to aggregate the data to obtain aggregated data.
[0160] Step S506, performing conversion operation on the RDD; In step S506, a machine learning algorithm may be applied to further analyze the aggregated data, for example, using algorithms in the MLlib library, such as logistic regression, decision tree, K-means, etc., to perform classification, regression or cluster analysis on the data.
[0161] Step S507, executing an action operation to perform memory calculation.
[0162] In step S507, the processing result can be stored back to the cloud storage, or further analyzed and applied. For example, using RDD's saveAsTextFile or other storage methods, the result data is written back to the cloud storage for persistent storage or for use by other systems.
[0163] Specifically, in Apache Spark, saveAsTextFile is an RDD (Resilient Distributed Dataset) and DataFrame action that is used to save data to the file system as a text file. This operation converts each element in the RDD or DataFrame into a string and saves it as a line in the text file.
[0164] Here are some key features of saveAsTextFile: Simplicity: saveAsTextFile is a very simple operation that does not require specifying a specific file format because the output is in text format by default.
[0165] Flexibility: Can be used to save any type of RDD or DataFrame as they will be converted into strings.
[0166] Distributed writing: Data will be written to multiple nodes in the cluster in a distributed manner, improving writing efficiency.
[0167] File system compatibility: Can be saved to a variety of file systems such as local file system, HDFS, Amazon S3, etc.
[0168] Partition control: By specifying an optional partitions parameter, you can control the number of partitions of the output file.
[0169] The partitions parameter is often used in databases and data storage systems to refer to the operation of splitting a data set into multiple parts. The meaning of the partitions parameter may vary slightly in different contexts, but here are some common uses: Database partitioning: In a database, the partitions parameter can specify the number of partitions for a table or index. Partitioning can help improve query performance by distributing data across different physical storage units to reduce the amount of data that needs to be scanned for a single query.
[0170] Hadoop: In the Hadoop Distributed File System, the partitions parameter can specify how many parts the input data is split into when executing a MapReduce job. This is usually used together with the mapred.reduce.tasks parameter, which defines the number of tasks in the Reduce phase.
[0171] Spark: In Apache Spark, the partitions parameter can be used to control the number of partitions of an RDD (Resilient Distributed Dataset) or DataFrame. This affects the parallel processing of data and memory usage.
[0172] Kafka: When using Apache Kafka, the partitions parameter defines the number of partitions in a topic. Partitions allow Kafka to process messages in parallel, increasing throughput.
[0173] Data warehouse: In a data warehouse solution, the partitions parameter may be used to define a partitioning strategy for the data table to more efficiently manage and query data.
[0174] Steps S501 to S507 together constitute a complete data processing and analysis process. From data reading, conversion, aggregation, to machine learning modeling, and then to the storage and application of results, each step utilizes the powerful functions of Apache Spark to improve the efficiency and performance of data processing to form a complete processing system.
[0175] Figure 6 is a flow chart of a method for converting analysis results into a visual chart according to an embodiment of the present invention. In one implementation, Figure 6 As shown, the analysis results are converted into visual charts, including: Step S601, creating an interactive chart based on the Python Plotly library; Among them, step S601 can use the Plotly library to create dynamic and interactive charts so that users can explore data more deeply. In an exemplary implementation, the following method can be used to implement: 1. Install Plotly: First, make sure the Plotly library is installed in the Python environment. You can install it through the pip installplotly command.
[0176] 2. Create charts: Use Plotly's Python API to create charts. Plotly supports a variety of chart types, such as line charts, bar charts, scatter charts, pie charts, etc.
[0177] Specifically, Plotly is a popular open source charting library that supports multiple programming languages, including Python, R, JavaScript, etc. Plotly's Python library allows users to create interactive charts that can be easily integrated into various applications such as web applications, Jupyter notebooks, etc.
[0178] Here are some key features of the Plotly Python library: Interactivity: Plotly charts are interactive, and users can hover to view data points, zoom, pan, etc.
[0179] Multiple chart types: Plotly supports multiple chart types, including line charts, scatter charts, bar charts, bar charts, box plots, heat maps, maps, 3D charts, etc.
[0180] Easy to use: Plotly provides a concise API that makes creating charts easy.
[0181] Customization: Users can customize various aspects of the chart, such as colors, labels, legends, titles, etc.
[0182] Integration: Plotly charts can be easily integrated into web applications and support multiple web frameworks such as Dash, Flask, Django, etc.
[0183] Data support: Plotly supports multiple data formats, including Pandas DataFrame, NumPy array, etc.
[0184] Export: Plotly charts can be exported to multiple formats, such as HTML, JPEG, PNG, SVG, etc.
[0185] Community and Documentation: Plotly has an active community and rich documentation resources to facilitate users to learn and use.
[0186] pip install plotly is a command for Python's package management tool pip, which is used to install the Plotly library. Execute this command in a command line interface (such as a terminal, command prompt, or PowerShell) to download and install Plotly and its dependencies into the Python environment.
[0187] Here are some details on using the command: Make sure pip is installed: Most Python distributions come with pip installed.
[0188] Use a virtual environment: It is recommended to install Plotly in a virtual environment to avoid affecting the global Python environment. You can use venv or conda to create a virtual environment.
[0189] Upgrade Plotly: If you want to upgrade to the latest version of Plotly, you can use the pip install--upgradeplotly command.
[0190] Install a specific version: If you need to install a specific version of Plotly, you can use the pip install plotly==xxx command, where xxx is the version number.
[0191] View installation options: pip install --help You can view pip's installation options, which may help understand how to customize the installation process.
[0192] Use domestic sources: You can use domestic pip mirror sources to speed up downloading, for example, use pip install -ihttps: / / pypi.tuna.tsinghua.edu.cn / simple plotly command.
[0193] Install other dependencies: Plotly may require other dependencies to work properly, such as numpy and pandas. These dependencies are usually installed automatically, but it is also possible to install them manually.
[0194] venv is a standard library of Python that provides the function of creating virtual environments. Virtual environments allow an isolated runtime environment to be created for each Python project, so that different projects can use different versions of libraries without interfering with each other.
[0195] Here are some key points about venv: Creating a virtual environment: Using python-mvenv <env-name>command to create a new virtual environment. <env-name>is the name given to the virtual environment.
[0196] Activate the virtual environment: After you create a virtual environment, you need to activate it before you can start using it. The activation method depends on your operating system: Windows: Run env-name\Scripts\activate.
[0197] Unix or MacOS: Run source env-name / bin / activate.
[0198] Installation package: After activating the virtual environment, you can use the pip command to install packages. These packages will only be installed in the virtual environment and will not affect the global Python environment.
[0199] Exit the virtual environment: Use the deactivate command to exit the currently activated virtual environment.
[0200] Deleting a virtual environment: If you no longer need a virtual environment, you can simply delete the directory containing it.
[0201] Benefits of using a virtual environment: Isolation: Each project can have its own dependencies to avoid version conflicts.
[0202] Safe: You can test and develop safely without having to worry about affecting the system-level Python environment.
[0203] Portable: The project and its dependencies can be easily copied and deployed in different environments.
[0204] Conda is a popular open source package manager and environment manager, which is mainly used to install and manage software packages and create and maintain different versions of software environments. Conda is part of Anaconda, a Python distribution for scientific computing, but it can also be used to manage packages in other languages, such as R, Ruby, Lua, Scala, etc.
[0205] Here are some key features and uses of Conda: Cross-platform: Conda supports Windows, macOS, and Linux.
[0206] Environmental Management: Use conda create --name myenv to create a new environment.
[0207] Use conda activate myenv to activate the environment.
[0208] Use conda deactivate to exit the currently activated environment.
[0209] Package Management: Use conda install <package>The software package can be installed.
[0210] Use conda update <package>Can update installed packages.
[0211] Use conda list to view all packages installed in the current environment.
[0212] Dependency Management: Conda can handle dependencies between packages and ensure that all dependencies are met.
[0213] Channels: Conda allows downloading packages from different sources (called channels). The default channels are Anaconda and Conda-Forge.
[0214] Version control: Conda can manage different versions of software packages and can easily switch to different versions of software packages.
[0215] Cross-language support: Although Conda was originally designed for Python, it also supports packages in other languages.
[0216] Ease of use: Conda's command line interface is simple and intuitive, making it easy to learn and use.
[0217] Integration: Conda can be integrated with popular scientific computing tools such as Jupyter Notebook and Spyder.
[0218] Extensibility: Conda supports custom packages and can install unofficial packages.
[0219] Conda is ideal for developers who need to manage multiple projects and environments, especially when the project depends on specific versions of libraries. By using Conda, developers can avoid "dependency hell" and ensure project reproducibility.
[0220] Step S602, converting the analysis result into a chart data source, and mapping the chart data source to chart elements of the interactive chart; Among them, the data analysis results processed by Spark can be converted into a format that can be used in a chart by adopting step S602. In an exemplary implementation, the following method can be adopted: 1. Data conversion: Convert the RDD or DataFrame in Spark to Pandas DataFrame or other data formats that are easy for Plotly to process.
[0221] 2. Map to chart elements: Map the data to the corresponding elements of the chart according to the data type and chart type that needs to be displayed, such as mapping data columns to the x-axis and y-axis.
[0222] Step S603, adding a time filter to the chart element to view the trend of the chart element in different time periods.
[0223] Among them, step S603 is adopted to allow the user to filter data according to time and view data trends in different time periods. In an exemplary implementation, the following method can be adopted: 1. Time filter: Add a time filter control to the chart interface, allowing users to select a specific time range.
[0224] 2. Dynamic update: Implement a dynamic update mechanism for the chart. When the user selects a different time range, the chart can be redrawn based on the filtered data.
[0225] Through steps S601 to S603, the data processed by Spark is first converted into a Pandas DataFrame, and then a line chart is created using Plotly to display the results of the time series analysis. Plotly charts come with interactive controls, including time filters, so users can directly select the time range on the chart interface to view data trends in different time periods.
[0226] Figure 7 is a flow chart of a method for analyzing the content of a visual chart based on an intelligent text mining module to obtain mining results according to an embodiment of the present invention. In one implementation, Figure 7 As shown, the content of the visualization chart is analyzed based on the intelligent text mining module to obtain mining results, including: Step S701, when the input data is image data and text data, calculating the correlation between the image data and the text data; In step S701, it is determined whether there is a certain correlation between the image and the text data, for example, whether the image content matches the emotion or theme described in the text. In an exemplary implementation, the following method can be used: 1. Feature extraction: Extract features (such as color, shape, texture, etc.) from image data, and extract features (such as keywords, topics, sentiment tags, etc.) from text data.
[0227] 2. Association analysis: Use statistical methods or machine learning algorithms (such as association rule mining, collaborative filtering, deep learning models, etc.) to analyze the association between two types of data.
[0228] Step S702, obtaining a first emotional tendency of the content of the image data based on the content of the image data; In step S702, the image content is analyzed and its main emotional tendency is determined, such as positive, negative or neutral. In an exemplary implementation, the following method can be used: 1. Image processing: Use image processing techniques (such as computer vision) to analyze image content.
[0229] 2. Sentiment analysis model: Apply pre-trained sentiment analysis models (such as CNN-based deep learning models) to identify emotional tendencies in images.
[0230] Step S703, acquiring the content of the text data to obtain a second emotional tendency of the content of the text data; In step S703, the text data is analyzed and its sentiment tendency is determined. In an exemplary implementation, the following method can be used: 1. Natural language processing: Use NLP technology for text analysis, including word segmentation, part-of-speech tagging, sentiment word recognition, etc.
[0231] 2. Sentiment analysis: Apply sentiment analysis algorithms (such as machine learning-based classifiers or deep learning models) to determine the emotional tendency of the text.
[0232] Step S704: combining the first sentiment tendency and the second sentiment tendency to obtain a mining result.
[0233] In step S704, the sentiment tendency of the image and text data is combined to obtain a more comprehensive mining result. In an exemplary implementation, the following method can be used: 1. Data fusion: The sentiment tendency data obtained in steps S702 and S703 are fused, and simple rules or more complex algorithms (such as decision trees, random forests, etc.) can be used.
[0234] 2. Comprehensive analysis: Based on the integrated sentiment tendency data, comprehensive analysis is performed to reveal patterns, trends or anomalies in the data.
[0235] Through steps S701 to S704, the following technical effects can be achieved: 1. Multimodal data analysis: By analyzing image and text data simultaneously, more comprehensive insights can be provided because these two types of data can complement each other and provide richer information.
[0236] 2. Sentiment tendency identification: Steps S702 and S703 allow the system to evaluate the emotional tendencies of image and text data, respectively, involving the use of computer vision and natural language processing techniques to identify and quantify emotional content in the data.
[0237] Sentiment analysis of image data can involve identifying specific objects, scenes, or colors in an image and associating them with positive or negative emotional states.
[0238] Sentiment analysis of text data typically involves identifying and evaluating sentiment words, phrases, or themes in the text and determining the overall emotional color.
[0239] 3. Emotional relevance analysis: Step S701 aims to explore the emotional relevance between image and text data. This helps to understand whether there is consistency or difference in the emotions expressed in different types of data, and the possible mutual influence between them.
[0240] 4. Comprehensive mining results: Step S704 provides a deeper analysis by combining the sentiment tendencies of the two data types. This comprehensive analysis can help reveal complex patterns in the data, for example, determining whether the image content and text description express similar sentiments in a particular case, or whether they provide conflicting sentiment signals.
[0241] 5. Enhanced decision support: These steps provide analytical results that can support decision makers in understanding the sentiment dynamics in customer feedback, social media trends, or other multimodal data sets.
[0242] 6. Application of sentiment analysis: These steps can be applied in a variety of scenarios, such as brand monitoring, market research, social media analysis, customer service optimization, etc., to help organizations better understand the public's emotional reactions and attitudes.
[0243] 7. Data-driven insights: Through these steps, organizations can gain data-driven insights that can be used to improve product design, adjust marketing strategies, optimize customer experience, etc.
[0244] Taken together, these steps provide a powerful approach to understand and interpret human emotional responses by combining sentiment analysis of image and text data, leading to more effective data mining and decision support in multiple domains.
[0245] Through the description of the above implementation methods, those skilled in the art can clearly understand that the method according to the above embodiment can be implemented by adding the necessary general hardware platform with the help of software, and of course it can also be implemented by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present invention, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), and includes a number of instructions for a terminal device (which can be a mobile phone, computer, server, or network device, etc.) to execute the methods of various embodiments of the present invention.
[0246] In this embodiment, a data mining device based on a data model is also provided, which is used to implement the above-mentioned embodiments and preferred implementation modes, and the descriptions that have been made will not be repeated. As used below, the term "module" can implement a combination of software and / or hardware of a predetermined function. Although the device described in the following embodiments is preferably implemented in software, the implementation of hardware, or a combination of software and hardware is also possible and conceivable.
[0247] Figure 8 is a structural schematic diagram of a data mining device based on a data model according to an embodiment of the present invention. Figure 8 As shown, the device comprises: A multimodal data classification module 81, used to identify input data, and determine whether the input data is image data and / or text data based on Apache Kafka and the multimodal data classification component; and Normalize the image data based on GPU, segment the text data based on GPT, and vectorize it based on TF-IDF or Word2Vec to obtain preprocessed data; A big data processing platform module 82 is used to analyze the pre-processed data to obtain analysis results; and Convert the analysis results into visual charts; The intelligent text mining module 83 is used to analyze the content of the visual chart to obtain mining results.
[0248] In one embodiment, the device is also used to: Create corresponding topics based on input data through Apache Kafka; Routing different information of input data to different topics so that the multimodal data classification component can obtain data from different topics to obtain classification results; The classification result is sent to the classified-data topic, where the classification result includes image data and text data, and the topic includes the classified-data topic.
[0249] In one embodiment, the device is also used to: Get input data from different sources and in different formats based on input adapters; Standardize the input data based on the formatting engine to obtain data in a unified format; Send data in a uniform format to Apache Kafka based on output adapters.
[0250] In one embodiment, the device is also used to: Use Apache Spark to perform parallel data processing on preprocessed data to obtain analytical results; Apache Spark's RDD is used in parallel data processing for in-memory computing.
[0251] In one embodiment, the device is also used to: Map, filter, and aggregate the preprocessed data to obtain transformed data; Use Spark SQL, DataFrames, and Datasets APIs to batch process the transformed data to obtain batch processed data; Performing data aggregation operation on batch processing data to obtain aggregated data; Use Spark's MLlib library to perform classification, regression, and clustering operations on aggregated data to obtain analysis results; Create RDD based on cloud storage resources; Perform transformation operations on RDD; Executes actions to perform in-memory computations.
[0252] In one embodiment, the device is also used to: Create interactive charts based on Python's Plotly library; converting analysis results into chart data sources, and mapping the chart data sources to chart elements of the interactive chart; Add time filters to chart elements to see trends for chart elements over different time periods.
[0253] In one embodiment, the device is also used to: When the input data is image data and text data, calculating the correlation between the image data and the text data; Based on the content of the image data, obtaining a first emotional tendency of the content of the image data; Acquire the content of the text data to obtain a second emotional tendency of the content of the text data; The first sentiment tendency and the second sentiment tendency are combined to obtain a mining result.
[0254] It should be noted that the above modules can be implemented by software or hardware. For the latter, it can be implemented in the following ways, but not limited to: the above modules are all located in the same processor; or the above modules are located in different processors in any combination.
[0255] An embodiment of the present invention further provides a computer-readable storage medium, in which a computer program is stored, wherein the computer program is configured to execute the steps of any of the above method embodiments when running.
[0256] In an exemplary embodiment, the computer-readable storage medium may include, but is not limited to, various media that can store computer programs, such as a USB flash drive, a read-only memory (ROM), a random access memory (RAM), a mobile hard disk, a magnetic disk or an optical disk.
[0257] An embodiment of the present invention further provides a computer device, including a memory and a processor, wherein a computer program is stored in the memory, and the processor is configured to run the computer program to execute the steps in any one of the above method embodiments.
[0258] In an exemplary embodiment, the computer device may further include a transmission device and an input / output device, wherein the transmission device is connected to the processor, and the input / output device is connected to the processor.
[0259] For specific examples in this embodiment, reference may be made to the examples described in the above embodiments and exemplary implementation modes, and this embodiment will not be described in detail herein.
[0260] Obviously, those skilled in the art should understand that the above modules or steps of the present invention can be implemented by a general computing device, they can be concentrated on a single computing device, or distributed on a network composed of multiple computing devices, they can be implemented by a program code executable by a computing device, so that they can be stored in a storage device and executed by the computing device, and in some cases, the steps shown or described can be executed in a different order than here, or they can be made into individual integrated circuit modules, or multiple modules or steps therein can be made into a single integrated circuit module for implementation. Thus, the present invention is not limited to any specific combination of hardware and software.
[0261] The above are only preferred embodiments of the present invention and are not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the principles of the present invention shall be included in the protection scope of the present invention.< / package> < / package>
Claims
1. A data mining method based on data model, It is characterized in that include: Identify input data, and determine whether the input data is image data and / or text data based on Apache Kafka and a multimodal data classification component; The image data is normalized based on GPU, and the text data is segmented based on GPT and vectorized based on TF-IDF or Word2Vec to obtain preprocessed data; Analyze the pre-processed data based on the big data processing platform module to obtain analysis results; Convert the analysis results into visual charts; The content of the visual chart is analyzed based on an intelligent text mining module to obtain mining results.
2. The method according to claim 1, It is characterized in that Identifying input data, and determining that the input data is image data and / or text data based on Apache Kafka and a multimodal data classification component, includes: Creating a corresponding topic based on the input data through the Apache Kafka; Routing different information of the input data to different topics, so that the multimodal data classification component obtains data from the different topics to obtain classification results; The classification result is sent to a classified-data topic, wherein the classification result includes the image data and the text data, and the topic includes the classified-data topic.
3. The method according to claim 2, It is characterized in that Before creating a topic based on the input data through Apache Kafka, include: Acquire the input data from different sources and in different formats based on an input adapter; Performing standardization processing on the input data based on a formatting engine to obtain data in a unified format; The data in the unified format is sent to the Apache Kafka based on an output adapter.
4. The method according to claim 1, It is characterized in that The pre-processed data is analyzed based on the big data processing platform module to obtain analysis results, including: Using Apache Spark to perform parallel data processing on the preprocessed data to obtain analysis results; The RDD of Apache Spark is used in the process of parallel data processing to perform memory computing.
5. The method according to claim 4, It is characterized in that Using Apache Spark to perform parallel data processing on the preprocessed data to obtain analysis results, including: Mapping, filtering, and aggregating the preprocessed data to obtain transformed data; Using Spark SQL, DataFrames, and Datasets APIs to batch process the transformed data to obtain batch processed data; Performing a data aggregation operation on the batch processed data to obtain aggregated data; Using Spark's MLlib library to perform classification, regression, and clustering operations on the aggregated data to obtain analysis results; The RDD of Apache Spark is used in the process of parallel data processing to perform memory computing, including: Create RDD based on cloud storage resources; Performing a conversion operation on the RDD; Executes actions to perform in-memory computations.
6. The method according to claim 1, It is characterized in that The analysis results are converted into visual charts, including: Create interactive charts based on Python's Plotly library; converting the analysis results into a chart data source, and mapping the chart data source to a chart element of the interactive chart; Add a time filter to the chart element to view the trend of the chart element in different time periods.
7. The method according to claim 1, It is characterized in that The content of the visualization chart is analyzed based on the intelligent text mining module to obtain mining results, including: In the case where the input data is image data and text data, calculating the correlation between the image data and the text data; Based on the content of the image data, obtaining a first emotional tendency of the content of the image data; Acquiring the content of the text data to obtain a second emotional tendency of the content of the text data; The first sentiment tendency and the second sentiment tendency are combined to obtain the mining result.
8. A data mining device based on a data model, It is characterized in that include: A multimodal data classification module, configured to identify input data, and determine whether the input data is image data and / or text data based on Apache Kafka and a multimodal data classification component; and The image data is normalized based on GPU, and the text data is segmented based on GPT and vectorized based on TF-IDF or Word2Vec to obtain preprocessed data; A big data processing platform module is used to analyze the pre-processed data to obtain analysis results; as well as, Convert the analysis results into visual charts; The intelligent text mining module is used to analyze the content of the visual chart to obtain mining results.
9. A computer-readable storage medium, It is characterized in that The computer-readable storage medium stores a computer program, wherein the computer program implements the steps of the method described in any one of claims 1 to 7 when executed by a processor.
10. A computer device comprising a memory, a processor and a computer program stored on the memory and executable on the processor, It is characterized in that When the processor executes the computer program, the steps of the method described in any one of claims 1 to 7 are implemented.
Citation Information
Patent Citations
Method for effectively exerting data value
CN110162556A
Emergency sentiment analysis method fusing theme and multiple modes
CN115392232A
Social media user behavior trajectory modeling and analyzing method
CN115687430A
Open source intelligence-oriented multi-modal data sentiment analysis method
CN116561639A
Intelligent data analysis method and system
CN117314006A
Cited By
Data management method, system, equipment and medium for integrated data medium
CN119474034A
Digital data conversion method and system based on artificial intelligence
CN120067350A
