AI big data real-time processing and analysis method

Abstract

The invention relates to the technical field of AI big data processing, and discloses an AI big data real-time processing and analysis method, which comprises the following steps: step 1, data acquisition and preprocessing; step 2, feature extraction and intelligent adaptation; step 3, feature association and data screening; step 4, data analysis; 5, system architecture and technical support; and step 6, edge calculation support. For different types of data, such as images, texts, sensor data and the like, the edge device firstly performs standardization processing, converts the data into a general format, facilitates subsequent unified processing, converts the images into a specific resolution or color format, and converts the texts into a unified coding format; and the sensor data is converted into a specific data unit or range, so that the beneficial effect of uniformly processing different data types to improve the data processing efficiency is achieved.

Description

Technical Field

[0001] The present invention relates to the field of AI big data processing technology, and specifically to a method for real-time processing and analysis of AI big data. Background Art

[0002] As an important product of the development of modern information technology, AI has brought many conveniences to people's lives. The evolution of business through AI big data services is an important assistance in the current social business process, especially in mining key data from massive text and image data, thereby providing important assistance for people to improve efficiency in work and life. The prior art publication number is CN 118245680 B, which is a real-time processing and analysis method of AI big data, and particularly relates to the field of AI big data processing technology. The method includes using a data acquisition unit to acquire user demand data, and using a feature extraction unit to extract key features of the user demand data; using a feature association unit to associate the user demand data after the key features are extracted with an AI big database to extract associated data of the user demand data from the AI ​​big database; using a feature recognition unit to perform feature recognition on the associated data and output the recognition result; using a data screening unit to screen the recognition result, and sending the screened associated data to the user end through a data analysis unit, and collecting user feedback data for analysis; using an adjustment unit to adjust the processing process based on the analysis result of the data analysis unit.

[0003] Although the aforementioned technology solves the problem that the existing technology cannot accurately control the process of processing and analyzing AI big data, feature extraction in the aforementioned technology is one of the key steps in data processing, but different data types require different feature extraction methods. The aforementioned technical solution does not mention specific feature extraction technology, nor does it explain whether it can efficiently process large-scale data sets, which actually results in the big data processing and analysis process taking longer. Summary of the invention

[0004] Technical issues solved: In view of the shortcomings of the existing technology, the present invention provides an AI big data real-time processing and analysis method, which has the advantages of unified processing of different data types to improve data processing efficiency, shorten the big data processing and analysis time, and solves the problems of the above-mentioned technology.

[0005] Technical solution: To achieve the above object, the present invention provides the following technical solution: a method for real-time processing and analysis of AI big data, comprising the following steps: Step 1: Data acquisition and preprocessing: Real-time data acquisition from multiple data sources: sensors, user behavior records, log files, and social media. Support the collection of structured, semi-structured, and unstructured data. Improve data quality through automated cleaning tools: outlier detection. Convert data formats from different sources: JSON, XML, text, and images into a standard format. Compress large-scale data to reduce storage and transmission costs. Step 2: Feature extraction and intelligent adaptation: Select the optimal feature extraction algorithm for different data types: text, image, audio, video and time series data, including: Text data: Use the TF-IDF deep learning model to extract semantic features; Image data: Use convolutional neural network (CNN) to extract visual features; Audio data: Use Mel-frequency cepstral coefficients to extract audio features; Time series data: Use LSTM to extract time series features; Adaptive feature extraction: Through meta-learning technology, it automatically adapts to different data types and distributions to improve the generalization ability of feature extraction; Feature fusion: Fuse multimodal features: text, image, and time series features to generate a unified feature representation; Step 3: Feature association and data screening: Associate the extracted multimodal features with the data in the AI ​​big database, support cross-modal association analysis: association between text and images, audio and video, use the distributed computing framework Spar to accelerate the association calculation of large-scale data, and use the machine learning model: random forest to intelligently screen the associated data based on user demand data and association analysis results; Step 4: Data analysis: Use the lightweight inference engine TensorRT to perform real-time analysis on the filtered data, support multi-dimensional analysis of time, space, and behavior, generate visual analysis results, and use users' real-time feedback data: click behavior, ratings, and comments, as input to optimize the analysis results. Based on user feedback and analysis results, dynamically adjust the parameters and screening strategies of the feature extraction algorithm to improve the system's adaptability. Through online learning technology, continuously update the feature extraction and analysis models to improve the intelligence level of the system. Step 5: System architecture and technical support: Use the distributed computing framework Spark and GPU cluster to accelerate the processing and analysis of large-scale data, and use the distributed storage system HDFS and cache mechanism Redis to support efficient data storage and access; Step 6: Edge computing support: Perform preliminary feature extraction and data screening on edge devices to reduce data transmission delays, and upload tasks that cannot be processed by edge devices to the cloud for further processing.

[0006] Preferably, in step 1, outlier detection is performed based on the Z-score method, and the detection calculation is: ; in: is a data point, is the mean of the data, is the standard deviation of the data, if , then the data point is considered to be an outlier; The data compression rate calculation in step 1 is: ; The original data size is the data size before compression, and the compressed data size is the data size after compression.

[0007] Preferably, the TF-IDF deep learning model in step 2 is: ; in: is the word frequency, indicating the word In the documentation The frequency of occurrence in The specific formula is: Word frequency ; Inverse Document Frequency ; in, is the total number of documents in the corpus.

[0008] Preferably, in step 2, the convolutional neural network extracts visual features from the image through a convolution operation, which is calculated as: ; in: is the pixel matrix of the input image, with a size of , is the height, is the width, is the number of channels; is the bias term; In step 2, the Mel frequency cepstral coefficient is used to extract the features of the audio data and is calculated as: ; in: It is The logarithmic energy of the mel-bands; is the timeframe index.

[0009] Preferably, in step 2, a long short-term memory network (LSTM) is used to extract features of time series data, and the calculation is: Forget Gate ; in: is the output of the forget gate, a value between 0 and 1; is the weight matrix of the forget gate, which is used to learn the relationship between the input and the previous state; is the bias term of the forget gate, which is used to adjust the output of the weight matrix; Input Gate ; in: is the output of the input gate, a value between 0 and 1; is the weight matrix of the input gate, which is used to learn the relationship between the input and the previous state; is the bias term of the input gate, which is used to adjust the output of the weight matrix; Cell status update ; ; in: is a candidate state, representing the potential value of new information; is the weight matrix of the candidate state, which is used to learn the relationship between the input and the previous state; is the bias term of the candidate state, which is used to adjust the output of the weight matrix; Represents the cell state information of the previous time step that is retained; Output Gate ; ; in: The activation function is the Sigmoid function, is the bias term, is the output of the output gate, which is a value between 0 and 1; is the weight matrix of the output gate, which is used to learn the relationship between the input and the previous state; is the bias term of the output gate, which is used to adjust the output of the weight matrix; Is the cell state The activation value of , compresses the value of the cell state to the (-1, 1) interval.

[0010] Preferably, the cross-modal association analysis in step 3 is used to find the association between text and image, audio and video. There are data of modality A and modality B, and the cosine similarity is used to calculate the similarity or correlation between them. For two vectors and , the cosine similarity is defined as: ; in: and They are text data and image data respectively; is a vector and The cosine of the angle between them; Use Euclidean distance to measure the distance between feature vectors. The smaller the distance, the more similar it is: ; in: and is a vector and No. elements; There are two vectors in The square of the difference in the dimension ensures that the distance is non-negative and amplifies the difference; It is to add the squares of the differences in all dimensions to get a total square difference; It is the square root of the square difference, that is, the Euclidean distance, which measures the straight-line distance between two vectors in Euclidean space. The smaller it is, the closer the two vectors are.

[0011] Preferably, the time dimension analysis calculation in step 4 is: calculating the mean and variance of the data in the time dimension; ; in: For time period The mean of the internal data; In step 4, the spatial dimension analysis calculates the spatial distribution of the data and clusters the spatial data using the K-means algorithm: ; in: is the cluster center set; For the Cluster centers.

[0012] Preferably, the distributed computing framework Spark accelerated computing in step 5 is: Spark converts the dataset Divide into Partitions: ; The calculation results for each partition Through parallel calculation, we get: ; Final global result It is the combination of the results of each partition: .

[0013] Preferably, the step 5 also includes validity calculation, calculating the processing time of each partition and speedup : ; in: This is the processing time for a single node.

[0014] Preferably, the edge device preliminary feature extraction calculation is as follows: the edge device performs preliminary processing on the data, extracts key features, and reduces the amount of data: Assume the original data is ,in is the number of samples, is the characteristic number; Feature extraction: Extract using feature extraction algorithm key features, and obtain a new feature matrix ; in, is the feature extraction function; Data transfer volume calculation: Raw data transfer volume ,in The data size for each feature: bytes, Data transfer after extraction ; Data transmission reduction ratio: .

[0015] Compared with the prior art, the present invention provides a method for real-time processing and analysis of AI big data, which has the following beneficial effects: 1. In the present invention, for different types of data, such as images, texts, sensor data, etc., the edge device will first perform standardization processing and convert the data into a universal format to facilitate subsequent unified processing. The image is converted into a specific resolution or color format, the text is converted into a unified encoding format, and the sensor data is converted into a specific data unit or range. Next, the edge device will use the corresponding algorithm to extract key features from the standardized data. The features can represent the main information of the original data, thereby reducing the amount of data that needs to be processed. For image data, features such as edges, corners or textures are extracted; for text data, features such as keywords or word frequencies are extracted. After extracting the features, the edge device will filter the data according to preset rules or models. Only data that meets specific conditions or contains important information will be uploaded to the cloud for further processing, which greatly reduces unnecessary data transmission and improves the overall processing efficiency, achieving the beneficial effect of unified processing of different data types and improving data processing efficiency.

[0016] 2. The present invention converts the data into a unified format by standardizing different types of data on edge devices, which is convenient for subsequent unified processing. This step can significantly reduce the complexity and redundancy of the data, perform feature extraction on the edge device, extract the key features of the data, and these features can represent the main information of the original data, greatly reducing the amount of data that needs to be transmitted, and screen the feature data through preset rules or models, and only upload data containing key information to the cloud, further reducing the amount of data that needs to be transmitted. Since the amount of data transmitted is greatly reduced, the network bandwidth occupancy is also reduced accordingly, thereby shortening the data transmission time. Reducing the amount of data transmission can also reduce network latency, so that data can reach the cloud faster. Preliminary data preprocessing, such as standardization, feature extraction and screening, is performed on the edge device to reduce the complexity of the data. Since the amount of data received by the cloud is greatly reduced, the computing burden of the cloud is also reduced, and the cloud can process the remaining data faster, improving the processing efficiency and achieving the beneficial effect of shortening the processing and analysis time of big data. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 It is a schematic diagram of the process of the present invention; DETAILED DESCRIPTION

[0018] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0019] See also Figure 1, an AI big data real-time processing and analysis method, comprising the following steps: Step 1: Data acquisition and preprocessing: Real-time data acquisition from multiple data sources: sensors, user behavior records, log files, and social media. Support the collection of structured, semi-structured, and unstructured data. Improve data quality through automated cleaning tools: outlier detection. Convert data formats from different sources: JSON, XML, text, and images into a standard format. Compress large-scale data to reduce storage and transmission costs. Step 2: Feature extraction and intelligent adaptation: Select the optimal feature extraction algorithm for different data types: text, image, audio, video and time series data, including: Text data: Use the TF-IDF deep learning model to extract semantic features; Image data: Use convolutional neural network (CNN) to extract visual features; Audio data: Use Mel-frequency cepstral coefficients to extract audio features; Time series data: Use LSTM to extract time series features; Adaptive feature extraction: Through meta-learning technology, it automatically adapts to different data types and distributions to improve the generalization ability of feature extraction; Feature fusion: Fuse multimodal features: text, image, and time series features to generate a unified feature representation; Step 3: Feature association and data screening: Associate the extracted multimodal features with the data in the AI ​​big database, support cross-modal association analysis: association between text and images, audio and video, use the distributed computing framework Spar to accelerate the association calculation of large-scale data, and use the machine learning model: random forest to intelligently screen the associated data based on user demand data and association analysis results; Step 4: Data analysis: Use the lightweight inference engine TensorRT to perform real-time analysis on the filtered data, support multi-dimensional analysis of time, space, and behavior, generate visual analysis results, and use users' real-time feedback data: click behavior, ratings, and comments, as input to optimize the analysis results. Based on user feedback and analysis results, dynamically adjust the parameters and screening strategies of the feature extraction algorithm to improve the system's adaptability. Through online learning technology, continuously update the feature extraction and analysis models to improve the intelligence level of the system. Step 5: System architecture and technical support: Use the distributed computing framework Spark and GPU cluster to accelerate the processing and analysis of large-scale data, and use the distributed storage system HDFS and cache mechanism Redis to support efficient data storage and access; Step 6: Edge computing support: Perform preliminary feature extraction and data screening on edge devices to reduce data transmission delays, and upload tasks that cannot be processed by edge devices to the cloud for further processing.

[0020] For different types of data, such as images, text, sensor data, etc., edge devices will first perform standardization, which means converting the data into a common format to facilitate subsequent unified processing. For example, images are converted to a specific resolution or color format, text is converted to a unified encoding format, and sensor data is converted to a specific data unit or range.

[0021] Next, the edge device uses the corresponding algorithm to extract key features from the standardized data. These features can represent the main information of the original data, thereby reducing the amount of data that needs to be processed. For example, for image data, features such as edges, corners or textures are extracted; for text data, features such as keywords or word frequency are extracted.

[0022] After extracting features, edge devices will filter data according to preset rules or models, which means that only data that meets specific conditions or contains important information will be uploaded to the cloud for further processing. This kind of screening greatly reduces unnecessary data transmission and improves overall processing efficiency.

[0023] By preprocessing and filtering data on edge devices, the amount of data that needs to be transmitted to the cloud can be significantly reduced, which not only reduces network bandwidth usage but also reduces data transmission latency.

[0024] Since the edge devices have already performed preliminary processing and screening on the data, the amount of data received by the cloud is greatly reduced, which reduces the computing and storage burden on the cloud and enables it to process the remaining important data more efficiently.

[0025] Through edge computing, the efficiency of the entire data processing system has been significantly improved. The rapid preprocessing and screening of edge devices reduces the delay of data transmission, and the processing burden of the cloud is also reduced, allowing the entire system to respond and process data faster, improving overall performance.

[0026] Edge computing can achieve a unified processing flow for different types of data, such as images, text, and sensor data. Through steps such as standardization, feature extraction, and data screening, these data are converted into a universal, easy-to-process format, which not only simplifies the complexity of data processing, but also improves processing efficiency and accuracy.

[0027] Edge computing achieves unified processing of different data types and significantly improves data processing efficiency by performing preliminary data preprocessing and screening on the device side. This architecture not only reduces the amount of data transmission and the cloud processing burden, but also improves the response speed and processing capacity of the entire system. In practical applications, this method has significant advantages in processing large amounts of diverse data.

[0028] By standardizing different types of data on edge devices and converting the data into a unified format for subsequent unified processing, this step can significantly reduce data complexity and redundancy.

[0029] Feature extraction is performed on edge devices to extract the key features of the data. These features can represent the main information of the original data, greatly reducing the amount of data that needs to be transmitted.

[0030] By filtering feature data through preset rules or models, only data containing key information is uploaded to the cloud, which further reduces the amount of data that needs to be transmitted.

[0031] Reduce data transmission time: Since the amount of data transmitted is greatly reduced, the network bandwidth occupancy is also reduced accordingly, thereby shortening the data transmission time.

[0032] Lower network latency: Reducing the amount of data transferred also reduces network latency, allowing data to reach the cloud faster.

[0033] Perform preliminary data preprocessing on edge devices, such as standardization, feature extraction, and filtering, to reduce the complexity of the data, making the data received in the cloud more streamlined and organized.

[0034] Since the amount of data received by the cloud is greatly reduced, the computing burden on the cloud is also reduced, allowing the cloud to process the remaining data more quickly and improve processing efficiency.

[0035] Reduced cloud processing time: The amount of data processed in the cloud is reduced, and computing resources are used more efficiently, thereby reducing processing time.

[0036] Improved response speed: The cloud can respond to new data processing requests more quickly, improving the overall response speed of the system.

[0037] The initial processing and screening steps of edge devices are completed quickly locally without relying on cloud resources, which not only reduces dependence on the network but also improves the real-time nature of data processing.

[0038] Multiple edge devices perform data preprocessing and screening simultaneously, achieving parallel processing, further improving the efficiency of the overall system and shortening the time for big data processing.

[0039] Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and alterations may be made to the embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.

Claims

1. A method for real-time processing and analysis of AI big data, characterized in that: The following steps are involved: Step 1: Data acquisition and preprocessing: Real-time data acquisition from multiple data sources: sensors, user behavior records, log files, and social media. Support the collection of structured, semi-structured, and unstructured data. Improve data quality through automated cleaning tools: outlier detection. Convert data formats from different sources: JSON, XML, text, and images into a standard format. Compress large-scale data to reduce storage and transmission costs. Step 2: Feature extraction and intelligent adaptation: Select the optimal feature extraction algorithm for different data types: text, image, audio, video and time series data, including: Text data: Use the TF-IDF deep learning model to extract semantic features; Image data: Use convolutional neural network (CNN) to extract visual features; Audio data: Use Mel-frequency cepstral coefficients to extract audio features; Time series data: Use LSTM to extract time series features; Adaptive feature extraction: Through meta-learning technology, it automatically adapts to different data types and distributions to improve the generalization ability of feature extraction; Feature fusion: Fuse multimodal features: text, image, and time series features to generate a unified feature representation; Step 3: Feature association and data screening: Associate the extracted multimodal features with the data in the AI ​​big database, support cross-modal association analysis: association between text and images, audio and video, use the distributed computing framework Spar to accelerate the association calculation of large-scale data, and use the machine learning model: random forest to intelligently screen the associated data based on user demand data and association analysis results; Step 4: Data analysis: Use the lightweight inference engine TensorRT to perform real-time analysis on the filtered data, support multi-dimensional analysis of time, space, and behavior, generate visual analysis results, and use users' real-time feedback data: click behavior, ratings, and comments, as input to optimize the analysis results. Based on user feedback and analysis results, dynamically adjust the parameters and screening strategies of the feature extraction algorithm to improve the system's adaptability. Through online learning technology, continuously update the feature extraction and analysis models to improve the intelligence level of the system. Step 5: System architecture and technical support: Use the distributed computing framework Spark and GPU cluster to accelerate the processing and analysis of large-scale data, and use the distributed storage system HDFS and cache mechanism Redis to support efficient data storage and access; Step 6: Edge computing support: Perform preliminary feature extraction and data screening on edge devices to reduce data transmission delays, and upload tasks that cannot be processed by edge devices to the cloud for further processing.

2. The AI ​​big data real-time processing and analysis method according to claim 1, characterized in that: In the step 1, outlier detection is performed based on the Z-score method, and the detection calculation is: ； in: is a data point, is the mean of the data, is the standard deviation of the data, if , then the data point is considered to be an outlier; The data compression rate calculation in step 1 is: ； The original data size is the data size before compression, and the compressed data size is the data size after compression.

3. The AI ​​big data real-time processing and analysis method according to claim 2 is characterized by: The TF-IDF deep learning model in step 2 is: ； in: is the word frequency, indicating the word In the documentation The frequency of occurrence in The specific formula is: Word frequency ; Inverse Document Frequency ; in, is the total number of documents in the corpus.

4. The AI ​​big data real-time processing and analysis method according to claim 1, characterized in that: In step 2, the convolutional neural network extracts visual features from the image through convolution operation, which is calculated as: ； in: is the pixel matrix of the input image, with a size of , is the height, is the width, is the number of channels; is the bias term; In step 2, the Mel frequency cepstral coefficient is used to extract the features of the audio data and is calculated as: ； in: It is The logarithmic energy of the mel-bands; is the index of the timeframe.

5. The AI ​​big data real-time processing and analysis method according to claim 1, characterized in that: In step 2, the long short-term memory network (LSTM) is used to extract the features of time series data, which is calculated as: Forget Gate ； in: is the output of the forget gate, a value between 0 and 1; is the weight matrix of the forget gate, which is used to learn the relationship between the input and the previous state; is the bias term of the forget gate, which is used to adjust the output of the weight matrix; Input Gate ； in: is the output of the input gate, a value between 0 and 1; is the weight matrix of the input gate, which is used to learn the relationship between the input and the previous state; is the bias term of the input gate, which is used to adjust the output of the weight matrix; Cell status update ； ； in: is a candidate state, representing the potential value of new information; is the weight matrix of the candidate state, which is used to learn the relationship between the input and the previous state; is the bias term of the candidate state, which is used to adjust the output of the weight matrix; Represents the cell state information of the previous time step that is retained; Output Gate ； ； in: The activation function is the Sigmoid function, is the bias term, is the output of the output gate, which is a value between 0 and 1; is the weight matrix of the output gate, which is used to learn the relationship between the input and the previous state; is the bias term of the output gate, which is used to adjust the output of the weight matrix; Is the cell state The activation value of , compresses the value of the cell state to the (-1, 1) interval.

6. The AI ​​big data real-time processing and analysis method according to claim 1, characterized in that: The cross-modal association analysis in step 3 is used to find the association between text and image, audio and video. Suppose there are data of modality A and modality B, and the cosine similarity is used to calculate the similarity or correlation between them. For two vectors and , the cosine similarity is defined as: ； in: and They are text data and image data respectively; is a vector and The cosine of the angle between them; Use Euclidean distance to measure the distance between feature vectors. The smaller the distance, the more similar it is: ； in: and is a vector and No. elements; There are two vectors in The square of the difference in the dimension ensures that the distance is non-negative and amplifies the difference; It is to add the squares of the differences in all dimensions to get a total square difference; It is the square root of the square difference, that is, the Euclidean distance, which measures the straight-line distance between two vectors in Euclidean space. The smaller it is, the closer the two vectors are.

7. The AI ​​big data real-time processing and analysis method according to claim 6, characterized in that: The time dimension analysis calculation in step 4 is as follows: calculating the mean and variance of the data in the time dimension; ; in: For time period The mean of the internal data; In step 4, the spatial dimension analysis calculates the spatial distribution of the data and clusters the spatial data using the K-means algorithm: ； in: is the cluster center set; For the Cluster centers.

8. The AI ​​big data real-time processing and analysis method according to claim 1, characterized in that: The distributed computing framework Spark accelerates the computing in step 5 as follows: Spark converts the dataset Divide into Partitions: ； The calculation results for each partition Through parallel calculation, we get: ； Final global result It is the combination of the results of each partition: 。 9. The AI ​​big data real-time processing and analysis method according to claim 8, characterized in that: The step 5 also includes validity calculation, which calculates the processing time of each partition. and speedup : ； in: This is the processing time for a single node.

10. The AI ​​big data real-time processing and analysis method according to claim 9, characterized in that: The edge device preliminary feature extraction calculation is as follows: the edge device performs preliminary processing on the data, extracts key features, and reduces the amount of data: Assume the original data is ,in is the number of samples, is the characteristic number; Feature extraction: Extract using feature extraction algorithm key features, and obtain a new feature matrix ; in, is the feature extraction function; Data transfer volume calculation: Raw data transfer volume ,in The data size for each feature: bytes, Data transfer after extraction ; Data transmission reduction ratio: .

Images