Data perception automatic distribution method based on deep learning

Abstract

The invention discloses a data perception automatic distribution method based on deep learning, and relates to the technical field of data perception distribution. The method comprises the following steps: 1, formulating a data collection model for collecting data from different sources, formulating a data preprocessing model after data collection is completed, preprocessing the collected data, formulating a feature extraction model after preprocessing is completed, and performing feature extraction on the preprocessed data; 2, building a demand analysis model to analyze the demand of the user, and carrying out identification processing on the analyzed data; and step 3, making a distribution decision model in combination with the data features and user demands, making a distribution execution and monitoring model according to the optimal distribution path and the receiving end of the data, and distributing the data to the specified receiving end according to a decision result. By automatically sensing data features and user requirements, intelligent distribution of data is realized, manual intervention and distribution delay are reduced, and data security is enhanced.

Description

TECHNICAL FIELD

[0001] The application relates to the technical field of data perception distribution, in particular to a data perception automatic distribution method based on deep learning. BACKGROUND

[0002] At present, the data perception automatic distribution method is an intelligent data processing and distribution strategy combining deep learning technology and data perception capability, which can automatically distribute data to the most suitable receiving end or processing module according to the characteristics, content and user demand of data. The existing data distribution method has low distribution efficiency, cannot realize automatic perception and user demand acquisition, and cannot monitor data in real time during data distribution, so that data is prone to leakage and tampering. SUMMARY

[0003] The application aims to provide a data perception automatic distribution method based on deep learning to solve the problems in the background.

[0004] To achieve the above purpose, the application provides the following technical scheme: a data perception automatic distribution method based on deep learning, comprising the following steps: Step 1: a data collection model is developed to collect data from different sources, a data preprocessing model is developed after data collection is completed to preprocess the collected data, and a feature extraction model is developed to extract features from the preprocessed data; Step 2: a demand analysis model is built to analyze the user's demand, and the analyzed data is marked, a demand prediction model is developed after the marking is completed to predict the user's future data demand; Step 3: a distribution decision model is developed based on data characteristics and user demand, the distribution decision model determines the best distribution path and receiving end of data, a distribution execution and monitoring model is developed to distribute data to the specified receiving end according to the decision result, and a monitoring data collection model is developed to collect the data generated by monitoring.

[0005] Preferably, the data collection model in step 1 is specifically constructed as follows: (1) Multi-source data access: real-time or batch data collection from different data sources through API interface, log file, database query and sensor network; (2) Data format unification: convert the collected data into a unified format, which is more convenient and unified when processing the data later, for example, unify the time stamp in different databases to UTC time, and unify the text encoding to UTF-8.

[0006] Preferably, the data preprocessing model construction step in step one is as follows: (1) Denoising: using filtering algorithms or rule engines to remove noise and outliers in the data; (2) Missing value processing: select filling strategy or delete records with too many missing values according to data characteristics; (3) Data standardization / normalization: scale numerical data to a certain range, eliminate dimension influence, and improve training effect.

[0007] Preferably, the feature extraction model construction step in step one is as follows: (1) Structured data feature extraction: use statistical methods or domain knowledge to extract key features; (2) Unstructured data feature extraction including text data, image data and audio data: ① Text data: use NLP technology to extract semantic features; ② Image data: use CNN model to extract visual features; ③ Audio data: use MFCC, mel spectrum and other feature extraction methods to obtain audio features; (3) Feature fusion: fuse multi-modal features through concatenation, weighted sum or attention mechanism to form comprehensive feature representation.

[0008] Preferably, the demand analysis model construction step in step two is as follows: (1) User avatar construction: collect user's basic information, behavior data and preference settings, etc., and construct user avatar when collection is completed; (2) Real-time demand perception: dynamically adjust demand prediction results through real-time user behavior.

[0009] Preferably, the demand prediction model construction step in step two is as follows: (1) Recurrent neural network and its variants: ① RNN: pass historical information through hidden state, suitable for short-term time series prediction; ③ LSTM: introduce gating mechanism to solve long sequence dependence problem, for example, predict user's future purchase behavior based on historical purchase sequence; ③ GRU: simplify LSTM structure, improve computational efficiency; applicable scenarios: user demand has time dependence; (2) Convolutional neural network: ① 1D-CNN: process one-dimensional time series data, extract local features through convolution kernel; ② 2D-CNN: process image data, extract spatial features; applicable scenarios: user demand is related to local patterns; (3) Transformer model: ① Self-attention mechanism: captures global dependencies by computing attention weights for each position in the sequence; ② Multi-head attention: parallel computation of multiple attention heads to enhance model expressiveness; ③ Position encoding: introduces position information to compensate for the sequential dependence of RNN; Suitable scenarios: user demand prediction needs to handle long sequences and global dependencies; (4) Deep reinforcement learning: ① Q-learning: learns the optimal policy through the Q-value function, suitable for discrete action space; ② Deep Q Network (DQN): combines deep learning with Q-learning to handle high-dimensional state space; ③ Policy Gradient: directly optimizes the policy function, suitable for continuous action space; Suitable scenarios: user demand prediction needs to dynamically adjust the strategy.

[0010] Preferably, the distribution decision model in step three is specifically constructed as follows: (1) Distribution strategy formulation includes the following: ① Rule-based strategy: formulates distribution strategy based on pre-set rules, such as high-value data being preferentially received by senior users; ② Model-based strategy: uses reinforcement learning or optimization algorithms to dynamically adjust the distribution strategy to maximize distribution efficiency or user satisfaction; ③ Hybrid strategy: combines rules and models to flexibly select strategies according to scenarios; (2) Distribution path planning formulation includes the following: ① Network topology analysis: analyzes the topology of the data distribution network to identify key paths and bottlenecks; ② Path selection algorithm: uses Dijkstra algorithm, A* algorithm or Q-learning algorithm to select the optimal distribution path, considering factors such as delay, bandwidth and cost, etc.; ③ Load balancing: dynamically adjusts the distribution path to avoid overloading any node or link, ensuring the stability of the distribution system.

[0011] Preferably, the distribution execution and monitoring model in step three is specifically constructed as follows: (1) Distribution task scheduling: assigns distribution tasks to different distribution nodes (such as CDN nodes, edge computing nodes) and monitors task execution progress; (2) Real-time monitoring and feedback: tracks the distribution process in real time through log collection and performance indicator monitoring, and timely discovers and handles exceptions; (3) Dynamic adjustment: dynamically adjusts the distribution strategy or path based on monitoring results.

[0012] Preferably, the monitoring data collection model in step three is specifically constructed as follows: (1) Collect the data generated during the monitoring process, identify each collected data after the collection is completed, build a storage database to store the identified data, and periodically classify after the storage is completed, according to the storage time; (2) Continuous optimization: according to the evaluation results and user feedback, continuously optimize the distribution model, strategy and path planning algorithm, and improve the distribution efficiency and user experience.

[0013] Compared with the prior art, the beneficial effects of the present application are: The present application realizes to improve the distribution efficiency, realizes the intelligent distribution of data by automatically sensing the data characteristics and user demand, reduces the manual intervention and distribution delay, enhances the data security, monitors the security and integrity of data in real time during the distribution process, ensures that the data is not leaked or tampered, and optimizes the user experience, according to the historical behavior and preference of the user to realize personalized distribution, and improves the satisfaction of the user. BRIEF DESCRIPTION OF DRAWINGS

[0014] Figure 1 The method flowchart provided by the embodiment of the present application is shown. DETAILED DESCRIPTION

[0015] The technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, not all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor are within the scope of protection of the present application.

[0016] Please refer to Figure 1 The present application provides a technical solution: a data sensing automatic distribution method based on deep learning, comprising the following steps: Step one: develop a data collection model to collect data from different sources, develop a data preprocessing model after the data collection is completed, and preprocess the collected data, develop a feature extraction model after the preprocessing is completed, and extract features from the preprocessed data; Step two: build a demand analysis model to analyze the user's demand, and identify the analyzed data, develop a demand prediction model after the processing is completed, and predict the user's future data demand; Step three: combine the data characteristics and the user's demand to develop a distribution decision model, the distribution decision model finds the best distribution path and receiving end of the data, develops a distribution execution and monitoring model, distributes the data to the specified receiving end according to the decision result, and develops a monitoring data collection model to collect the data generated by the monitoring.

[0017] After the distribution is completed, the application monitors the distribution task in real time, traces in the monitoring process, receives abnormal data generated during the distribution implementation process, performs feature identification after receiving is completed, stores after feature identification, and can realize traceability when problems occur later. Through identification, the accuracy and efficiency of traceability can be improved. Through evaluation of the stored data and user feedback, the distribution model can be optimized, and the distribution method can always maintain the best state through continuous processing, avoiding problems in the use process that cannot be processed in the first time, leading to lag in the distribution method, affecting normal use later.

[0018] The specific construction steps of the data collection model in step one are as follows: (1) Multi-source data access: Through API interface, log file, database query and sensor network, etc. From different data sources (such as business system, Internet of Things device, social media, etc.), real-time or batch data collection; (2) Data format unification: Convert the collected data into a unified format (such as JSON, CSV), which is more convenient and unified when processing data later, for example, unify the timestamp in different databases to UTC time, and unify the text encoding to UTF-8.

[0019] The construction steps of the data preprocessing model in step one are as follows: (1) Noise removal: Use filtering algorithms (such as moving average, median filtering) or rule engines (such as regular expression matching) to remove noise and outliers in the data; (2) Missing value processing: According to the characteristics of the data, choose a filling strategy (such as mean filling, median filling, model prediction filling) or delete records with too many missing values; (3) Data standardization / nomalization: Scale numerical data to a set range (such as [0, 1] or [-1, 1]), eliminate dimension influence, and improve training effect.

[0020] The construction steps of the feature extraction model in step one are as follows: (1) Structured data feature extraction: Use statistical methods (such as mean, variance, frequency) or domain knowledge to extract key features, such as extracting access frequency, stay time, etc. in user behavior data; (2) Unstructured data feature extraction including text data, image data and audio data: ① Text data: Use NLP technology (such as TF-IDF, Word2Vec, BERT) to extract semantic features; ② Image data: Use CNN model (such as ResNet, VGG) to extract visual features; ③ Audio data: Use MFCC, Mel spectrum, etc. Feature extraction method to obtain audio features; (3) Feature Fusion: Multi-modal features (e.g., text + image) are fused through concatenation, weighted sum, or attention mechanisms to form a comprehensive feature representation.

[0021] The demand analysis model construction step in step two is as follows: (1) User Avatar Construction: Collect basic information (e.g., age, gender), behavior data (e.g., browsing history, purchase records), and preference settings, and construct a user avatar when the collection is complete. (2) Real-time demand perception: dynamically adjust demand prediction results through real-time user behavior (e.g., clicks, searches), for example, if a user suddenly searches for "mobile phone repair," temporarily increase the distribution priority of related data.

[0022] The demand prediction model construction step in step two is as follows: (1) Recurrent Neural Network and its variants: ① RNN: pass historical information through hidden states, suitable for short-term time series prediction; ③ LSTM: introduce gating mechanisms (input gate, forget gate, output gate) to solve long sequence dependency problems, for example, predict future purchase behavior based on historical purchase sequences; ③ GRU: simplify LSTM structure, improve computational efficiency; applicable scenarios: user demand has time dependence; (2) Convolutional Neural Network: ① 1D-CNN: process one-dimensional time series data, extract local features through convolution kernels, for example, predict user preference for music based on audio feature sequences; ② 2D-CNN: process image data, extract spatial features, for example, predict user purchase intention for clothing based on product images; applicable scenarios: user demand is related to local patterns (such as image, text data); (3) Transformer model: ① Self-attention mechanism: calculate attention weights between positions in the sequence to capture global dependencies; ② Multi-head attention: parallel computing multiple attention heads to enhance model expression ability; ③ Position encoding: introduce position information to make up for the sequential dependence of RNN; applicable scenarios: user demand prediction needs to handle long sequences and global dependencies; (4) Deep Reinforcement Learning: ① Q-learning: learn the optimal strategy through Q-value function, suitable for discrete action space; ② Deep Q Network (DQN): combine deep learning with Q-learning to handle high-dimensional state space; ③ Policy Gradient: directly optimize the policy function, suitable for continuous action space; applicable scenarios: user demand prediction needs to dynamically adjust strategies (such as personalized recommendation).

[0023] The distribution decision model construction step in step three is as follows: (1) Distribution strategy formulation includes the following: ① Rule-based strategy: Develop distribution strategies based on pre-set rules (such as data type, user level), for example, high-level users receive high-value data first; ② Model-based strategy: Use reinforcement learning (such as DQN, PPO) or optimization algorithms (such as genetic algorithm, particle swarm optimization) to dynamically adjust the distribution strategy to maximize distribution efficiency or user satisfaction; ③ Hybrid strategy: Combine rules and models, and flexibly select strategies according to scenarios, for example, use rule strategy in emergency and model strategy in normal situation; (2) Distribution path planning includes the following; ① Network topology analysis: Analyze the topology of the data distribution network (such as node number, link bandwidth), identify key paths and bottlenecks; ② Path selection algorithm: Use Dijkstra algorithm, A* algorithm or Q-learning algorithm to select the optimal distribution path, considering factors such as delay, bandwidth and cost, etc; ③ Load balancing: Avoid overloading any node or link by dynamically adjusting the distribution path to ensure the stability of the distribution system.

[0024] The specific construction steps of the distribution execution and monitoring model in step three are as follows; (1) Distribution task scheduling: Assign distribution tasks to different distribution nodes (such as CDN nodes, edge computing nodes), and monitor task execution progress; (2) Real-time monitoring and feedback: Track the distribution process in real time through log collection, performance indicator monitoring (such as delay, throughput), and timely discovery and handling of exceptions (such as network interruption, node failure); (3) Dynamic adjustment: Dynamically adjust the distribution strategy or path according to the monitoring results, for example, when any link is congested, switch to a backup link; The specific construction steps of the monitoring data collection model in step three are as follows; (1) Collect data generated during monitoring, and when collection is complete, identify each collected data, and when identification is complete, build a storage database to store identified data, and after storage is complete, periodically classify according to storage time; (2) Continuous optimization: Based on evaluation results and user feedback, continuously optimize distribution model, strategy and path planning algorithm to improve distribution efficiency and user experience.

[0025] It is to be understood that the terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting; it is not intended to exclude myriad other embodiments of the present application that other inventors can develop based on the same general inventive concepts embodied by the described embodiments. That is, although the present application is described in terms of particular embodiments and illustrative figures, it should be apparent that the scope of the present application is not limited to these specific embodiments.

[0026] While the embodiments of the application have been shown and described herein, it will be understood by those skilled in the art that many changes, modifications, substitutions and alterations to these embodiments can be made without departing from the principles and spirits of the application, the scope of which is defined by the appended claims and their equivalents.

Claims

1. A method for automatic distribution of data based on deep learning, characterized in that Comprise the following steps; Step one: make a data collection model to collect data from different sources, and make a data preprocessing model after the data collection is completed, and preprocess the collected data, and make a feature extraction model after the preprocessing is completed, and extract features from the preprocessed data; Step two: build a demand analysis model to analyze the user's demand, and mark the analyzed data, and make a demand prediction model after the processing is completed, and predict the user's future data demand; Step three: combine the data features and the user's demand to make a distribution decision model, which decides the best distribution path and receiving end of the data, and makes a distribution execution and monitoring model, which distributes the data to the specified receiving end according to the decision result, and makes a monitoring data collection model to collect the data generated by monitoring. 2.The deep learning based data-aware automatic distribution method according to claim 1, characterized in that: The specific construction steps of the data collection model in step one are as follows; (1) Multi-source data access: real-time or batch collection of data from different data sources through API interface, log file, database query and sensor network; (2) Data format unification: convert the collected data into a unified format, which is more convenient and unified for data processing in the later stage. 3.The deep learning based data-aware automatic distribution method according to claim 2, characterized in that: The construction steps of the data preprocessing model in step one are as follows; (1) De-noising: use filtering algorithm or rule engine to remove noise and outliers in the data; (2) Missing value processing: select filling strategy or delete records with too many missing values according to data characteristics; (3) Data standardization / normalization: scale numerical data to a specified range to eliminate dimension effect and improve training effect. 4.The deep learning based data-aware automatic distribution method according to claim 3, characterized in that: The construction steps of the feature extraction model in step one are as follows; (1) Structured data feature extraction: use statistical methods or domain knowledge to extract key features; (2) Unstructured data feature extraction includes: text data, image data and audio data: ① Text data: use NLP technology to extract semantic features; ② Image data: use CNN model to extract visual features; ③ Audio data: use MFCC or Mel frequency spectrum feature extraction method to obtain audio features; (3) Feature fusion: fuse multi-modal features through concatenation, weighted sum and attention mechanism to form comprehensive feature representation. 5.The deep learning based data-aware automatic distribution method according to claim 4, characterized in that: The construction steps of the demand analysis model in step two are as follows; (1) User avatar construction: collect user's basic information, behavior data and preference settings, and build user avatar after collection is completed; (2) Real-time demand sensing: dynamically adjust demand prediction results through real-time user behavior. 6.The deep learning based data-aware automatic distribution method according to claim 5, characterized in that: The specific construction steps of the demand prediction model in step two are as follows; (1) Recurrent neural network and its variants: ① RNN: pass historical information through hidden state, suitable for short-term time series prediction; ③ LSTM: introduce gating mechanism to solve long sequence dependence problem, including predicting user's future purchase behavior based on historical purchase sequence; ③ GRU: simplify LSTM structure to improve computational efficiency; (2) Convolutional neural network: ① 1D-CNN: process one-dimensional time series data, extract local features through convolution kernel; ② 2D-CNN: process image data, extract spatial features; Applicable scenarios: user demand is related to local patterns; (3) Transformer model: ① Self-attention mechanism: captures global dependencies by calculating attention weights for each position in the sequence; ② Multi-head attention: parallel computing multiple attention heads to enhance model expression ability; ③ Position encoding: introduces position information to make up for the sequential dependence of RNN; Applicable scenarios: user demand prediction needs to handle long sequences and global dependencies; (4) Deep reinforcement learning: ① Q-learning: learn the optimal strategy through Q-value function, suitable for discrete action space; ② Deep Q Network: combines deep learning and Q-learning to handle high-dimensional state space; ③ Policy Gradient: directly optimizes the policy function, suitable for continuous action space; Applicable scenarios: user demand prediction needs to dynamically adjust the strategy. 7.The deep learning based data-aware automatic distribution method according to claim 6, characterized in that: The specific construction steps of the distribution decision model in step three are as follows: (1) Distribution strategy formulation includes the following: ① Rule-based strategy: formulate distribution strategy based on pre-set rules; ② Model-based strategy: use reinforcement learning or optimization algorithm to dynamically adjust the distribution strategy to maximize distribution efficiency or user satisfaction; ③ Hybrid strategy: combine rules and models, and flexibly select strategies according to scenarios; (2) Distribution path planning formulation includes the following: ① Network topology analysis: analyze the topology of the data distribution network, identify key paths and bottlenecks; ② Path selection algorithm: use Dijkstra algorithm, A* algorithm and Q-learning algorithm to select the optimal distribution path, considering factors such as delay, bandwidth and cost; ③ Load balancing: avoid overloading any node or link by dynamically adjusting the distribution path to ensure the stability of the distribution system. 8.The deep learning based data-aware automatic distribution method according to claim 7, characterized in that: The specific construction steps of the distribution execution and monitoring model in step three are as follows: (1) Distribution task scheduling: assign distribution tasks to different distribution nodes and monitor task execution progress; (2) Real-time monitoring and feedback: track the distribution process in real time through log collection and performance indicator monitoring, and timely discover and handle exceptions; (3) Dynamic adjustment: dynamically adjust the distribution strategy or path according to the monitoring results. 9.The deep learning based data-aware automatic distribution method according to claim 8, characterized in that: The specific construction steps of the monitoring data collection model in step three are as follows: (1) Collect data generated during monitoring, and after collection is complete, identify each collected data, and after identification is complete, build a storage database to store the identified data, and after storage is complete, periodically classify, classify according to storage time; (2) Continuous optimization: based on evaluation results and user feedback, continuously optimize distribution model, strategy and path planning algorithm to improve distribution efficiency and user experience.

Images