Method and device for improving data through efficiency based on large model and medium

Abstract

The invention discloses a method and equipment for improving data nonstop efficiency based on a large model, and a medium, belongs to the technical field of large models and data processing, and aims to solve the technical problem of how to reduce time delay of data transmission and processing in a large data processing process, ensure integrity and consistency of data, and improve data nonstop efficiency. According to the technical scheme, the method comprises the steps that multi-source data collection and processing are conducted, specifically, real-time data streams are integrated and synchronized in batches, data are efficiently collected from a database, an API, a file and Internet of Things equipment through a data source adapter, then preprocessing and optimization preprocessing are conducted on the data, and an original data set is obtained; feature engineering: performing feature extraction and feature conversion on the original data set to obtain a pre-training data set; training and optimizing a large model; model deployment and integration: the trained model is exported as a SavedModel format of deployable TensorFlow or a TorchScript format of PyTorch, and real-time data pushing is realized through an API (Application Program Interface) according to a data direct requirement; and a real-time analysis engine.

Description

Technical Field

[0001] The present invention relates to the field of large models and data processing technology, and in particular to a method, device and medium for improving data direct access efficiency based on a large model. Background Art

[0002] Big data technology refers to the technology required for data sets that cannot be captured, managed and processed by conventional software tools within a certain time frame, covering all aspects of big data collection, storage, processing, analysis, mining and visualization. Big data technology is widely used in e-commerce, government affairs, medical care, media, security, finance, telecommunications, transportation, sports, manufacturing, energy and utilities.

[0003] Real-time data processing refers to the process in which computers collect and process field data at the actual time when they occur.

[0004] Therefore, how to reduce the time delay of data transmission and processing in the process of big data processing, ensure the integrity and consistency of data, and improve the timeliness of data delivery is a technical problem that needs to be solved urgently. Summary of the invention

[0005] The technical task of the present invention is to provide a method, device and medium for improving the efficiency of data direct access based on a large model, so as to solve the problem of how to reduce the time delay of data transmission and processing in the process of big data processing, ensure the integrity and consistency of data, and improve the timeliness of data direct access.

[0006] The technical task of the present invention is achieved in the following way: a method for improving the efficiency of direct data access based on a large model, the method is as follows:

[0007] Multi-source data collection and processing: Integrate and synchronize real-time data streams in batches, efficiently collect data from databases, APIs, files, and IoT devices through data source adapters, and then pre-process and optimize pre-processing to obtain the original data set;

[0008] Feature engineering: extract and transform features from the original data set to obtain a pre-training data set;

[0009] Large model training and optimization: According to project requirements and resource conditions, select TensorFlow or PyTorch deep learning framework, and according to the characteristics of the problem, select convolutional neural network (CNN), recurrent neural network (RNN) and Transformer network architecture, use pre-training data sets to train and optimize the large model, and obtain the trained large model;

[0010] Model deployment and integration: Export the trained model to a deployable TensorFlow SavedModel or PyTorch TorchScript format, and push data in real time through the API interface according to the data direct access requirements;

[0011] Real-time analysis engine: Use Apache Flink and Apache Spark Streaming to build real-time data processing processes, implement data filtering, aggregation, association and other processing logic to meet the needs of real-time analysis, and integrate the trained big model into the real-time data processing process to perform real-time data reasoning and prediction. Based on the output results of the trained big model, build a decision support system to provide users with real-time decision suggestions.

[0012] As a preferred method, the real-time data stream integration is to use Apache Kafka, Apache Flink or Amazon Kinesis stream processing framework to achieve real-time data collection and transmission, and monitor the data stream status in real time to ensure the continuity and stability of data collection;

[0013] Batch data synchronization includes: setting up scheduled tasks, using cron jobs or Apache Airflow to regularly synchronize data from data sources, and using incremental data synchronization mechanism to synchronize only newly added or changed data after collection, reducing the amount of data transmission;

[0014] The data source adapter adopts a plug-in architecture, allowing new data source adapters to be added dynamically to adapt to changing data source requirements.

[0015] As a preference, the data is preprocessed and optimized as follows:

[0016] Data cleaning: Automatically clean data based on data cleaning rules that identify and process outliers and missing values, and dynamically adjust cleaning rules based on real-time feedback from data to adapt to changes in data sources;

[0017] Format conversion: Unify the data format and ensure that all collected data is converted into JSON or Parquet standard formats that can be processed by large models;

[0018] Feature standardization: Use the big model to standardize the data so that the data has a consistent distribution and scale before entering the big model;

[0019] Data desensitization: Use large models to assist in identifying sensitive data fields, ensure compliance with data protection regulations during data delivery, and dynamically apply desensitization strategies such as encryption and masking based on data usage scenarios and user permissions;

[0020] Preprocessing optimization: Use distributed computing frameworks (such as Apache Spark) to parallelize data preprocessing tasks to improve processing efficiency, and use large models to predict and guide the preprocessing process, such as predicting which data fields may require more cleaning work.

[0021] As a preference, feature extraction is specifically as follows:

[0022] Data exploration: Conduct exploratory analysis on raw data to obtain data distribution, outliers, and missing values;

[0023] Feature identification: Automatically identify important features using statistical methods and machine learning algorithms (e.g., decision trees, random forests);

[0024] Feature construction: Based on business knowledge and domain expert opinions, new features are constructed, such as time window statistical features, text embedding features, etc.

[0025] As a preference, the feature conversion is specifically as follows:

[0026] Standardization: Apply Z-score standardization or Min-Max standardization method to make the feature data have a unified standard scale;

[0027] Encoding: Encoding the category features through One-Hot Encoding or Label Encoding;

[0028] Dimensionality reduction: Use principal component analysis (PCA) and t-SNE methods to reduce feature dimensions and improve model training efficiency.

[0029] As a preferred method, large model training and optimization are as follows:

[0030] Dataset preparation: Divide the dataset into training set, validation set and test set;

[0031] Hyperparameter tuning: adjust the learning rate, batch size, number of layers, and number of neurons hyperparameters through cross-validation method;

[0032] Training process: Use GPU or TPU hardware to accelerate the training process and monitor training loss and accuracy metrics;

[0033] Regularization: Apply L1, L2 regularization or dropout method to prevent overfitting;

[0034] Model pruning: reduce redundant parameters in the model, improve model efficiency and reasoning speed;

[0035] Quantization: Reduce model size and improve inference speed through model quantization.

[0036] Preferably, the model deployment and integration is as follows:

[0037] Server configuration: Configure the server hardware and software environment to ensure that the requirements for running large models after training are met;

[0038] Containerization: Use Docker container technology to encapsulate the trained large model and its dependencies into a portable container;

[0039] API encapsulation: Create an API interface for the trained large model so that the trained large model can be called by other systems;

[0040] Data flow integration: ensuring that the input and output of the trained large model are seamlessly connected with the data collection, pre-processing and post-processing stages;

[0041] Performance monitoring: After deployment, monitor the performance and resource consumption of the trained model to ensure stable operation of the system;

[0042] Interface definition: Define the RESTful API interface based on the needs of direct data access, and clarify the interface's functions, parameters, and return format;

[0043] Interface implementation: Use frameworks such as Flask or FastAPI to implement API interfaces and ensure high performance and security of the interfaces;

[0044] Data push: Real-time data push is achieved through WebSocket or Server-Sent Events (SSE) technology;

[0045] Message queue selection: Use RabbitMQ or Apache Kafka message queue system to achieve asynchronous processing and transmission of data;

[0046] Message format: Define a unified message format to ensure data consistency and parsability;

[0047] Message reliability: Ensure the reliability of the message queue and prevent data loss.

[0048] A system for improving data direct access efficiency based on a large model, the system is used to implement the method for improving data direct access efficiency based on a large model as described above; the system comprises:

[0049] Multi-source data acquisition and processing unit, used to integrate and synchronize real-time data streams in batches, efficiently collect data from databases, APIs, files, and IoT devices through data source adapters, and then pre-process and optimize pre-processing of the data to obtain the original data set;

[0050] Feature engineering unit, used to extract and transform features from the original data set to obtain a pre-training data set;

[0051] The large model training and optimization unit is used to select the TensorFlow or PyTorch deep learning framework according to project requirements and resource conditions, and select the network architecture of convolutional neural network (CNN), recurrent neural network (RNN) and Transformer according to the characteristics of the problem, and use the pre-training data set to train and optimize the large model to obtain the trained large model;

[0052] Model deployment and integration unit, used to export trained models into deployable TensorFlow SavedModel or PyTorch TorchScript formats, and push data in real time through API interfaces according to data direct access requirements;

[0053] The real-time analysis engine unit is used to build real-time data processing processes using Apache Flink and Apache Spark Streaming, implement data filtering, aggregation, association and other processing logic to meet the needs of real-time analysis, and integrate the trained large model into the real-time data processing process to perform real-time data reasoning and prediction. Based on the output results of the trained large model, a decision support system is built to provide users with real-time decision suggestions.

[0054] An electronic device comprising: a memory and at least one processor;

[0055] Wherein, the memory stores computer-executable instructions;

[0056] The at least one processor executes the computer-executable instructions stored in the memory, so that the at least one processor executes the method for improving data direct access efficiency based on a large model as described above.

[0057] A computer-readable storage medium stores computer-executable instructions. When a processor executes the computer-executable instructions, the method for improving data direct access efficiency based on a large model as described above is implemented.

[0058] The method, device and medium for improving data direct access efficiency based on a large model of the present invention have the following advantages:

[0059] (i) The present invention optimizes the data flow to optimize the data flow process from source to destination and reduce the time delay of data transmission and processing;

[0060] (ii) The present invention realizes data fusion and integration: using large models to achieve cross-departmental and cross-domain data fusion, ensuring the integrity and consistency of data when it is directly received;

[0061] (III) The present invention realizes real-time data processing: through a large model, real-time data processing and analysis are realized, thus improving the timeliness of data access;

[0062] (iv) The present invention achieves automated data preprocessing: the large model can automatically process and clean the data, identify errors and inconsistencies in the data, and thus improve the data quality, which is crucial to ensure the accuracy of information in the data delivery process;

[0063] (V) The present invention achieves intelligent data classification and labeling: The large model can intelligently classify and label data, which enables relevant data to be located and distributed to appropriate recipients more quickly during the data delivery process;

[0064] (vi) The present invention achieves data mining and insight: large models can quickly extract useful information from large amounts of data and discover patterns and associations between data, which helps to quickly generate valuable insights in the data-direct process;

[0065] (VII) The present invention can use large models to understand and extract key information from unstructured data, such as text and speech, so that these data can be effectively utilized in the data direct access process;

[0066] (VIII) The large model of the present invention can predict future trends or behaviors based on historical data, which is very useful for automated decision support in data access, such as predicting which data is most important to a specific grassroots unit;

[0067] (IX) The present invention uses massive amounts of data for training to ensure that the model can capture complex patterns and relationships in the data, and applies deep learning techniques, such as neural networks, to build large models with a large number of parameters;

[0068] (10) The present invention efficiently cleans, normalizes and extracts features from input data, reduces noise, improves data quality, and implements data enhancement strategies to expand the data set and enhance the generalization ability of the model;

[0069] (XI) The present invention uses model pruning, quantization and other technologies to reduce the model size, improve the model's reasoning speed, and implement knowledge distillation to migrate the knowledge of large models to smaller models to maintain performance while reducing computing resource consumption;

[0070] (XII) The present invention uses a dedicated reasoning engine, such as TensorRT or ONNX Runtime, to optimize the reasoning process of the model and supports distributed reasoning of the model, using multi-core CPUs, GPUs or dedicated accelerators to process data in parallel;

[0071] (XIII) The present invention designs an intelligent routing algorithm based on data characteristics and model features to efficiently distribute data to different processing paths, and implements a dynamic routing strategy to automatically adjust the data processing flow based on system load and model performance;

[0072] (XIV) The present invention introduces an efficient cache mechanism to cache frequently accessed data or model inference results to reduce repeated calculations, and implements a cache elimination mechanism based on strategies such as LRU to keep the data in the cache as the most useful;

[0073] (XV) The present invention uses real-time monitoring data to evaluate model performance, dynamically adjusts based on feedback, and implements online learning or incremental learning strategies so that the model can continuously learn and update from new data;

[0074] (XVI) The present invention optimizes the entire data processing link end-to-end, reduces the delay of data processing and transmission, and combines hardware characteristics, such as using specific hardware accelerators to optimize data processing speed. BRIEF DESCRIPTION OF THE DRAWINGS

[0075] The present invention is further described below in conjunction with the accompanying drawings.

[0076] Attached Figure 1 The flowchart of the method for improving the efficiency of direct data access based on large models. DETAILED DESCRIPTION

[0077] The method, device and medium for improving data direct access efficiency based on a large model of the present invention are described in detail below with reference to the accompanying drawings and specific embodiments of the specification.

[0078] Embodiment 1:

[0079] As attached Figure 1 As shown, this embodiment provides a method for improving data direct access efficiency based on a large model, and the method is specifically as follows:

[0080] S1. Multi-source data collection and processing: Integrate and synchronize real-time data streams in batches, efficiently collect data from databases, APIs, files, and IoT devices through data source adapters, and then pre-process and optimize pre-processing of the data to obtain the original data set;

[0081] S2. Feature engineering: extract and transform features from the original data set to obtain a pre-training data set;

[0082] S3. Large model training and optimization: According to project requirements and resource conditions, select TensorFlow or PyTorch deep learning framework, and according to the characteristics of the problem, select convolutional neural network (CNN), recurrent neural network (RNN) and Transformer network architecture, use pre-training data sets to train and optimize the large model, and obtain the trained large model;

[0083] S4, Model deployment and integration: Export the trained model to a deployable TensorFlow SavedModel or PyTorch TorchScript format, and push data in real time through the API interface according to the data direct access requirements;

[0084] S5. Real-time analysis engine: Use Apache Flink and Apache Spark Streaming to build real-time data processing processes, implement data filtering, aggregation, association and other processing logic to meet the needs of real-time analysis, and integrate the trained big model into the real-time data processing process to perform real-time data reasoning and prediction. Based on the output results of the trained big model, build a decision support system to provide users with real-time decision suggestions.

[0085] The integrated real-time data stream in step S1 of this embodiment is specifically: using Apache Kafka, Apache Flink or Amazon Kinesis stream processing framework to achieve real-time data collection and transmission, and real-time monitoring of data stream status to ensure the continuity and stability of data collection.

[0086] The batch data synchronization in step S1 of this embodiment specifically includes: setting a scheduled task, using a cron job or Apache Airflow to periodically synchronize data from the data source, and using an incremental data synchronization mechanism to synchronize only newly added or changed data after collection, thereby reducing the amount of data transmission.

[0087] The data source adapter in step S1 of this embodiment adopts a plug-in architecture, allowing new data source adapters to be added dynamically to adapt to changing data source requirements.

[0088] The data preprocessing and optimization preprocessing in step S1 of this embodiment are specifically as follows:

[0089] S101, Data cleaning: Automatically clean data according to data cleaning rules that identify and process outliers and missing values, and dynamically adjust cleaning rules based on real-time feedback from data to adapt to changes in data sources;

[0090] S102, Format conversion: Unify the data format and ensure that all collected data is converted into JSON or Parquet standard formats that can be processed by the large model;

[0091] S103, feature standardization: Use the big model to standardize the data so that the data has a consistent distribution and scale before entering the big model;

[0092] S104, Data Desensitization: Use large models to assist in identifying sensitive data fields, ensure compliance with data protection regulations during data delivery, and dynamically apply desensitization strategies such as encryption and masking based on data usage scenarios and user permissions;

[0093] S105. Preprocessing optimization: Use distributed computing frameworks (such as Apache Spark) to parallelize data preprocessing tasks to improve processing efficiency, and use large models to predict and guide the preprocessing process, such as predicting which data fields may require more cleaning work.

[0094] The feature extraction in step S2 of this embodiment is specifically as follows:

[0095] S2-101, Data Exploration: Conduct exploratory analysis on the original data to obtain data distribution, outliers and missing values;

[0096] S2-102, Feature Identification: Automatically identify important features using statistical methods and machine learning algorithms (e.g., decision trees, random forests);

[0097] S2-103, Feature construction: Based on business knowledge and domain expert opinions, construct new features, such as time window statistical features, text embedding features, etc.;

[0098] The feature conversion in step S2 of this embodiment is specifically as follows:

[0099] S2-201, Standardization: Apply Z-score standardization or Min-Max standardization method to make the feature data have a unified standard scale;

[0100] S2-202, Encoding: Encode the category features through One-Hot Encoding or Label Encoding;

[0101] S2-203. Dimensionality reduction: Use principal component analysis (PCA) and t-SNE methods to reduce feature dimensions and improve model training efficiency.

[0102] The large model training and optimization in step S3 of this embodiment are specifically as follows:

[0103] S301, data set preparation: divide the data set into training set, validation set and test set;

[0104] S302, Hyperparameter tuning: Adjust the learning rate, batch size, number of layers, and number of neurons hyperparameters through cross-validation method;

[0105] S303, training process: use GPU or TPU hardware to accelerate the training process and monitor training loss and accuracy indicators;

[0106] S304, Regularization: Apply L1, L2 regularization or dropout method to prevent overfitting;

[0107] S305, model pruning: reduce redundant parameters in the model and improve model efficiency and reasoning speed;

[0108] S306, Quantization: Reduce model size and improve inference speed through model quantization.

[0109] The model deployment and integration in step S4 of this embodiment are as follows:

[0110] S401, Server configuration: Configure the server hardware and software environment to ensure that the requirements for running large models after training are met;

[0111] S402, Containerization: Use Docker container technology to encapsulate the trained large model and its dependencies into a portable container;

[0112] S403, API encapsulation: creating an API interface for the trained large model so that the trained large model can be called by other systems;

[0113] S404, data flow integration: ensure that the input and output of the trained large model are seamlessly connected with the data collection, pre-processing and post-processing links;

[0114] S405, Performance monitoring: After deployment, monitor the performance and resource consumption of the trained model to ensure stable operation of the system;

[0115] S406, Interface definition: Based on the requirements of direct data access, define the RESTful API interface and clarify the functions, parameters and return format of the interface;

[0116] S407, Interface implementation: Use frameworks such as Flask or FastAPI to implement API interfaces to ensure high performance and security of the interfaces;

[0117] S408, data push: Real-time data push is achieved through WebSocket or Server-Sent Events (SSE) technology;

[0118] S409, message queue selection: Use RabbitMQ or Apache Kafka message queue system to achieve asynchronous processing and transmission of data;

[0119] S410, Message format: Define a unified message format to ensure data consistency and parsability;

[0120] S411. Message reliability: Ensure the reliability of the message queue and prevent data loss.

[0121] Embodiment 2:

[0122] This embodiment provides a system for improving data direct access efficiency based on a large model, and the system is used to implement the method for improving data direct access efficiency based on a large model in Embodiment 1; the system includes:

[0123] Multi-source data acquisition and processing unit, used to integrate and synchronize real-time data streams in batches, efficiently collect data from databases, APIs, files, and IoT devices through data source adapters, and then pre-process and optimize pre-processing of the data to obtain the original data set;

[0124] Feature engineering unit, used to extract and transform features from the original data set to obtain a pre-training data set;

[0125] The large model training and optimization unit is used to select the TensorFlow or PyTorch deep learning framework according to project requirements and resource conditions, and select the network architecture of convolutional neural network (CNN), recurrent neural network (RNN) and Transformer according to the characteristics of the problem, and use the pre-training data set to train and optimize the large model to obtain the trained large model;

[0126] Model deployment and integration unit, used to export trained models into deployable TensorFlow SavedModel or PyTorch TorchScript formats, and push data in real time through API interfaces according to data direct access requirements;

[0127] The real-time analysis engine unit is used to build real-time data processing processes using Apache Flink and Apache Spark Streaming, implement data filtering, aggregation, association and other processing logic to meet the needs of real-time analysis, and integrate the trained large model into the real-time data processing process to perform real-time data reasoning and prediction. Based on the output results of the trained large model, a decision support system is built to provide users with real-time decision suggestions.

[0128] The specific applications of this system are as follows:

[0129] (1) Service deployment: Deploy the API interface and message queue to the server or cloud platform, and configure load balancing and automatic expansion to ensure the stability and scalability of the service;

[0130] (ii) System integration: Integrate the real-time analysis engine with the existing data collection and pre-processing systems to ensure smooth flow and efficient processing of data throughout the entire process;

[0131] (III) Monitoring and maintenance: monitor the performance and health of the API interface and message queue, and regularly maintain and optimize the real-time analysis engine to adapt to data changes and business needs;

[0132] (iv) User interaction: providing a user interface (UI) or dashboard to enable users to intuitively view data analysis results and receive decision-making recommendations, and supporting users to query data and subscribe to data direct services through API interfaces;

[0133] (V) Distributed computing implementation, as follows:

[0134] (1) Framework selection and configuration, as follows:

[0135] Apache Spark: Choose Spark as the primary distributed computing framework because it provides Resilient Distributed Datasets (RDDs) and DataFrame APIs for easy large-scale data processing; and configure a Spark cluster, including setting the correct master URL, worker nodes, executors, and memory allocation;

[0136] Apache Flink:

[0137] For applications that require stream processing, choose Flink, which supports event-driven computing, and configure Flink's parallelism to ensure that it matches the cluster resources.

[0138] (2) Task scheduling, as follows:

[0139] ① Task division: Use Spark's repartition() or coalesce() method to repartition the data to create more even subtasks; in Flink, control the parallelism of the operation by setting the parallelism parameter;

[0140] ②Data dependency management: In Spark, the concepts of wide dependency and narrow dependency are used to optimize task scheduling and reduce the overhead of data shuffle; in Flink, its data flow graph is used to optimize the dependencies between tasks;

[0141] (3) Data partitioning, as follows:

[0142] ① Partitioning strategy: Choose a suitable partition key based on data characteristics, such as hash partitioning or range partitioning; in Spark, you can use the partitionBy() method to partition the DataFrame;

[0143] ② Dynamic partition adjustment: Monitor data processing performance and dynamically adjust the number of partitions as needed to optimize resource utilization.

[0144] (VI) Memory computing implementation: Memory database application; details are as follows:

[0145] (1) Redis configuration: Configure Redis data structures, such as strings, hashes, lists, sets, and ordered sets, to store different types of data, and use Redis persistence features to ensure data security;

[0146] (2) Memcached integration: Integrate Memcached into the data processing flow to cache query results and intermediate data;

[0147] (3) Data caching, as follows:

[0148] ① Hot data cache: Identify hot data, cache it using an in-memory database, and set cache invalidation policies, such as LRU (least recently used) or TTL (time to live);

[0149] ②Cache update mechanism: When the underlying data is updated, the data in the cache is updated synchronously to ensure data consistency;

[0150] (VII) Model selection implementation, as follows:

[0151] (1) Characteristic analysis, as follows:

[0152] ① Data type identification: For structured data (such as database tables), a decision tree-based model or a deep neural network may be selected; for unstructured data (such as images, text), a CNN or Transformer model may be selected;

[0153] ②Data pattern analysis: For time series data, analyze the periodicity, trend, and seasonality of the data to determine whether a time series model such as RNN or LSTM is needed;

[0154] (2) Architecture design, as follows:

[0155] ①CNN architecture: For image recognition tasks, design a CNN architecture that includes convolutional layers, pooling layers, and fully connected layers, and choose the appropriate filter size, step size, and activation function;

[0156] ②RNN / LSTM architecture: For time series data, design a network with multiple LSTM units to capture long-term dependencies and determine the size and number of hidden layers, as well as whether a bidirectional LSTM is needed;

[0157] (3) Parameter tuning is implemented as follows:

[0158] Search strategy: Grid search, specifically: define a grid of hyperparameters, systematically traverse all possible parameter combinations, and use parallel computing resources to accelerate the search process;

[0159] (4) Bayesian optimization: Uses an optimization method based on a probabilistic model, such as a Gaussian process, to select the next hyperparameter combination. This is usually more efficient than grid search because it takes into account previous evaluation results.

[0160] (5) Verification method:

[0161] ① Cross-validation: Implement k-fold cross-validation to evaluate model performance, ensure the stability and reliability of the evaluation results, and select an appropriate k value to balance computational cost and evaluation quality;

[0162] (VIII) Model compression and acceleration implementation, as follows:

[0163] (1) Pruning technology, as follows:

[0164] ①Structural pruning: Delete the entire neuron or filter in the network, which usually does not destroy the compatibility of the framework, and evaluate the performance of the pruned model to ensure that the accuracy loss is within an acceptable range;

[0165] ② Unstructured pruning: Deleting individual weights in the network, which may result in a sparse matrix and may require specialized libraries or hardware support;

[0166] (2) Quantification techniques, as follows:

[0167] ① Weight and activation quantization: Convert floating-point weights and activations to 8-bit or 16-bit integers. This can be achieved through quantization tables, and use Quantization-Aware Training (QAT) to simulate the quantization effect and reduce the error introduced by quantization;

[0168] (3) Inference optimization: Use specialized inference engines, such as TensorRT or ONNX Runtime, to further optimize the model, including fusing operations and optimizing memory access patterns;

[0169] (IX) Encrypted transmission implementation, as follows:

[0170] (1) The implementation of the Agreement is as follows:

[0171] ①SSL / TLS configuration: Configure SSL / TLS between the server and the client to ensure that all data transmission is carried out through an encrypted channel; for Web services, configure HTTPS instead of HTTP;

[0172] ② Transport layer security: Ensure the use of the latest TLS versions, such as TLS1.3, to provide stronger encryption and security, and implement the correct encryption suite to avoid using known insecure algorithms;

[0173] (2) Certificate management, as follows:

[0174] ① Certificate procurement and deployment: Purchase an SSL certificate from a trusted certificate authority (CA), deploy the certificate on the server, and configure the web server (such as Nginx, Apache) to use the certificate;

[0175] ② Certificate renewal and monitoring: Regularly check the validity period of the certificate, renew it before it expires, and use certificate monitoring tools to remind you that the certificate is about to expire;

[0176] (3) Differential privacy implementation, as follows:

[0177] Noise is added as follows:

[0178] Privacy mechanism design: Before data is released, a noise addition mechanism is designed based on the definition of differential privacy. Random noise is added to the data using algorithms such as the Laplace mechanism or Gaussian mechanism.

[0179] Algorithm implementation: Implement the noise addition algorithm to ensure that the algorithm can protect the overall statistical characteristics of the data set without leaking individual information;

[0180] (4) Privacy budget, as follows:

[0181] Privacy budget setting: According to data sensitivity and privacy requirements, a reasonable privacy budget ε (epsilon) is set. The privacy budget determines the amount of noise that can be added to the data and the balance between data availability and privacy protection.

[0182] Budget management: During data processing, monitor the used privacy budget to ensure that it does not exceed the set threshold; for multiple data publishing operations, reasonably allocate the privacy budget;

[0183] (X) Microservice architecture, as follows:

[0184] (1) Service splitting, as follows:

[0185] ① Functional decomposition: decompose complex system functions into small, single-responsibility service units. For example, a data direct access system may include data collection services, data preprocessing services, model training services, model inference services, etc.;

[0186] ② Independent deployment: Each microservice can be deployed and expanded independently, so that resources can be allocated to specific services according to needs, improving resource utilization;

[0187] (2) Service communications, as follows:

[0188] ①API Gateway: Set up the API gateway as the single entry point of the system to manage the routing of external requests to internal services; the API gateway can also provide authentication, current limiting, monitoring and other functions;

[0189] ②Service interface: Define clear service interfaces (APIs) to ensure that different services can exchange data efficiently through these interfaces;

[0190] (3) Service governance, as follows:

[0191] ① Service discovery, as follows:

[0192] Registration and discovery: Using service discovery tools such as Consul and Eureka, microservices register themselves when they start up and find the location of other services through the service discovery mechanism at runtime;

[0193] Dynamic configuration: The service discovery mechanism can also be combined with the configuration center to achieve dynamic configuration management of services;

[0194] ② Load balancing, as follows:

[0195] ① Request distribution: Use tools such as Nginx, HAProxy or Kubernetes to distribute requests to different service instances according to preset rules (such as polling, minimum number of connections, etc.);

[0196] ② Elastic scaling: Combined with container orchestration systems (such as Kubernetes), the number of service instances can be automatically adjusted according to load conditions;

[0197] ③ Circuit breaker downgrade, as follows:

[0198] ① Circuit breaker mechanism: Integrate libraries such as Hystrix or Resilience4j. When the number of service call failures exceeds the threshold, the circuit breaker is automatically triggered to prevent system avalanche.

[0199] ② Degradation strategy: Define degradation strategies, such as returning default values ​​or caching results, to ensure that the system can still provide basic functions when some services are unavailable;

[0200] (4) Performance optimization, as follows:

[0201] ① Resource utilization: In the data preprocessing and feature extraction stages, use multi-threading or multi-processing to process data in parallel, making full use of the multi-core characteristics of the CPU;

[0202] ②Thread / process pool: implement thread pool or process pool management to avoid the overhead caused by frequent creation and destruction of threads / processes;

[0203] (5) Distributed computing, as follows:

[0204] ①Task scheduling: In distributed computing frameworks such as Spark and Flink, tasks are divided reasonably to achieve distributed parallel processing of data;

[0205] ②Data partitioning: partition the data reasonably to ensure that the data is evenly distributed on different computing nodes and improve the efficiency of parallel computing;

[0206] (6) Cache mechanism, as follows:

[0207] ① Hot data cache: Use memory databases such as Redis and Memcached to cache hot data and reduce access to backend storage systems;

[0208] ② Calculation result cache: Time-consuming calculation results, such as model inference results, are cached for quick response to subsequent requests;

[0209] (7) Data update, as follows:

[0210] ① Cache invalidation: Design cache invalidation and update mechanisms, such as using TTL (Time To Live) or LRU (Least Recently Used) strategies;

[0211] ②Data synchronization: ensure that the cache can be synchronized in time when the data source is updated to maintain data consistency;

[0212] (8) Monitoring and operation and maintenance, as follows:

[0213] ① Log integration: Integrate log collection systems such as ELK Stack or Fluentd to achieve centralized management and analysis of logs;

[0214] ② Log structuring: Ensure that log output is structured to facilitate subsequent analysis and processing;

[0215] ③Performance monitoring: monitor system performance indicators such as response time, throughput, etc. through log analysis;

[0216] ④Anomaly detection: Analyze abnormal information in the log to promptly discover and locate system problems;

[0217] (9) Automated deployment, as follows:

[0218] ①CI / CD process: Establish a CI / CD process and use tools such as Jenkins and GitLab CI / CD to automate the code testing, building, and deployment process;

[0219] ② Version control: Through version control, ensure that the deployed version is traceable, which facilitates problem troubleshooting and rollback;

[0220] (10) Monitoring alarms, as follows:

[0221] ① System monitoring: Use tools such as Prometheus and Grafana to monitor various system indicators, such as CPU usage, memory usage, etc.;

[0222] ②Alarm mechanism: Set alarm rules to notify operation and maintenance personnel via email, text message, etc. when system indicators are abnormal.

[0223] Embodiment 3:

[0224] This embodiment also provides an electronic device, including: a memory and at least one processor;

[0225] Wherein, the memory stores computer-executable instructions;

[0226] The at least one processor executes the computer-executable instructions stored in the memory, so that the at least one processor executes the method for improving data direct access efficiency based on a large model as described in any one of the present inventions.

[0227] The processor may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The processor may be a microprocessor or any conventional processor, etc.

[0228] The memory can be used to store computer programs and / or modules. The processor realizes various functions of the electronic device by running or executing the computer programs and / or modules stored in the memory, and calling the data stored in the memory. The memory can mainly include a program storage area and a data storage area, wherein the program storage area can store an operating system, at least one application required for a function, etc.; the data storage area can store data created according to the use of the terminal, etc. In addition, the memory can also include a high-speed random access memory, and can also include a non-volatile memory, such as a hard disk, a memory, a plug-in hard disk, a smart memory card (SMC), a secure digital (SD) card, a flash memory card, at least one disk storage period, a flash memory device, or other volatile solid-state storage devices.

[0229] Embodiment 4:

[0230] This embodiment also provides a computer-readable storage medium, which stores a plurality of instructions, which are loaded by a processor to enable the processor to execute the method for improving data direct access efficiency based on a large model in any embodiment of the present invention. Specifically, a system or device equipped with a storage medium can be provided, on which a software program code that implements the functions of any of the above embodiments is stored, and a computer (or CPU or MPU) of the system or device reads and executes the program code stored in the storage medium.

[0231] In this case, the program code itself read from the storage medium can realize the function of any one of the above-mentioned embodiments, and thus the program code and the storage medium storing the program code constitute a part of the present invention.

[0232] The storage medium embodiments for providing the program code include a floppy disk, a hard disk, a magneto-optical disk, an optical disk (such as CD-ROM, CD-R, CD-RW, DVD-ROM, DVD-RAM, DVD-RW, DVD+RW), a magnetic tape, a non-volatile memory card, and a ROM. Alternatively, the program code can be downloaded from a server computer by a communication network.

[0233] In addition, it should be clear that the functions of any of the above embodiments can be implemented not only by executing the program code read by the computer, but also by enabling an operating system operating on the computer to complete part or all of the actual operations based on instructions from the program code.

[0234] In addition, it can be understood that the program code read from the storage medium is written to a memory provided in an expansion board inserted into the computer or written to a memory provided in an expansion unit connected to the computer, and then based on the instructions of the program code, a CPU installed on the expansion board or the expansion unit is enabled to perform part or all of the actual operations, thereby realizing the functions of any of the above-mentioned embodiments.

[0235] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for improving data direct access efficiency based on a large model, characterized in that: The method is as follows: Multi-source data collection and processing: Integrate and synchronize real-time data streams in batches, efficiently collect data from databases, APIs, files, and IoT devices through data source adapters, and then pre-process and optimize pre-processing to obtain the original data set; Feature engineering: extract and transform features from the original data set to obtain a pre-training data set; Large model training and optimization: According to project requirements and resource conditions, select TensorFlow or PyTorch deep learning framework, and according to the characteristics of the problem, select convolutional neural network, recurrent neural network and Transformer network architecture, use pre-training data sets to train and optimize the large model, and obtain the trained large model; Model deployment and integration: Export the trained model to a deployable TensorFlow SavedModel or PyTorch TorchScript format, and push data in real time through the API interface according to the data direct access requirements; Real-time analysis engine: Use Apache Flink and Apache Spark Streaming to build real-time data processing processes, implement data filtering, aggregation, association and other processing logic to meet the needs of real-time analysis, and integrate the trained big model into the real-time data processing process to perform real-time data reasoning and prediction. Based on the output results of the trained big model, build a decision support system to provide users with real-time decision suggestions.

2. The method for improving data direct access efficiency based on a large model according to claim 1 is characterized in that: Integrating real-time data streams specifically involves: using Apache Kafka, Apache Flink, or Amazon Kinesis stream processing frameworks to achieve real-time data collection and transmission, and real-time monitoring of data stream status to ensure the continuity and stability of data collection; Batch data synchronization includes: setting up scheduled tasks, using cron jobs or Apache Airflow to regularly synchronize data from data sources, and using incremental data synchronization mechanism to synchronize only newly added or changed data after collection, reducing the amount of data transmission; The data source adapter adopts a plug-in architecture, allowing new data source adapters to be added dynamically to adapt to changing data source requirements.

3. The method for improving data direct access efficiency based on a large model according to claim 1 is characterized in that: The data preprocessing and optimization preprocessing are as follows: Data cleaning: Automatically clean data based on data cleaning rules that identify and process outliers and missing values, and dynamically adjust cleaning rules based on real-time feedback from data to adapt to changes in data sources; Format conversion: Unify the data format and ensure that all collected data is converted into JSON or Parquet standard formats that can be processed by large models; Feature standardization: Use the big model to standardize the data so that the data has a consistent distribution and scale before entering the big model; Data desensitization: Use large models to assist in identifying sensitive data fields, ensure compliance with data protection regulations during data delivery, and dynamically apply desensitization strategies based on data usage scenarios and user permissions; Preprocessing optimization: Use distributed computing frameworks to parallelize data preprocessing tasks and use large models to predict and guide the preprocessing process.

4. The method for improving data direct access efficiency based on a large model according to claim 1 is characterized in that: The feature extraction is as follows: Data exploration: Conduct exploratory analysis on raw data to obtain data distribution, outliers, and missing values; Feature recognition: Automatically identify important features using statistical methods and machine learning algorithms; Feature construction: Build new features based on business knowledge and domain expert opinions.

5. The method for improving data direct access efficiency based on a large model according to claim 1 is characterized in that: The feature transformation is as follows: Standardization: Apply Z-score standardization or Min-Max standardization method to make the feature data have a unified standard scale; Encoding: Encode the category features through one-hot encoding or label encoding; Dimensionality reduction: Use principal component analysis and t-SNE methods to reduce feature dimensions and improve model training efficiency.

6. The method for improving data direct access efficiency based on a large model according to claim 1, characterized in that: The details of large model training and optimization are as follows: Dataset preparation: Divide the dataset into training set, validation set and test set; Hyperparameter tuning: adjust the learning rate, batch size, number of layers, and number of neurons hyperparameters through cross-validation method; Training process: Use GPU or TPU hardware to accelerate the training process and monitor training loss and accuracy metrics; Regularization: Apply L1, L2 regularization or dropout method to prevent overfitting; Model pruning: reduce redundant parameters in the model, improve model efficiency and reasoning speed; Quantization: Reduce model size and improve inference speed through model quantization.

7. The method for improving data direct access efficiency based on a large model according to any one of claims 1 to 6, characterized in that: The model deployment and integration are as follows: Server configuration: Configure the server hardware and software environment to ensure that the requirements for running large models after training are met; Containerization: Use Docker container technology to encapsulate the trained large model and its dependencies into a portable container; API encapsulation: Create an API interface for the trained large model so that the trained large model can be called by other systems; Data flow integration: ensuring that the input and output of the trained large model are seamlessly connected with the data collection, pre-processing and post-processing stages; Performance monitoring: After deployment, monitor the performance and resource consumption of the trained model to ensure stable operation of the system; Interface definition: Define the RESTful API interface based on the needs of direct data access, and clarify the interface's functions, parameters, and return format; Interface implementation: Use frameworks such as Flask or FastAPI to implement API interfaces and ensure high performance and security of the interfaces; Data push: Real-time data push is achieved through WebSocket or Server-Sent Events technology; Message queue selection: Use RabbitMQ or Apache Kafka message queue system to achieve asynchronous processing and transmission of data; Message format: Define a unified message format to ensure data consistency and parsability; Message reliability: Ensure the reliability of the message queue and prevent data loss.

8. A system for improving data direct access efficiency based on a large model, characterized in that: The system is used to implement the method for improving data direct access efficiency based on a large model as described in any one of claims 1 to 7; the system includes: Multi-source data acquisition and processing unit, used to integrate and synchronize real-time data streams in batches, efficiently collect data from databases, APIs, files, and IoT devices through data source adapters, and then pre-process and optimize pre-processing of the data to obtain the original data set; Feature engineering unit, used to extract and transform features from the original data set to obtain a pre-training data set; The large model training and optimization unit is used to select the TensorFlow or PyTorch deep learning framework according to project requirements and resource conditions, and select the network architecture of convolutional neural network, recurrent neural network and Transformer according to the characteristics of the problem, and use the pre-training data set to train and optimize the large model to obtain the trained large model; Model deployment and integration unit, used to export trained models into deployable TensorFlow SavedModel or PyTorch TorchScript formats, and push data in real time through API interfaces according to data direct access requirements; The real-time analysis engine unit is used to build real-time data processing processes using Apache Flink and Apache Spark Streaming, implement data filtering, aggregation, association and other processing logic to meet the needs of real-time analysis, and integrate the trained large model into the real-time data processing process to perform real-time data reasoning and prediction. Based on the output results of the trained large model, a decision support system is built to provide users with real-time decision suggestions.

9. An electronic device, characterized in that: include: memory and at least one processor; Wherein, the memory stores computer-executable instructions; The at least one processor executes the computer-executable instructions stored in the memory, so that the at least one processor executes the method for improving data direct access efficiency based on a large model as described in any one of claims 1 to 4.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer execution instructions, and when the processor executes the computer execution, the method for improving data direct access efficiency based on a large model as described in any one of claims 1 to 7 is implemented.

Images