Intelligent data processing device based on large model, medium and electronic equipment

Abstract

The invention provides an intelligent data processing device based on a large model, a medium and electronic equipment, and belongs to the technical field of data processing. The target module is used for determining a preset large model target and designing an overall framework of a large model; the data acquisition module is used for acquiring original data through a preset data source and storing the original data into an original database; the setting module is used for setting the calculation unit to match different processing paths according to types and processing requirements of the standard data after the standard data is obtained through preprocessing; the data processing module is used for transmitting the standard data to a corresponding processing path based on a high-speed transmission channel based on a calculation unit, performing parallel compression processing to obtain standard intelligent data, and storing the standard intelligent data in a data lake; and the integration module is used for integrating the data acquisition module, the setting module and the data processing module into an overall architecture of a large model, configuring the data acquisition module, the setting module and the data processing module to a test environment for test verification to obtain feedback information, and adjusting a data processing flow and a calculation strategy according to the feedback information.

Description

Technical Field

[0001] The present invention relates to the field of data processing technology, and in particular to an intelligent data processing device, medium and electronic equipment based on a large model. Background Art

[0002] At present, with the development of technologies such as the Internet, Internet of Things, 5G, and artificial intelligence, the amount of data is growing explosively. This makes the storage, processing, and analysis of data more complex and important. In order to extract valuable information from huge amounts of data, it is necessary to rely on powerful data processing capabilities and intelligent analysis methods; usually, a simple single-threaded data processing process is too cumbersome and inefficient.

[0003] Therefore, the present invention proposes an intelligent data processing device, medium and electronic device based on a large model. Summary of the invention

[0004] The present invention provides an intelligent data processing device, medium and electronic device based on a large model, which is used through five modules: the target module determines the large model target and designs the overall architecture; the data acquisition module acquires and stores the original data through the data source; the setting module performs the calculation unit setting and data preprocessing; the data processing module performs data transmission, parallel processing and storage in the data lake based on the calculation unit; the integration module integrates each module and performs test verification, and adjusts the data processing process and calculation strategy according to feedback. The overall solution aims to optimize the data processing process and improve the efficiency of intelligent data processing.

[0005] In one aspect, the present invention provides an intelligent data processing device, a medium and an electronic device based on a large model, comprising: Target module: Determine the preset large model target according to the processing requirements of intelligent data and design the overall architecture of the large model; Data acquisition module: obtains raw data through the preset data source and stores the raw data in the original database; Setting module: Setting the calculation unit. At the same time, after preprocessing the original data to obtain standard data, different processing paths are matched according to the type and processing requirements of the standard data. Data processing module: Based on the computing unit, standard data is transmitted to the corresponding processing path based on the high-speed transmission channel, and standard intelligent data is obtained after parallel compression processing and stored in the data lake; Integration module: Integrate the data acquisition module, setting module and data processing module into the overall architecture of the large model, and configure them to the test environment for testing and verification to obtain feedback information, and adjust the data processing flow and calculation strategy based on the feedback information.

[0006] On the other hand, the target module comprises: Demand determination unit: defines the application scenarios of intelligent data processing services and determines the processing requirements of processing devices; Task type unit: Based on the processing requirements of the processing device, clarify the intelligent data processing task type and preset the large model target according to the task type.

[0007] On the other hand, the target module further includes: Architecture unit: According to the task type, match the model type and its architecture to generate the overall architecture of the large model; Hardware acceleration unit: By configuring preset hardware, it serves as the external hardware part of the large model to perform parallel calculations on the large model.

[0008] On the other hand, the data acquisition module includes: Data source unit: determines the data source according to the preset source of intelligent data, and obtains the connection information of all data sources; Interface unit: configures the data interface according to the data type of any data source, and connects through HTTP requests based on the connection information to obtain the original data of the data source; Data storage unit: Design the original database table structure according to the preset structure, and store the original data in the original database according to the joint primary key of timestamp-data source identifier.

[0009] On the other hand, the setting module includes: Computing unit configuration: According to the amount of original data and processing requirements, multiple computing units are set up for parallel data processing; Standardization unit: obtain the original data of any data source of the original database, clean the original data, remove and replace the outliers to obtain the first data; The first data is normalized to obtain standard data, specifically: ;in, represents the standard data at the i-th time node in the first data, Represents the data value at the i-th time node in the first data, represents the mean of the data values ​​at all time nodes in the first data, represents the standard deviation of the data values ​​at all time nodes in the first data, and n represents that there are n time nodes in the first data. Represents the minimum value of the data values ​​at all time nodes in the first data. Indicates the maximum value of the data values ​​at all time nodes in the first data.

[0010] On the other hand, the setting module further includes: Classification unit: the number of classifications is determined according to the number of calculation units, and feature engineering is performed on the raw data of any data source to obtain a first feature sequence of the raw data; If the number of sequences of the first feature sequence is greater than the number of the proposed classifications, the first feature sequence is subjected to principal component analysis and dimensionality reduction to obtain a second feature sequence that is consistent with the number of the proposed classifications; otherwise, the first feature sequence is directly used as the second feature sequence; Any feature of the second feature sequence is used as a classification standard and a calculation unit for matching to obtain a desired unit, and the standard data is subjected to feature selection, and the feature with the highest expected value is selected as the classification, and is allocated to the corresponding desired unit; The required unit selects the processing path of the required unit according to the data type-processing requirement-processing path.

[0011] On the other hand, the data processing module includes: Serialization unit: According to the standard data classified under any classification calculation unit, the standard data is serialized to obtain byte stream data, and the byte stream data is transmitted to the high-speed transmission channel through the network transmission protocol according to the processing path of the required unit. The transmission rate is: ; Wherein, R represents the transmission rate of the required unit high-speed transmission, represents the bandwidth of the high-speed transmission channel, represents the delay of the high-speed transmission channel, Represents byte stream data, represents the high-speed processing efficiency function; If the transmission rate is less than a preset threshold, the high-speed transmission channel is reallocated, otherwise normal transmission is performed; Task distribution unit: Through the high-speed transmission channel, the scheduler distributes data according to the type and demand of the data, and builds and distributes data processing tasks according to the load balancing algorithm to obtain a multi-task queue; For the multi-task queue, the system performs parallel processing to obtain first intelligent data after processing; Compression unit: encodes and compresses the first intelligent data, constructs a frequency table according to the symbols of the first intelligent data, merges the two symbols with the lowest frequency to form a new node, until the frequency tree is constructed, generates a binary code for each symbol according to the frequency tree, and obtains the standard intelligent data of the first intelligent data, which is specifically: ;in, represents the compression code of the jth symbol, Indicates the jth symbol The Huffman value of {} represents the compression function, It means that the first intelligent data has a total of m symbols. express The size of the interval from the starting point of the frequency tree, express The size of the interval from the end point in the frequency tree; Data Lake Unit: Build a data lake for intelligent data and transfer standard intelligent data from all data sources into the data lake.

[0012] On the other hand, the integrated module comprises: Integration unit: seamlessly integrate the data acquisition module, setting module, and data processing module, formulate interfaces between modules according to data flow and communication protocol, and complete preliminary integration; Perform integration testing on the preliminary integration. When each module meets the expected standards, complete the integration and obtain the large model. Otherwise, reconfigure the interfaces between the modules. Test unit: build the test environment, configure the large model into the test environment, monitor the performance of each module and the structure of intelligent data processing in real time, and generate test logs; Feedback unit: Obtain feedback information on the operation of the large model based on the test log, and adjust the data processing flow and calculation strategy according to the feedback information.

[0013] Compared with the prior art, the present invention has the following beneficial effects: The present invention provides an intelligent data processing device, medium and electronic device based on a large model, which is used through five modules: the target module determines the large model target and designs the overall architecture; the data acquisition module acquires and stores the original data through the data source; the setting module performs the calculation unit setting and data preprocessing; the data processing module performs data transmission, parallel processing and storage in the data lake based on the calculation unit; the integration module integrates each module and performs test verification, and adjusts the data processing process and calculation strategy according to feedback. The overall solution aims to optimize the data processing process and improve the efficiency of intelligent data processing. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0015] Figure 1 It is a structural diagram of an intelligent data processing device based on a large model provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0016] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0017] Embodiment 1: like Figure 1 As shown, the intelligent data processing device, medium and electronic device based on the large model provided by the embodiment of the present invention include: Target module: Determine the preset large model target according to the processing requirements of intelligent data and design the overall architecture of the large model; Data acquisition module: obtains raw data through the preset data source and stores the raw data in the original database; Setting module: Setting the calculation unit. At the same time, after preprocessing the original data to obtain standard data, different processing paths are matched according to the type and processing requirements of the standard data. Data processing module: Based on the computing unit, standard data is transmitted to the corresponding processing path based on the high-speed transmission channel, and standard intelligent data is obtained after parallel compression processing and stored in the data lake; Integration module: Integrate the data acquisition module, setting module and data processing module into the overall architecture of the large model, and configure them to the test environment for testing and verification to obtain feedback information, and adjust the data processing flow and calculation strategy based on the feedback information.

[0018] In this embodiment, smart data is data that contains structure and information.

[0019] In this embodiment, processing requirements refer to the processing methods, steps and strategies required for specific tasks or goals during the intelligent data processing process.

[0020] In this embodiment, the preset large model target refers to the actual application needs, technical requirements and system goals during the intelligent data processing and analysis process.

[0021] In this embodiment, the overall architecture is a high-level structure of the entire system, ensuring that each module can work in coordination to achieve the preset goals.

[0022] In this embodiment, the preset data source refers to various data sources that have been determined and configured in the large model architecture design phase and are used to provide raw data.

[0023] In this embodiment, raw data refers to data that has not been processed or cleaned in any way in the data processing system.

[0024] In this embodiment, the original database is a database for storing unprocessed or preliminarily processed data collected from various data sources.

[0025] In this embodiment, the computing unit refers to a software component responsible for executing specific computing tasks in the entire data processing architecture. It performs the required computing operations and performs data processing, conversion and optimization according to different processing requirements and standard data types.

[0026] In this embodiment, preprocessing is an operation of cleaning, formatting, converting and standardizing the raw data.

[0027] In this embodiment, standard data refers to pre-processed data that has been cleaned, converted, normalized and organized according to predetermined rules, formats and requirements.

[0028] In this embodiment, the processing path is a specific process route for performing different types of processing, conversion, analysis and other operations on data.

[0029] In this embodiment, the high-speed transmission channel refers to a communication path technology used to quickly and efficiently transmit data within the system.

[0030] In this embodiment, compression processing is to reduce the volume of data through specific technology, thereby improving storage efficiency and transmission efficiency.

[0031] In this embodiment, standard intelligent data refers to data that has been pre-processed and formatted to meet specific standards and specifications.

[0032] In this embodiment, the data lake is a storage warehouse for storing a large amount of data in a centralized manner.

[0033] In this embodiment, the integration integrates the independently developed modules according to the design architecture so that they can smoothly carry out data flow and information exchange.

[0034] In this embodiment, the test environment refers to an environment used to verify and test whether a system, module or function works as expected during the development process.

[0035] In this embodiment, feedback information refers to information generated by the system during the test process based on the execution results of the data processing module when the large model is run in the test environment.

[0036] In this embodiment, the computing strategy refers to how to select and execute different algorithms, computing methods and optimization means according to different data types, processing requirements and computing resource constraints during the data processing process to complete the data processing task efficiently and accurately.

[0037] The working principle and beneficial effects of the above technical solution are: through modular design, the acquisition, preprocessing, processing and storage process of intelligent data is optimized, and the data processing efficiency is improved. The integrated module realizes the coordination and optimization of the overall architecture, ensures the efficiency and flexibility of data processing, and helps to improve the performance of the intelligent system.

[0038] Embodiment 2: Based on the above embodiment 1, the target module includes: Demand determination unit: defines the application scenarios of intelligent data processing services and determines the processing requirements of processing devices; Task type unit: Based on the processing requirements of the processing device, clarify the intelligent data processing task type and preset the large model target according to the task type.

[0039] In this embodiment, the application scenario refers to the actual usage background and demand of the intelligent data processing service in a specific environment.

[0040] The working principle and beneficial effects of the above technical solution are: the application scenario and processing requirements are clarified through the demand determination unit, the task type unit defines the specific task type according to the requirements, and the large model target is preset, so as to achieve accurate target setting and task classification for intelligent data processing. This method improves the pertinence and processing efficiency of the system, and helps to optimize and refine intelligent data processing services.

[0041] Embodiment 3: Based on the above embodiment 1, the target module further includes: Architecture unit: According to the task type, match the model type and its architecture to generate the overall architecture of the large model; Hardware acceleration unit: By configuring preset hardware, it serves as the external hardware part of the large model to perform parallel calculations on the large model.

[0042] In this embodiment, the preset hardware refers to a hardware device specially configured to accelerate large model calculations, such as a CPU or a GPU.

[0043] In this embodiment, parallel processing refers to a process of decomposing a computing task into multiple subtasks and executing these subtasks simultaneously.

[0044] The working principle and beneficial effects of the above technical solution are: the architecture unit matches the appropriate model architecture according to the task type to ensure the adaptability and efficiency of large model design. The hardware acceleration unit uses external hardware for parallel computing, which improves the computing efficiency and processing speed of the model and helps optimize the performance of intelligent data processing.

[0045] Embodiment 4: Based on the above embodiment 1, the data acquisition module includes: Data source unit: determines the data source according to the preset source of intelligent data, and obtains the connection information of all data sources; Interface unit: configures the data interface according to the data type of any data source, and connects through HTTP requests based on the connection information to obtain the original data of the data source; Data storage unit: Design the original database table structure according to the preset structure, and store the original data in the original database according to the joint primary key of timestamp-data source identifier.

[0046] In this embodiment, the preset source refers to a data source that has been defined in the system design phase.

[0047] In this embodiment, the connection information refers to all detailed information required when interacting with a data source, including UPL, IP address, protocol, etc.

[0048] In this embodiment, the data interface refers to a communication method for exchanging data between different systems, applications or components.

[0049] In this embodiment, the HTTP request is a way for the client to send a request to the server to obtain resources.

[0050] In this embodiment, the preset structure refers to the database table structure and data storage method predefined in the system design stage.

[0051] In this embodiment, the composite primary key refers to a primary key composed of multiple columns (fields) and is used to uniquely identify each row of data in the table.

[0052] The working principle and beneficial effects of the above technical solution are: determine the data source through the data source unit, and configure the data interface through the interface unit to obtain the original data. The data storage unit designs a suitable database structure to ensure efficient storage and management of data, improve the accuracy of data processing and the stability of the system.

[0053] Embodiment 5: Based on the above embodiment 1, the setting module includes: Computing unit configuration: According to the amount of original data and processing requirements, multiple computing units are set up for parallel data processing; Standardization unit: obtain the original data of any data source of the original database, clean the original data, remove and replace the outliers to obtain the first data; The first data is normalized to obtain standard data, specifically: ;in, represents the standard data at the i-th time node in the first data, Represents the data value at the i-th time node in the first data, Represents the mean of the data values ​​at all time nodes in the first data, represents the standard deviation of the data values ​​at all time nodes in the first data, n represents that there are n time nodes in the first data, Represents the minimum value of the data values ​​at all time nodes in the first data. Represents the maximum value of the data values ​​at all time nodes in the first data.

[0054] In this embodiment, the data volume refers to the amount of original data involved in the processing process.

[0055] In this embodiment, data cleaning improves the quality and applicability of the data by identifying and correcting errors, inconsistencies, or missing values ​​in the data.

[0056] In this embodiment, an outlier refers to data that deviates significantly from other data points in a data set.

[0057] In this embodiment, the first data refers to the original data after data cleaning and removal of outliers.

[0058] In this embodiment, standard normalization converts data of different dimensions or scales into a unified standard scale.

[0059] The working principle and beneficial effects of the above technical solution are: by configuring multiple computing units to process data in parallel, the data processing efficiency is improved. The standardization unit removes outliers and standardizes data through data cleaning and normalization, ensuring the accuracy and consistency of the data, and providing a high-quality data foundation for subsequent analysis.

[0060] Embodiment 6: Based on the above embodiment 5, the setting module further includes: Classification unit: the number of classifications is determined according to the number of calculation units, and feature engineering is performed on the raw data of any data source to obtain a first feature sequence of the raw data; If the number of sequences of the first feature sequence is greater than the number of the proposed classifications, the first feature sequence is subjected to principal component analysis and dimensionality reduction to obtain a second feature sequence that is consistent with the number of the proposed classifications; otherwise, the first feature sequence is directly used as the second feature sequence; Any feature of the second feature sequence is used as a classification standard and a calculation unit for matching to obtain a desired unit, and the standard data is subjected to feature selection, and the feature with the highest expected value is selected as the classification, and is allocated to the corresponding desired unit; The required unit selects the processing path of the required unit according to the data type-processing requirement-processing path.

[0061] In this embodiment, feature engineering refers to processing and transforming raw data to generate useful features to improve model performance.

[0062] In this embodiment, the first feature sequence refers to a set of features obtained by performing feature engineering processing on the original data.

[0063] In this embodiment, the principal component analysis dimensionality reduction operation maps the data from a high-dimensional space to a low-dimensional space while retaining the original feature information of the data as much as possible. Through PCA (component analysis), the complexity of the data can be reduced.

[0064] In this embodiment, feature selection refers to selecting the most important and most predictive features from the features of the original data during the machine learning process.

[0065] In this embodiment, the expected value is an indicator indicating the degree of matching in feature selection.

[0066] The working principle and beneficial effects of the above technical solution are: through feature engineering processing and principal component analysis dimensionality reduction, data features are optimized to ensure that the number of classifications is consistent with the calculation unit. Through feature selection and matching, data is ensured to be allocated to appropriate units, and the best processing path is selected according to processing requirements, thereby improving the efficiency and accuracy of data classification and processing.

[0067] Embodiment 7: Based on the above embodiment 6, the data processing module includes: Serialization unit: According to the standard data classified under any classification calculation unit, the standard data is serialized to obtain byte stream data, and the byte stream data is transmitted to the high-speed transmission channel through the network transmission protocol according to the processing path of the required unit. The transmission rate is: ; Wherein, R represents the transmission rate of the required unit high-speed transmission, represents the bandwidth of the high-speed transmission channel, represents the delay of the high-speed transmission channel, Represents byte stream data, represents the high-speed processing efficiency function; If the transmission rate is less than a preset threshold, the high-speed transmission channel is reallocated, otherwise normal transmission is performed; Task distribution unit: Through the high-speed transmission channel, the scheduler distributes data according to the type and demand of the data, and builds and distributes data processing tasks according to the load balancing algorithm to obtain a multi-task queue; For the multi-task queue, the system performs parallel processing to obtain first intelligent data after processing; Compression unit: encodes and compresses the first intelligent data, constructs a frequency table according to the symbols of the first intelligent data, merges the two symbols with the lowest frequency to form a new node, until the frequency tree is constructed, generates a binary code for each symbol according to the frequency tree, and obtains the standard intelligent data of the first intelligent data, which is specifically: ;in, represents the compression code of the jth symbol, Indicates the jth symbol The Huffman value of {} represents the compression function, It means that the first intelligent data has a total of m symbols. express The size of the interval from the starting point of the frequency tree, express The size of the interval from the end point in the frequency tree; Data Lake Unit: Build a data lake for intelligent data and transfer standard intelligent data from all data sources into the data lake.

[0068] In this embodiment, serialization processing is to convert data from an object form in memory into a format that can be stored or transmitted, so as to facilitate reconstruction or reading in different systems or environments.

[0069] In this embodiment, byte stream data refers to a data stream stored continuously in bytes.

[0070] In this embodiment, the network transmission protocol is a rule and standard for regulating the data exchange process in a computer network.

[0071] In this embodiment, the transmission rate refers to the speed at which byte stream data is transmitted through a network transmission channel.

[0072] In this embodiment, bandwidth refers to the maximum data transmission capacity of a high-speed transmission channel.

[0073] In this embodiment, the high-speed processing efficiency function refers to a function of factors affecting data processing speed and efficiency.

[0074] In this embodiment, the preset threshold refers to a defined standard used to determine whether the transmission rate of the high-speed transmission channel meets the standard.

[0075] In this embodiment, the scheduler reasonably allocates data processing tasks according to the type and demand of data.

[0076] In this embodiment, the load balancing algorithm is a technology that evenly distributes computing task data traffic to multiple processing units.

[0077] In this embodiment, the multi-task queue is a data structure for storing and managing multiple tasks to be processed.

[0078] In this embodiment, the first intelligent data refers to the preliminary processing results obtained after the task distribution unit (scheduler) completes the task allocation and performs parallel processing.

[0079] In this embodiment, the coding compression converts the original data into a more compact form that occupies less space through a certain coding algorithm.

[0080] In this embodiment, the frequency table is used to record the occurrence frequency of each symbol (such as a character, a byte, etc.) in a data set.

[0081] In this embodiment, the frequency tree refers to a tree structure, which is constructed according to frequency information of different symbols (such as bytes or characters).

[0082] In this embodiment, standard intelligent data refers to data that has been preprocessed, compressed, encoded, and the like.

[0083] In this embodiment, the Huffman value refers to a binary code value generated by the Huffman coding algorithm for each symbol.

[0084] The working principle and beneficial effects of the above technical solution are: optimize data processing through serialization, compression, task distribution and data lake management. Use load balancing and parallel computing to improve processing efficiency, and use Huffman coding to compress intelligent data, build a data lake to centrally manage standard data, and improve data transmission and storage efficiency.

[0085] Embodiment 8: Based on the above embodiment 1, the integrated module includes: Integration unit: seamlessly integrate the data acquisition module, setting module, and data processing module, formulate interfaces between modules according to data flow and communication protocol, and complete preliminary integration; Perform integration testing on the preliminary integration. When each module meets the expected standards, complete the integration and obtain the large model. Otherwise, reconfigure the interfaces between the modules. Test unit: build the test environment, configure the large model into the test environment, monitor the performance of each module and the structure of intelligent data processing in real time, and generate test logs; Feedback unit: Obtain feedback information on the operation of the large model based on the test log, and adjust the data processing flow and calculation strategy according to the feedback information.

[0086] In this embodiment, preliminary integration refers to combining and configuring various independent modules or subsystems according to a predetermined design during the system development process to form a preliminary, executable overall system.

[0087] In this embodiment, integration testing is to verify whether multiple modules can work together normally as expected after integration, ensuring that different parts of the system can interact effectively.

[0088] In this embodiment, the test log refers to a record file generated during the software testing process, which contains detailed information about the test execution.

[0089] The working principle and beneficial effects of the above technical solution are: by seamlessly integrating data acquisition, setting and processing modules, integrated testing is performed to ensure the normal function of the modules; by adjusting the data processing flow through real-time monitoring and feedback, the performance and calculation strategy of large models are improved, and the overall system efficiency and reliability are optimized.

[0090] Embodiment 9: A computer-readable storage medium stores a computer program, which, when executed by a processor, causes the processor to execute the logical process of the large model-based intelligent data processing device according to any one of the above.

[0091] The working principle and beneficial effects of the above technical solution are: by storing the computer program in a readable storage medium, the processor executes the intelligent data processing logic based on the large model, automates the data processing process, improves computing efficiency and intelligence level, and optimizes system performance.

[0092] Embodiment 10: An electronic device comprises a memory and a processor, wherein the memory stores a computer program, and when the computer program is executed by the processor, the processor executes the logical process of any one of the large model-based intelligent data processing devices.

[0093] The working principle and beneficial effects of the above technical solution are: by storing the computer program in the memory, the processor executes the intelligent data processing logic based on the large model, realizes automated intelligent data processing, improves equipment performance and processing efficiency, and optimizes the data processing process.

[0094] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. An intelligent data processing device based on a large model, characterized in that: include: Target module: Determine the preset large model target according to the processing requirements of intelligent data and design the overall architecture of the large model; Data acquisition module: obtains raw data through the preset data source and stores the raw data in the original database; Setting module: Setting the calculation unit. At the same time, after preprocessing the original data to obtain standard data, different processing paths are matched according to the type and processing requirements of the standard data. Data processing module: Based on the computing unit, standard data is transmitted to the corresponding processing path based on the high-speed transmission channel, and standard intelligent data is obtained after parallel compression processing and stored in the data lake; Integration module: Integrate the data acquisition module, setting module and data processing module into the overall architecture of the large model, and configure them to the test environment for testing and verification to obtain feedback information, and adjust the data processing flow and calculation strategy based on the feedback information.

2. The large model-based intelligent data processing device according to claim 1, characterized in that: The target module comprises: Demand determination unit: defines the application scenarios of intelligent data processing services and determines the processing requirements of processing devices; Task type unit: Based on the processing requirements of the processing device, clarify the intelligent data processing task type and preset the large model target according to the task type.

3. The large model-based intelligent data processing device according to claim 1, characterized in that: The target module further includes: Architecture unit: According to the task type, match the model type and its architecture to generate the overall architecture of the large model; Hardware acceleration unit: By configuring preset hardware, it serves as the external hardware part of the large model to perform parallel calculations on the large model.

4. The large model-based intelligent data processing device according to claim 1, characterized in that: The data acquisition module comprises: Data source unit: determines the data source according to the preset source of intelligent data, and obtains the connection information of all data sources; Interface unit: configures the data interface according to the data type of any data source, and connects through HTTP request based on the connection information to obtain the original data of the data source; Data storage unit: Design the original database table structure according to the preset structure, and store the original data in the original database according to the joint primary key of timestamp-data source identifier.

5. The large model-based intelligent data processing device according to claim 1, characterized in that: The setting module includes: Computing unit configuration: According to the amount of original data and processing requirements, multiple computing units are set up for parallel data processing; Standardization unit: obtain the original data of any data source of the original database, clean the original data, remove and replace the outliers to obtain the first data; The first data is normalized to obtain standard data, specifically: ;in, represents the standard data at the i-th time node in the first data, Represents the data value at the i-th time node in the first data, represents the mean of the data values ​​at all time nodes in the first data, represents the standard deviation of the data values ​​at all time nodes in the first data, and n represents that there are n time nodes in the first data. Represents the minimum value of the data values ​​at all time nodes in the first data. Indicates the maximum value of the data values ​​at all time nodes in the first data.

6. The large model-based intelligent data processing device according to claim 5, characterized in that: The setting module further includes: Classification unit: the number of classifications is determined according to the number of calculation units, and feature engineering is performed on the raw data of any data source to obtain a first feature sequence of the raw data; If the number of sequences of the first feature sequence is greater than the number of the proposed classifications, the first feature sequence is subjected to principal component analysis and dimensionality reduction to obtain a second feature sequence that is consistent with the number of the proposed classifications; otherwise, the first feature sequence is directly used as the second feature sequence; Any feature of the second feature sequence is used as a classification standard and a calculation unit for matching to obtain a desired unit, and the standard data is subjected to feature selection, and the feature with the highest expected value is selected as the classification, and is allocated to the corresponding desired unit; The required unit selects the processing path of the required unit according to the data type-processing requirement-processing path.

7. The large model-based intelligent data processing device according to claim 6, characterized in that: The data processing module comprises: Serialization unit: According to the standard data classified under any classification calculation unit, the standard data is serialized to obtain byte stream data, and the byte stream data is transmitted to the high-speed transmission channel through the network transmission protocol according to the processing path of the required unit. The transmission rate is: ; Wherein, R represents the transmission rate of the required unit high-speed transmission, represents the bandwidth of the high-speed transmission channel, represents the delay of the high-speed transmission channel, Represents byte stream data, represents the high-speed processing efficiency function; If the transmission rate is less than a preset threshold, the high-speed transmission channel is reallocated, otherwise normal transmission is performed; Task distribution unit: Through the high-speed transmission channel, the scheduler distributes data according to the type and demand of the data, and builds and distributes data processing tasks according to the load balancing algorithm to obtain a multi-task queue; For the multi-task queue, the system performs parallel processing to obtain first intelligent data after processing; Compression unit: encodes and compresses the first intelligent data, constructs a frequency table according to the symbols of the first intelligent data, merges the two symbols with the lowest frequency to form a new node, until the frequency tree is completed, generates a binary code for each symbol according to the frequency tree, and obtains the standard intelligent data of the first intelligent data, which is specifically: ;in, represents the compression code of the jth symbol, Indicates the jth symbol The Huffman value of {} represents the compression function, It means that the first intelligent data has a total of m symbols. express The size of the interval from the starting point of the frequency tree, express The size of the interval from the end point in the frequency tree; Data Lake Unit: Build a data lake for intelligent data and transfer standard intelligent data from all data sources into the data lake.

8. The large model-based intelligent data processing device according to claim 1, characterized in that: The integrated module comprises: Integration unit: seamlessly integrate the data acquisition module, setting module, and data processing module, formulate interfaces between modules according to data flow and communication protocol, and complete preliminary integration; Perform integration testing on the preliminary integration. When each module meets the expected standards, complete the integration and obtain the large model. Otherwise, reconfigure the interfaces between the modules. Test unit: build the test environment, configure the large model into the test environment, monitor the performance of each module and the structure of intelligent data processing in real time, and generate test logs; Feedback unit: Obtain feedback information on the operation of the large model based on the test log, and adjust the data processing flow and calculation strategy according to the feedback information.

Images