Analyzing and processing system based on production scene data
By designing an analysis and processing system based on production scenario data, the data island problem is solved, cross-departmental integration and secure sharing of data is realized, data processing efficiency and security are improved, and data processing is provided with a rich data source for big data analysis.
Patent Information
- Application Number
- CN202411491133.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-10-24
- Publication Date
- 2024-11-26
AI Technical Summary
In the prior art, data from various departments cannot be jointly analyzed and processed, resulting in data silos, and data cannot be integrated and shared across departments and systems.
Analytical and processing system based on production scenario data was designed, including data collection module, data lake construction module and library design module. Through data collection, classification, encrypted storage and unified standardized processing of departmental data, OLTP and OLAP architectures are built, and utilizing Blockchain technology and neural network models perform data matching and analysis processing.
It realizes cross-departmental integration of data, improves data integrity and consistency, enhances data security and access controllability, improves data processing efficiency and accuracy, and forms a comprehensive and secure data repository, which is a large-scale Data analysis provides a rich source of data.
Smart Images

Figure CN119026174A_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the field of data processing and relates to data lake technology, specifically to an analysis and processing system based on production scenario data. Background Art
[0002] In the process of enterprise development, when informatization develops to a certain stage, multiple departments will appear, each with its own data. The data between departments are often stored and defined separately; the data of each department is like an isolated island, which is impossible or extremely difficult to connect and interact with other data within the enterprise. This situation is called a data island.
[0003] An invention patent application with publication number CN116680188A discloses a data processing method, apparatus, device and computer-readable storage medium. The invention collects a production data set under preset conditions, generates a data change trend report for a target scenario according to the business parameter type in the production data set, constructs a test data call template for the target scenario according to the data change trend report, calls the test data set for the target scenario from a target database, and performs a performance stress test on the target scenario based on the test data set. When analyzing the production data set, the invention does not jointly process the production data of multiple departments, which may result in a lack of connectivity among departments when performing production tasks, resulting in the inability to share and communicate data between business departments, and the inability to provide sufficient data support, resulting in data islands.
[0004] The present invention provides a heating pipeline leakage position detection method based on smart ball and sound wave positioning to solve the above technical problems. Summary of the invention
[0005] The present invention aims to solve at least one of the technical problems existing in the prior art; to this end, the present invention proposes an analysis and processing system based on production scenario data, which is used to solve the technical problem of data islands caused by the lack of joint analysis and processing of data from various departments in the prior art.
[0006] To achieve the above-mentioned purpose, the first aspect of the present invention provides an analysis and processing system based on production scenario data, including: a data collection module, a data lake construction module and a combined database design module; Data collection module: used to collect scene data, department data and production data; Data lake construction module: used to classify and combine department data to obtain a data lake; Combined database design module: used to build OLTP architecture and OLAP architecture according to scenario data; match the OLTP architecture and OLAP architecture with the data lake to obtain the database, and analyze and process the production data by using the database.
[0007] Preferably, the step of classifying and merging department data into tables to obtain a data lake includes: Extracting department data; the department data includes work content and work type; Set unified data standards; divide department data into business data and system data according to work types; divide the work content of system data and business data into several system sub-topic tables and several business sub-topics based on data standards, encrypt and store several system sub-topic tables and several business sub-topics, and then combine them to obtain a data lake.
[0008] Preferably, the work contents of system data and business data are divided into a plurality of system sub-topic tables and a plurality of business sub-topic tables based on data standards, including: Extracting system data, business data and data standards; the data standards include standard coding rules, standard data formats and standard data dictionaries; Standardize the format of system data and business data according to the standard data format, standardize the data naming, data type and data length; standardize the coding of all system data and business data; use the standard data dictionary as the specification of data standards and as a reference for correction; Identify the themes of work content in system data and business data, and summarize the data of each theme to obtain system sub-theme tables and business sub-theme tables.
[0009] Preferably, the step of encrypting and storing a plurality of system sub-topic tables and a plurality of business sub-topic tables and then combining them to obtain a data lake includes: Extract system sub-topic table and business sub-topic table; Preprocess the system sub-topic table and the business sub-topic table to obtain the standard system topic table and the standard business topic table; establish different permission modes and open different permission modes to staff of different work types; filter out high-security data from the standard system topic table and the standard business topic table, save and encrypt the high-security data through blockchain technology, and load the encrypted data into the storage system; Build an initial data lake architecture, integrate the encrypted and stored system sub-topic tables and business sub-topic tables into the initial data lake architecture through the data processing platform, and record data access records and operation records in real time.
[0010] Preferably, the preprocessing of the system sub-topic table and the business sub-topic table includes: Extract system sub-topic table and business sub-topic table; The system sub-theme table and business sub-theme table are cleaned and duplicate and abnormal items are removed in turn. The missing items in the system sub-theme table and business sub-theme table with missing values are supplemented, and the consistency and completeness of the data are checked according to the data dictionary.
[0011] Preferably, the step of constructing the OLTP architecture and the OLAP architecture respectively according to the scenario data includes: A1: extracting scenario data; the scenario data includes work time, work data volume and operation type; A2: Setting a time threshold and a data volume threshold according to the operation type; the operation type includes business operation and background operation; A3: Determine whether the working time is greater than or equal to the time threshold; if yes, jump to A4; if no, mark the corresponding scene data as immediate data; A4: Determine whether the amount of working data is greater than or equal to the data amount threshold; if yes, mark the corresponding scene data as deferred data; if no, mark the corresponding scene data as immediate data; An initial OLTP architecture and an initial OLAP architecture are constructed, immediate data is input into the initial OLTP architecture to obtain an OLTP architecture, and delayed data is input into the initial OLAP architecture to obtain an OLAP architecture.
[0012] Preferably, the setting of the timeliness threshold and the data volume threshold according to the operation type includes: Extract operation type; The operation type is input into the threshold model to obtain the time threshold and the data volume threshold; wherein the threshold model is constructed based on the neural network model.
[0013] Preferably, the threshold model is constructed based on a neural network model, including: Historical operation types and corresponding historical timeliness thresholds and historical data volume thresholds are extracted from the database, and the historical operation types and the corresponding historical timeliness thresholds and historical data volume thresholds are used as training data and test data. The neural network model is trained using the training data, and the trained neural network model is tested using the test data. According to the test results, the parameters of the neural network model are adjusted to obtain a threshold model whose input data is the operation type and whose output data is the corresponding timeliness threshold and data volume threshold.
[0014] Preferably, the obtaining and using database based on the matching of OLTP architecture and OLAP architecture with the data lake includes: B1: Match data that conforms to the OLTP architecture and OLAP architecture in the data lake; B2: Determine whether the system sub-topic table and the business sub-topic table conform to the OLTP architecture; if yes, mark the corresponding system sub-topic table and the business sub-topic table as TP matching tables; if no, jump to B3; B3: Determine whether the system sub-theme table and the business sub-theme table conform to the OLAP architecture; if yes, mark the corresponding system sub-theme table and the business sub-theme table as AP matching tables; if no, mark the corresponding system sub-theme table and the business sub-theme table as non-matching tables; Building a high-performance RDS cluster, the RDS cluster includes a master node and a slave node; The TP matching table and the AP matching table are transmitted through the master node and the slave node and combined to obtain a usage database.
[0015] Preferably, the analysis and processing of production data by using a database includes: Extracting production data; the production data includes current work content and current work type; The production data is input into the database for matching, divided into current sub-topic tables, and then the current sub-topic tables are transferred to the data lake through the RDS cluster; Determine whether the current sub-topic table conforms to the OLTP architecture and OLAP architecture; if yes, input the current sub-topic table into the corresponding architecture and transmit it to the corresponding work department; if no, manually analyze the current sub-topic table to obtain the analysis results, and update the OLTP architecture and OLAP architecture in the data lake according to the analysis results.
[0016] Compared with the prior art, the present invention has the following beneficial effects: 1. The present invention collects scenario data, department data and production data through a data collection module, clearly distinguishes the data into business data and system data, and further divides the data into multiple sub-topics, so that the data structure is clearer and convenient for on-demand retrieval and analysis, and combines tables to build a data lake. Through unified data standards and classifications, cross-departmental and cross-system data integration is achieved, the integrity and consistency of the data are improved, and blockchain technology is used to encrypt high-security data to ensure the security of sensitive information during transmission and storage, and prevent data leakage; OLTP architecture and OLAP architecture are constructed according to different scenario data, which can handle work tasks with different needs respectively, and by establishing different permission modes, the access to data by personnel of different work types is restricted, thereby enhancing the controllability and security of data access, and integrating the encrypted system sub-topic table and business sub-topic table into the data lake to form a comprehensive and secure data repository, providing a rich data source for big data analysis.
[0017] 2. The present invention divides and merges the overall data of the enterprise by combining libraries and tables, extracts relevant topics, analyzes different scenario data, and quickly determines whether the scenario data is delayed data or immediate data through a neural network model, thereby improving the efficiency and accuracy of data processing. The delayed data and immediate data are input into the initial scenario architecture, and the scenario architecture is improved so that it can quickly match the normal work of the enterprise, effectively improving the manageability and analyzability of the data. The system can more intelligently match the normal work flow of the enterprise, improve the speed and accuracy of business processing, and can also better control data access rights through reasonable data division and merging of libraries. Combined with the intelligent classification of the neural network model, it helps to enhance data security and privacy protection. BRIEF DESCRIPTION OF THE DRAWINGS
[0018] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.
[0019] Figure 1 The following is a flowchart of an embodiment of the present invention.
[0020] Figure 2 It is a system composition diagram of the present invention.
[0021] Figure 3 This is a structural diagram of an integrated data lake in one embodiment of the present invention. DETAILED DESCRIPTION
[0022] The technical solution of the present invention will be clearly and completely described below in conjunction with the embodiments. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0023] See also Figure 1-Figure 3 , the first aspect of the present invention provides an analysis and processing system based on production scenario data, including: a data collection module, a data lake construction module and a combined database design module; Data collection module: used to collect scene data, department data and production data.
[0024] For example, the time required to complete each task, the amount of work data and the type of operation are collected through the network platform, and the work content and work type of each department of the enterprise, as well as the current work content and current work type of the tasks that need to be completed are collected.
[0025] In this embodiment, the enterprise includes a sales department, a production department, a quality inspection department, and a finance department; the work content of the sales department includes market analysis, sales execution, and customer maintenance and development, and the work type is business work; the work content of the production department includes production plan formulation, production execution, and material management, and the work type is business work; the work content of the quality inspection department includes quality control and rework of defective products, and the work type is system work; the work content of the finance department includes preparation of financial statements, asset management, and salary statistics, and the work type is system work.
[0026] Data lake construction module: used to classify and merge department data into tables to obtain a data lake.
[0027] Exemplarily, the data standards set in this embodiment include data naming, data type, data length, data format, data encoding, etc. All data are modified and integrated according to the set data standards, the encoding of department data is standardized according to the encoding format of the data standard, and a standard data dictionary is set for reference; department data is divided into business data and system data according to the work type, and the business data includes sales department data and production department data, and the system data includes quality inspection department data and finance department data.
[0028] The bag-of-words model is used to convert the topics of work content in business data and system data into numerical features. The numerical features are identified through a topic model algorithm (such as LDA), and the work content of the same topic in business data and system data is combined to obtain a system sub-topic table and a business sub-topic table.
[0029] Carry out data cleaning on the system sub-theme table and the business sub-theme table in turn, detect and eliminate the duplicated data in the data, detect and eliminate the data that obviously does not belong to the corresponding theme, detect whether there are missing items in each system sub-theme table and the business sub-theme table. If there are missing items, supplement them manually; check the consistency and completeness of the system sub-theme table and the business sub-theme table according to the data dictionary to ensure the quality of the data, mark the processed system sub-theme table as the standard system theme table, and mark the processed business sub-theme table as the standard business theme table; establish two permission modes: read-only permission and modifiable permission. The standard system theme table has read-only permission for the staff corresponding to the business data, and modifiable permission for the staff corresponding to the system data; the standard business theme table has modifiable permission for the staff corresponding to the business data, and read-only permission for the staff corresponding to the system data.
[0030] High-security data can be filtered from standard system subject tables and standard business subject tables. The data can be filtered according to the confidentiality involved, and a confidentiality threshold can be set. Data with a confidentiality greater than the confidentiality threshold can be marked as high-security data. The high-security data can be saved and encrypted through blockchain technology and stored in the storage system. An initialized data lake architecture can be constructed, and the encrypted and stored high-security data can be input into the initialized data lake architecture to save and prevent the original records from being modified or deleted, and the access records and operation records of accessing the high-security data can be recorded in real time.
[0031] Extract historical operation types and corresponding historical timeliness thresholds and historical data volume thresholds from the database, use the historical operation types and corresponding historical timeliness thresholds and historical data volume thresholds as training data and test data, use the training data to train the neural network model, use the test data to test the trained neural network model, adjust the parameters of the neural network model according to the test results, and obtain a threshold model whose input data is the operation type and whose output data is the corresponding timeliness threshold and data volume threshold; input the operation type into the threshold model to obtain the timeliness threshold and data volume threshold, and obtain a timeliness threshold of 1 second for business operations and a data volume threshold of 10 megabytes, a timeliness threshold of 2 seconds for background operations and a data volume threshold of 20 megabytes; in this embodiment, the scene data is a foreground operation, the working timeliness is 0.5 seconds, the working data volume is 50 kilobytes, and the operation type is a business operation; determine whether the working timeliness is greater than or equal to the timeliness threshold, and if it is determined that the working timeliness is less than the timeliness threshold, the scene data of the foreground operation is marked as immediate data.
[0032] In some other embodiments, the scenario data is report statistics, the work time is 10 seconds, the work data volume is 75 megabytes, and the operation type is background operation; determine whether the work time is greater than or equal to the time threshold, and if the work time is greater than the time threshold, continue to determine whether the work data volume is greater than or equal to the data volume threshold, and if the work data volume is greater than the data volume threshold, then mark the scenario data of the report statistics as delayed data.
[0033] Construct the initial OLTP architecture and the initial OLAP architecture, input all immediate data into the initial OLTP architecture so that the initial OLTP architecture stores all immediate data, and mark the initial OLTP architecture after the input is completed as the OLTP architecture; input all deferred data into the initial OLAP architecture so that the initial OLAP architecture stores all deferred data, and mark the initial OLAP architecture after the input is completed as the OLAP architecture.
[0034] Determine whether the system sub-topic table and the business sub-topic table comply with the OLTP architecture. If they comply with the OLTP architecture, mark the corresponding system sub-topic table and the business sub-topic table as TP matching tables. If they do not comply with the OLTP architecture, continue to determine whether the system sub-topic table and the business sub-topic table comply with the OLAP architecture. If they comply with the OLAP architecture, mark the corresponding system sub-topic table and the business sub-topic table as AP matching tables. If they do not comply with the OLAP architecture, mark the corresponding system sub-topic table and the business sub-topic table as unmatched tables.
[0035] It should be noted that if an unmatched table appears, it means that necessary data is missing in the unmatched table, and it needs to be supplemented and matched again manually. Under normal circumstances, an unmatched table will not appear.
[0036] Build a high-performance RDS cluster, which includes several master nodes and several slave nodes. The TP matching table and the AP matching table are transmitted through the master node and the slave node and combined to obtain a usage database. The master node has a faster transmission speed. For work tasks with a shorter working time, the master node can be used for transmission. For work tasks with a longer working time and a larger data transmission volume, the master node may not be able to transmit all of them together, and the slave node is required for auxiliary transmission.
[0037] In actual application, the production data of this embodiment is salary payment, the current work content is statistical performance, salary accounting and payment records, and the current work type is system operation; the production data is salary payment input and matched using the database, and the current sub-topic tables are divided into performance table, accounting table and record table; the performance table is written and sent by the staff of each department and summarized to the finance department; the finance department compares and checks the performance table with the accounting table, and pays the salary after the accounting is completed, and records the payment time and amount; the current sub-topic table is transmitted to the data lake through the master node and slave node of the RDS cluster.
[0038] Determine whether the current sub-topic table complies with the OLTP architecture and the OLAP architecture, determine whether the performance table complies with the OLTP architecture, and whether the accounting table and issuance record comply with the OLAP architecture, send the performance table to the staff of each department for filling, send the accounting table to the finance department, and store the issuance record in the storage system.
[0039] In another embodiment, if the current sub-topic table does not conform to the OLTP architecture and the OLAP architecture, the current sub-topic table is manually analyzed to obtain an analysis result, and the OLTP architecture and the OLAP architecture in the data lake are updated according to the analysis result so that they can distinguish the current sub-topic table.
[0040] Part of the data in the above formula is calculated by removing the dimension and taking its numerical value. The formula is a formula closest to the actual situation obtained by software simulation of a large amount of collected data; the preset parameters and preset thresholds in the formula are set by technical personnel in this field according to actual conditions or obtained through simulation of a large amount of data.
[0041] Working principle of the present invention: The present invention collects scenario data, department data and production data of an enterprise, sets data standards for uniformly standardizing the department data, divides the department data into system data and business data by identifying the work type in the department data, identifies the subject of the work content in the system data and business data, divides the system data and business data into system sub-theme tables and business sub-theme tables according to the subject, encrypts and stores the system sub-theme tables and business sub-theme tables, sets different permission modes for different work departments, and builds a data lake containing all department data of the enterprise; builds an OLTP architecture and an OLAP architecture according to the scenario data, and quickly determines the type of scenario data according to the operation type and the neural network model; matches the system sub-theme table and the business sub-theme table with the OLTP architecture and the OLAP architecture to obtain a TP matching table and an AP matching table, and transmits them through a constructed high-performance RDS cluster; matches the current production data with the database, identifies the current sub-theme table in the production data, transmits it to the data lake through the RDS cluster to match it with the OLTP architecture and the OLAP architecture, and distributes and transmits the current sub-theme table that meets the department theme.
[0042] The above embodiments are only used to illustrate the technical method of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical method of the present invention may be modified or replaced by equivalents without departing from the spirit and scope of the technical method of the present invention.
Claims
1. An analysis and processing system based on production scenario data, characterized in that: include: Data collection module, data lake construction module and database design module; Data collection module: used to collect scene data, department data and production data; Data lake construction module: used to classify and combine department data to obtain a data lake; Combined database design module: used to build OLTP architecture and OLAP architecture according to scenario data; Based on the OLTP architecture and OLAP architecture, the database is matched with the data lake to analyze and process the production data.
2. The analysis and processing system based on production scenario data according to claim 1 is characterized in that: The data lake is obtained by classifying and merging department data, including: Extracting department data; the department data includes work content and work type; Set unified data standards; divide department data into business data and system data according to work types; divide the work content of system data and business data into several system sub-topic tables and several business sub-topics based on data standards, encrypt and store several system sub-topic tables and several business sub-topics, and then combine them to obtain a data lake.
3. The analysis and processing system based on production scenario data according to claim 2 is characterized in that: The work content of dividing the system data and business data into a number of system sub-topic tables and a number of business sub-topic tables based on the data standard includes: Extracting system data, business data and data standards; the data standards include standard coding rules, standard data formats and standard data dictionaries; Standardize the format of system data and business data according to the standard data format, standardize the data naming, data type and data length; standardize the coding of all system data and business data; use the standard data dictionary as the specification of data standards and as a reference for correction; Identify the themes of work content in system data and business data, and summarize the data of each theme to obtain system sub-theme tables and business sub-theme tables.
4. The analysis and processing system based on production scenario data according to claim 2 is characterized in that: The encrypted storage of the plurality of system sub-topic tables and the plurality of business sub-topic tables is combined to obtain a data lake, including: Extract system sub-topic table and business sub-topic table; Preprocess the system sub-topic table and the business sub-topic table to obtain the standard system topic table and the standard business topic table; establish different permission modes and open different permission modes to staff of different work types; filter out high-security data from the standard system topic table and the standard business topic table, save and encrypt the high-security data through blockchain technology, and load the encrypted data into the storage system; Build an initial data lake architecture, integrate the encrypted and stored system sub-topic tables and business sub-topic tables into the initial data lake architecture through the data processing platform, and record data access records and operation records in real time.
5. The analysis and processing system based on production scenario data according to claim 4 is characterized in that: The preprocessing of the system sub-topic table and the business sub-topic table includes: Extract system sub-topic table and business sub-topic table; The system sub-theme table and business sub-theme table are cleaned and duplicate and abnormal items are removed in turn. The missing items in the system sub-theme table and business sub-theme table with missing values are supplemented, and the consistency and completeness of the data are checked according to the data dictionary.
6. The analysis and processing system based on production scenario data according to claim 1 is characterized in that: The OLTP architecture and the OLAP architecture are constructed according to the scenario data, including: A1: extracting scenario data; the scenario data includes work time, work data volume and operation type; A2: Setting a time threshold and a data volume threshold according to the operation type; the operation type includes business operation and background operation; A3: Determine whether the working time is greater than or equal to the time threshold; if yes, jump to A4; if no, mark the corresponding scene data as immediate data; A4: Determine whether the amount of working data is greater than or equal to the data amount threshold; if yes, mark the corresponding scene data as deferred data; if no, mark the corresponding scene data as immediate data; An initial OLTP architecture and an initial OLAP architecture are constructed, immediate data is input into the initial OLTP architecture to obtain an OLTP architecture, and delayed data is input into the initial OLAP architecture to obtain an OLAP architecture.
7. The analysis and processing system based on production scenario data according to claim 6 is characterized in that: The setting of the timeliness threshold and the data volume threshold according to the operation type includes: Extract operation type; The operation type is input into the threshold model to obtain the time threshold and the data volume threshold; wherein the threshold model is constructed based on the neural network model.
8. The analysis and processing system based on production scenario data according to claim 7 is characterized in that: The threshold model is constructed based on a neural network model, including: Historical operation types and corresponding historical timeliness thresholds and historical data volume thresholds are extracted from the database, and the historical operation types and the corresponding historical timeliness thresholds and historical data volume thresholds are used as training data and test data. The neural network model is trained using the training data, and the trained neural network model is tested using the test data. According to the test results, the parameters of the neural network model are adjusted to obtain a threshold model whose input data is the operation type and whose output data is the corresponding timeliness threshold and data volume threshold.
9. The analysis and processing system based on production scenario data according to claim 1 is characterized in that: The database is obtained by matching the data lake based on the OLTP architecture and the OLAP architecture, including: B1: Match data that conforms to the OLTP architecture and OLAP architecture in the data lake; B2: Determine whether the system sub-topic table and the business sub-topic table conform to the OLTP architecture; if yes, mark the corresponding system sub-topic table and the business sub-topic table as TP matching tables; if no, jump to B3; B3: Determine whether the system sub-theme table and the business sub-theme table conform to the OLAP architecture; if yes, mark the corresponding system sub-theme table and the business sub-theme table as AP matching tables; if no, mark the corresponding system sub-theme table and the business sub-theme table as non-matching tables; Building a high-performance RDS cluster, the RDS cluster includes a master node and a slave node; The TP matching table and the AP matching table are transmitted through the master node and the slave node and combined to obtain a usage database.
10. The analysis and processing system based on production scenario data according to claim 1, characterized in that: The analysis and processing of production data by using a database includes: Extracting production data; the production data includes current work content and current work type; The production data is input into the database for matching, divided into current sub-topic tables, and then the current sub-topic tables are transferred to the data lake through the RDS cluster; Determine whether the current sub-topic table conforms to the OLTP architecture and OLAP architecture; if yes, input the current sub-topic table into the corresponding architecture and transmit it to the corresponding work department; if no, manually analyze the current sub-topic table to obtain the analysis results, and update the OLTP architecture and OLAP architecture in the data lake according to the analysis results.
Citation Information
Cited By
Semiconductor film diffusion quality tracing method and system based on block chain
CN121012619A
A blockchain-based semiconductor film diffusion quality traceability method and system
CN121012619B
