Data dictionary configuration method and device, medium and equipment

Abstract

The invention provides a data dictionary configuration method and device, a medium and equipment, and the method comprises the steps: obtaining corresponding data based on a plurality of data sources, carrying out the preprocessing of the obtained data, and obtaining the preprocessed data; constructing a data dictionary based on the preprocessed data; based on the constructed data dictionary, importing data in the data dictionary into a database in batches; acquiring and processing the data stored in the database by using a back-end interface, so that the data is presented in a tree structure; and the front-end page obtains the tree structure data based on the back-end interface, transmits the obtained tree structure data to the corresponding tree component at the front end, and dynamically generates a tree selection box.

Description

Technical Field

[0001] The present invention relates to the technical field of data dictionary configuration, and in particular to a data dictionary configuration method, device, medium and equipment. Background Art

[0002] In modern information systems, as business requirements continue to change and become more complex, the management of system parameters and business parameters faces increasing challenges. Traditional data management methods often require the creation of separate data tables for each type of business constant, resulting in large maintenance workloads, data redundancy, and lack of flexibility. In addition, in scenarios where data needs to be updated and displayed in real time, existing systems often cannot respond quickly to dynamic changes, affecting user experience and system efficiency. Summary of the invention

[0003] The main purpose of the present invention is to solve the technical problem in the prior art that in scenarios where real-time updating and display of data is required, the existing system cannot quickly respond to dynamic changes, thus affecting user experience and system efficiency.

[0004] A first aspect of the present invention provides a data dictionary configuration method, comprising: Acquire corresponding data based on multiple data sources respectively, and preprocess the acquired data to obtain preprocessed data; Build a data dictionary based on the preprocessed data; Based on the constructed data dictionary, batch import the data in the data dictionary into the database; Use the backend interface to obtain and process the data stored in the database so that the data is presented in a tree structure; The front-end page obtains tree structure data based on the back-end interface, passes the obtained tree structure data to the corresponding tree component of the front-end, and dynamically generates a tree selection box.

[0005] Optionally, in a first implementation of the first aspect of the present invention, acquiring corresponding data based on multiple data sources respectively, and preprocessing the acquired data to obtain preprocessed data includes: Acquire corresponding data based on multiple data sources respectively, integrate the data from different data sources using ETL tools and / or data pipelines to obtain integrated data, and store the integrated data in a single storage module; Perform integrity check, consistency check, uniqueness verification, timeliness assessment and desensitization on the data in the storage module to obtain processed data; Based on the processed data, natural language processing technology is used to perform word segmentation and part-of-speech tagging to obtain data after word segmentation and tagging; Based on the data after word segmentation and annotation, the K-means algorithm is used to perform clustering and classification to obtain the data after clustering and classification; The Apriori algorithm is used to perform correlation analysis on the data after clustering and classification, and a knowledge graph is constructed based on the correlation analysis.

[0006] Optionally, in a second implementation of the first aspect of the present invention, the method further includes: Build a classification model based on random forest; The constructed classification model is trained using the clustered and classified data to obtain a trained classification model; Use the trained classification model to automatically classify the new data obtained.

[0007] Optionally, in a third implementation of the first aspect of the present invention, the data dictionary also includes prediction data; Construct a prediction model, select feature data that meets preset requirements based on the data dictionary, and train the prediction model by supervised learning or unsupervised learning method using the selected feature data that meets preset requirements to obtain a trained prediction model; The trained prediction model is evaluated by cross-validation, AUC-ROC curve, precision and recall indicators to obtain the current optimal prediction model; Use the current optimal prediction model to predict feature values; Regularly obtain user feedback data, and use the feedback data and updated feature data to train and update the current prediction model.

[0008] Optionally, in a fourth implementation of the first aspect of the present invention, the step of importing the data in the data dictionary into the database in batches based on the constructed data dictionary includes: Acquire Excel data based on the constructed data dictionary, and convert the acquired Excel data into a preset format to obtain Excel data that meets the preset format; Use SQL to batch import Excel data that meets the preset format into the database.

[0009] Optionally, in a fifth implementation of the first aspect of the present invention, the acquiring and processing of data stored in the database using the backend interface so that the data is presented in a tree structure includes: Create a RESTful API and obtain the dictionary code through the created RESTful API; According to the dictionary code, use recursive query or CTE to obtain the corresponding data based on the database; A tree structure is constructed based on the corresponding data obtained.

[0010] Optionally, in a sixth implementation of the first aspect of the present invention, the method further includes: By including Canal and / or Debezium to configure the database change monitoring mechanism, when the data dictionary is updated, an update notification is triggered and transmitted to the backend through the message queue; After receiving the update notification, the backend sends it to the frontend; The front end uses WebSocket or polling technology to receive update notifications pushed by the back end in real time, thereby achieving real-time updates of the front end.

[0011] A second aspect of the present invention provides a data dictionary configuration device, comprising: A data acquisition module is used to acquire corresponding data based on multiple data sources respectively, and preprocess the acquired data to obtain preprocessed data; Dictionary building module, used to build a data dictionary based on preprocessed data; Import database module, based on the constructed data dictionary, import the data in the data dictionary into the database in batches; Build a tree structure module, which is used to obtain and process the data stored in the database using the back-end interface so that the data is presented in a tree structure; Generate a tree selection box module, which is used for the front-end page to obtain tree structure data based on the back-end interface, and pass the obtained tree structure data to the corresponding tree component of the front-end to dynamically generate a tree selection box.

[0012] Optionally, in a first implementation of the second aspect of the present invention, the data acquisition module includes: Acquire corresponding data based on multiple data sources respectively, integrate the data from different data sources using ETL tools and / or data pipelines to obtain integrated data, and store the integrated data in a single storage module; Perform integrity check, consistency check, uniqueness verification, timeliness assessment and desensitization on the data in the storage module to obtain processed data; Based on the processed data, natural language processing technology is used to perform word segmentation and part-of-speech tagging to obtain data after word segmentation and tagging; Based on the data after word segmentation and annotation, the K-means algorithm is used to perform clustering and classification to obtain the clustered and classified data; The Apriori algorithm is used to perform correlation analysis on the data after clustering and classification, and a knowledge graph is constructed based on the correlation analysis.

[0013] Optionally, in a second implementation of the second aspect of the present invention, the device also includes: an automatic classification module, used to construct a classification model based on random forest; train the constructed classification model using the data after clustering classification to obtain a trained classification model; and automatically classify the acquired new data using the trained classification model.

[0014] Optionally, in a third implementation of the second aspect of the present invention, the data dictionary also includes prediction data; Construct a prediction model, select feature data that meets preset requirements based on the data dictionary, and train the prediction model by supervised learning or unsupervised learning method using the selected feature data that meets preset requirements to obtain a trained prediction model; The trained prediction model is evaluated by cross-validation, AUC-ROC curve, precision and recall indicators to obtain the current optimal prediction model; Use the current optimal prediction model to predict feature values; Regularly obtain user feedback data, and use the feedback data and updated feature data to train and update the current prediction model.

[0015] Optionally, in a fourth implementation of the second aspect of the present invention, the database import module includes: being used to obtain Excel data based on a constructed data dictionary, and converting the obtained Excel data into a preset format to obtain Excel data that meets the preset format; and using SQL to batch import the Excel data that meets the preset format into the database.

[0016] Optionally, in a fifth implementation of the second aspect of the present invention, the tree structure building module includes: creating a RESTful API and obtaining a dictionary code through the created RESTful API; obtaining corresponding data based on a database using a recursive query or CTE according to the obtained dictionary code; and building a tree structure based on the obtained corresponding data.

[0017] Optionally, in a sixth implementation of the second aspect of the present invention, the device also includes: a real-time update module, which is used to configure the database change monitoring mechanism through Canal and / or Debezium. When the data dictionary is updated, an update notification is triggered and transmitted to the back end through a message queue; after the back end receives the update notification, it sends it to the front end in parallel; the front end uses WebSocket or polling technology to receive the update notification pushed by the back end in real time, so as to realize real-time update of the front end.

[0018] A third aspect of the present invention provides an electronic device, the electronic device comprising a memory and at least one processor, the memory storing instructions; The at least one processor calls the instructions in the memory so that the electronic device executes each step of the data dictionary configuration method described above.

[0019] A fourth aspect of the present invention provides a computer-readable storage medium having instructions stored thereon, and the instructions, when executed by a processor, implement the various steps of the above-mentioned data dictionary configuration method.

[0020] Compared with the prior art, the present invention has the following beneficial effects: 1. The present invention is applicable to various scenarios and multi-level constant dictionary type maintenance; 2. It is not necessary to maintain related data tables for each business constant type; 3. Suitable for dictionary maintenance where business constants often change and the order of constants has strict requirements. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] Other features, objects and advantages of the present invention will become more apparent from the detailed description of non-limiting embodiments made with reference to the following drawings: Figure 1 A first flow chart of a data dictionary configuration method provided in an embodiment of the present invention.

[0022] Figure 2 A second flow chart of the data dictionary configuration method provided in an embodiment of the present invention.

[0023] Figure 3 A third flow chart of the data dictionary configuration method provided in an embodiment of the present invention.

[0024] Figure 4 A schematic diagram of the structure of a data dictionary configuration device provided in an embodiment of the present invention.

[0025] Figure 5 A schematic diagram of the structure of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0026] The embodiment of the present invention provides a data dictionary configuration method, device, medium and equipment, which respectively obtains corresponding data based on multiple data sources, and preprocesses the obtained data to obtain preprocessed data; constructs a data dictionary based on the preprocessed data; based on the constructed data dictionary, batch imports the data in the data dictionary into the database; uses the back-end interface to obtain and process the data stored in the database, so that the data is presented in a tree structure; the front-end page obtains the tree structure data based on the back-end interface, and passes the obtained tree structure data to the corresponding tree component of the front-end, and dynamically generates a tree selection box. The present invention solves the technical problem that in scenarios where real-time update and display of data are required, the existing system cannot respond quickly to dynamic changes, affecting the user experience and system efficiency.

[0027] The terms "first", "second", "third", "fourth", etc. (if any) in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments described herein can be implemented in an order other than that illustrated or described herein. In addition, the terms "including" or "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0028] For ease of understanding, the specific process of the embodiment of the present invention is described below. Figure 1 , a first embodiment of the data dictionary configuration method in an embodiment of the present invention includes: 101. Obtain corresponding data based on multiple data sources respectively, and preprocess the acquired data to obtain preprocessed data; In this embodiment, corresponding data is obtained based on multiple data sources respectively, and data from different data sources are integrated using ETL tools and / or data pipelines to obtain integrated data, and the integrated data is stored in a single storage module; this embodiment can efficiently integrate information from multiple data sources through ETL tools and / or data pipelines; The data in the storage module is subjected to integrity check, consistency check, uniqueness verification, timeliness evaluation and desensitization processing to obtain processed data; this embodiment can ensure data quality by performing integrity check, consistency check, uniqueness verification, timeliness evaluation and desensitization processing on the data; Based on the processed data, natural language processing technology is used to perform word segmentation and part-of-speech tagging to obtain data after word segmentation and tagging; The K-means algorithm is used to perform clustering and classification based on the data after word segmentation and annotation to obtain clustered and classified data; when the data in the data dictionary is updated, the classification model based on random forest is used to automatically update the acquired new data; The classification model uses the clustered classified data to train the constructed classification model to obtain a trained classification model; The Apriori algorithm is used to perform correlation analysis on the data after clustering and classification, and a knowledge graph is constructed based on the correlation analysis.

[0029] This embodiment performs classification and analysis based on an automated algorithm, which not only improves data processing efficiency but also improves data processing accuracy.

[0030] 102. Build a data dictionary based on the preprocessed data; This embodiment constructs a data dictionary based on the preprocessed data to ensure the accuracy and reliability of the data in the dictionary.

[0031] 103. Based on the constructed data dictionary, import the data in the data dictionary into the database in batches; In this embodiment, Excel data is acquired based on the constructed data dictionary, and the acquired Excel data is converted into a preset format to obtain Excel data that meets the preset format; Use SQL to batch import Excel data that meets the preset format into the database.

[0032] 104. Using the back-end interface to obtain and process the data stored in the database, so that the data is presented in a tree structure; In this embodiment, a RESTful API is created, and the dictionary code is obtained through the created RESTful API; According to the dictionary code, use recursive query or CTE to obtain the corresponding data based on the database; A tree structure is constructed based on the corresponding data obtained.

[0033] 105. The front-end page obtains the tree structure data based on the back-end interface, and transmits the obtained tree structure data to the corresponding tree component of the front-end to dynamically generate a tree selection box.

[0034] This embodiment is applicable to various scenarios and multi-level constant dictionary type maintenance, reduces the creation of configuration tables, and makes the scenarios more flexible and variable.

[0035] See also Figure 2 , a second embodiment of the data dictionary configuration method in an embodiment of the present invention includes: 201. Obtain corresponding data based on multiple data sources respectively, and preprocess the acquired data to obtain preprocessed data; In this embodiment, corresponding data is obtained based on multiple data sources respectively, and data from different data sources are integrated using ETL tools and / or data pipelines to obtain integrated data, and the integrated data is stored in a single data warehouse or data lake; data quality checks are performed on the data stored in the single data warehouse or data lake, and the data quality checks include: integrity checks, consistency checks, uniqueness verification, and timeliness assessments to ensure that the data is accurate; data stored in the single data warehouse or data lake are also desensitized, specifically including: desensitizing data involving privacy to ensure compliance and security.

[0036] Based on the processed data, natural language processing technology is used to perform word segmentation and part-of-speech tagging to obtain data after word segmentation and tagging; In this embodiment, the NLP tool jieba (for Chinese), NLTK or spaCy (for English) is used to segment the text and mark the part of speech of each word.

[0037] Based on the data after word segmentation and annotation, the K-means algorithm is used to perform clustering and classification to obtain the data after clustering and classification; In this embodiment, when the data in the data dictionary is updated, the classification model constructed based on random forest is used to automatically update the acquired new data; The classification model uses the data after clustering and classification to train the constructed classification model to obtain a trained classification model.

[0038] The Apriori algorithm is used to perform correlation analysis on the data after clustering and classification, and a knowledge graph is constructed based on the correlation analysis.

[0039] In this embodiment, the data dictionary also includes prediction data; In this embodiment, a prediction model is constructed, feature data that meets preset requirements is selected based on a data dictionary, and the prediction model is trained by supervised learning or unsupervised learning method using the selected feature data that meets preset requirements to obtain a trained prediction model; The trained prediction model is evaluated by cross-validation, AUC-ROC curve, precision and recall indicators to obtain the current optimal prediction model; Use the current optimal prediction model to predict feature values; Regularly obtain user feedback data, and use the feedback data and updated feature data to train and update the current prediction model.

[0040] 202. Construct a data dictionary based on the preprocessed data; 203. Based on the constructed data dictionary, batch import the data in the data dictionary into the database; In this embodiment, the Excel data in the data dictionary is exported to CSV or a file that conforms to the database import format. Script writing: Write SQL scripts or use database management tools (MySQL Workbench, pgAdmin) to batch import data into the database. The script needs to consider data deduplication, error handling, and transaction control to ensure the stability and consistency of data import. Performance optimization: When importing large-scale data, use batch inserts (MySQL's LOAD DATA INFILE, SQLServer's BULK INSERT) to improve efficiency.

[0041] 204. Using the back-end interface to obtain and process the data stored in the database, so that the data is presented in a tree structure; In this embodiment, a RESTful API is created, and the dictionary code is obtained through the created RESTful API; According to the dictionary code, use recursive query or CTE to obtain the corresponding data based on the database; A tree structure is constructed based on the corresponding data obtained.

[0042] 205. The front-end page obtains tree structure data based on the back-end interface, and transmits the obtained tree structure data to the corresponding tree component of the front-end, and dynamically generates a tree selection box; In this embodiment, the Tree component of a suitable front-end UI framework (Ant Design, Element UI) is selected to receive the data returned by the back-end and automatically generate a tree selection box. Interaction design: realize user-friendly interaction, expand / collapse nodes, search and filter, and select node linkage. Responsive layout: ensure good display effects on different screen sizes and improve user experience.

[0043] 206. Real-time update of front-end; In this embodiment, by including Canal and / or Debezium to configure the database change monitoring mechanism, when the data dictionary is updated, an update notification is triggered and transmitted to the backend through the message queue; After receiving the update notification, the backend sends it to the frontend; The front end uses WebSocket or polling technology to receive update notifications pushed by the back end in real time, thereby achieving real-time updates of the front end.

[0044] See also Figure 3 , a third embodiment of the data dictionary configuration method in the embodiment of the present invention includes: 301. Create a data dictionary in Excel; In this embodiment, establishing a data dictionary excel includes: Collecting data includes: identifying data sources, including ERP systems, WMS (warehouse management system), TMS (transportation management system), IoT sensor data, customer service records, and social media feedback.

[0045] Data preprocessing includes: Data integration: Use ETL (Extract, Transform, Load) tools or data pipelines to bring together data from different systems into a single data warehouse or data lake.

[0046] Data quality check: Perform data quality checks, including integrity checks, consistency checks, uniqueness verification, and timeliness assessments to ensure data accuracy.

[0047] Data desensitization: Data involving privacy (customer information) is desensitized to ensure compliance and security.

[0048] Perform word segmentation and part-of-speech tagging: Use NLP tools such as jieba (for Chinese), NLTK or spaCy (for English) to segment the text and mark the part of speech of each word. Entity recognition: Identify entities unique to the logistics field, such as location (warehouse location, destination), time (delivery period), and item (cargo type), using named entity recognition (NER) technology. Sentiment analysis: Perform sentiment analysis on customer feedback or social media text to determine satisfaction with logistics services.

[0049] Clustering and classification: Based on business understanding, select the variables that best represent the characteristics of logistics activities, such as transportation distance, cargo weight, and transportation time; Cluster analysis: Use the K-means algorithm to group logistics activities and identify different types of logistics service models or customer groups. Intelligent classification: Build a classification model, random forest, automatically classify new data points, and distinguish between ordinary transportation and cold chain logistics, air transportation and sea transportation.

[0050] Association rule mining: Use the Apriori algorithm to explore the correlation between different logistics activities, product types, and customer preferences to find opportunities to improve efficiency or reduce costs. Recommendation system: Based on the discovered association rules, develop a recommendation system to provide customers with personalized services or optimize logistics solutions.

[0051] Construction and optimization of prediction models: Select appropriate data sets and use supervised learning or unsupervised learning methods to train models. For example, use historical data to predict cargo damage rate and delay probability. Feature engineering: Create meaningful feature combinations and combine season, weather, and cargo type to predict transportation time to improve model prediction accuracy. Model evaluation and selection: Evaluate model performance through cross-validation, AUC-ROC curves, precision, and recall rate indicators to select the optimal model for deployment.

[0052] Knowledge graph construction: clarify the core entities in the knowledge graph (suppliers, warehouses, transportation tools) and the relationships between them (supply, storage, transportation). Graph database design: use Neo4j and JanusGraph graph databases to store and query knowledge graphs, supporting complex queries and reasoning. Graph visualization: display the knowledge graph through visualization tools, allowing non-technical users to intuitively understand the structure and relationships of the logistics network.

[0053] Continuous iteration and optimization: Establish a feedback mechanism to collect user feedback on the use of the data dictionary. Model retraining: Regularly retrain the model with new data to maintain the timeliness and accuracy of the model. Automated update: Develop automated scripts or platforms to monitor data changes and automatically trigger the dictionary update process to ensure the timeliness and reliability of the data dictionary.

[0054] 302. SQL batch inserts default excel data; In this embodiment, the Excel data is exported to CSV or a file that conforms to the database import format. Script writing: Write SQL scripts or use database management tools (MySQL Workbench, pgAdmin) to batch import data into the database. The script needs to consider data deduplication, error handling, and transaction control to ensure the stability and consistency of data import. Performance optimization: When importing large-scale data, use batch inserts (MySQL's LOAD DATA INFILE, SQL Server's BULKINSERT) to improve efficiency.

[0055] 303. Backend interface - Get tree level field data according to dictionary code value, first level, second level, third level, and sort; In this embodiment, a RESTful API is designed to receive the dictionary code as a parameter and return the tree structure data under the specified code, including the code, name, and child node information of each level. Hierarchical construction: The tree structure is generated in SQL through recursive query or CTE (common table expression) on the back end to ensure that the data is correctly sorted by level. Performance considerations: Optimize query logic and use caching strategies to reduce database pressure, especially when facing a large number of concurrent requests.

[0056] 304. Front-end page - automatically load constant selection box according to back-end tree; In this embodiment, the Tree component of a suitable front-end UI framework (Ant Design, Element UI) is selected to receive the data returned by the back-end and automatically generate a tree selection box. Interaction design: realize user-friendly interaction, expand / collapse nodes, search and filter, and select node linkage. Responsive layout: ensure good display effects on different screen sizes and improve user experience.

[0057] 305. Project deployment, dictionary item hot loading of the latest database data; In this embodiment, a database change monitoring mechanism (using Canal, Debezium) is configured during deployment, and once the dictionary data is updated, an event notification is triggered immediately. Hot update mechanism: The front end uses WebSocket or polling to receive update notifications pushed by the background in real time, and the latest dictionary items can be loaded without restarting.

[0058] 306. The business automatically edits the business dictionary options, status, sorting, and description information according to the real-time scenario; refreshes the business dictionary cache, and maintains the synchronization of hot-loaded constant data on each device; In this embodiment, business personnel are assigned editing permissions, allowing them to directly modify the status, sorting, and description information of dictionary items under specific conditions. Transaction processing: When modifying dictionary data, ensure the consistency of transactions to avoid incomplete or conflicting data. Cache refresh: After the modification is completed, clear the relevant cache immediately to ensure that all service instances can obtain the latest dictionary data immediately, and achieve data synchronization under active-active deployment. 307. The customer accesses the page and loads the latest maintenance constant data; In this embodiment, when a client visits a page for the first time, the front-end application should immediately initiate a request to obtain the latest dictionary data from the back-end and quickly render the page. User experience: Ensure that data is loaded quickly and provide a basic user experience even when the network is unstable. For example, skeleton screens and placeholders can be used to reduce latency. Version compatibility: If there are major changes in dictionary data that affect the front-end display logic, version control should be considered to ensure compatibility with old client versions or prompt upgrades.

[0059] The above describes the data dictionary configuration method in the embodiment of the present invention. The following describes the data dictionary configuration device in the embodiment of the present invention. Figure 4 , an embodiment of a data dictionary configuration device in an embodiment of the present invention includes: The data acquisition module 401 is used to acquire corresponding data based on multiple data sources respectively, and preprocess the acquired data to obtain preprocessed data; In this embodiment, the data acquisition module 401 includes: The data acquisition module 4011 is used to respectively acquire corresponding data based on multiple data sources, integrate the data from different data sources using ETL tools and / or data pipelines to obtain integrated data, and store the integrated data in a single data warehouse or data lake; The data quality check module 4012 is used to perform data quality checks on the data stored in a single data warehouse or data lake. The data quality checks include: integrity check, consistency check, uniqueness verification and timeliness assessment to ensure data accuracy; the data stored in a single data warehouse or data lake is also desensitized, specifically including: desensitizing data involving privacy to ensure compliance and security.

[0060] The word segmentation and tagging module 4013 is used to perform word segmentation and part-of-speech tagging based on the processed data using natural language processing technology to obtain word segmentation and tagged data; In this embodiment, the NLP tool jieba (for Chinese), NLTK or spaCy (for English) is used to segment the text and mark the part of speech of each word.

[0061] The clustering classification module 4014 is used to perform clustering classification based on the data after word segmentation and annotation using the K-means algorithm to obtain clustered classified data; In this embodiment, when the data in the data dictionary is updated, the classification model constructed based on random forest is used to automatically update the acquired new data; The classification model uses the data after clustering and classification to train the constructed classification model to obtain a trained classification model.

[0062] The association analysis module 4015 is used to perform association analysis based on the clustered and classified data using the Apriori algorithm, and to construct a knowledge graph based on the association analysis.

[0063] In this embodiment, the data dictionary also includes prediction data; In this embodiment, a prediction model is constructed, feature data that meets preset requirements is selected based on a data dictionary, and the prediction model is trained by supervised learning or unsupervised learning method using the selected feature data that meets preset requirements to obtain a trained prediction model; The trained prediction model is evaluated by cross-validation, AUC-ROC curve, precision and recall indicators to obtain the current optimal prediction model; Use the current optimal prediction model to predict feature values; Regularly obtain user feedback data, and use the feedback data and updated feature data to train and update the current prediction model.

[0064] A dictionary building module 402 is used to build a data dictionary based on the preprocessed data; The database import module 403 is used to import the data in the data dictionary into the database in batches based on the constructed data dictionary; In this embodiment, the import database module 403 includes: exporting the Excel data in the data dictionary to CSV or a file that conforms to the database import format; writing scripts: writing SQL scripts or using database management tools (MySQL Workbench, pgAdmin) to batch import data into the database. The script needs to consider data deduplication, error handling and transaction control to ensure the stability and consistency of data import. Performance optimization: When importing large-scale data, use batch inserts (MySQL's LOAD DATA INFILE, SQL Server's BULK INSERT) to improve efficiency.

[0065] A tree structure building module 404 is used to obtain and process the data stored in the database using the back-end interface so that the data is presented in a tree structure; In this embodiment, the tree structure building module 404 includes: A dictionary code acquisition module 4041 is used to create a RESTful API and acquire a dictionary code through the created RESTful API; A module 4042 for acquiring data based on a database is used to acquire corresponding data based on a database by using a recursive query or CTE according to the acquired dictionary code; The tree structure module 4043 is used to construct a tree structure based on the corresponding data obtained.

[0066] Generate tree selection box module 405, used for the front-end page to obtain tree structure data based on the back-end interface, and pass the obtained tree structure data to the corresponding tree component of the front-end to dynamically generate a tree selection box; In this embodiment, the tree selection box generation module 405 includes: selecting the Tree component of a suitable front-end UI framework (AntDesign, Element UI), receiving the data returned by the back-end, and automatically generating a tree selection box. Interaction design: realizing user-friendly interaction, expanding / folding nodes, searching and filtering, and selecting nodes for linkage. Responsive layout: ensuring good display effects on different screen sizes and improving user experience.

[0067] A real-time update module 406, used to update the front end in real time; The real-time update module 406 includes: By including Canal and / or Debezium to configure the database change monitoring mechanism, when the data dictionary is updated, an update notification is triggered and transmitted to the backend through the message queue; After receiving the update notification, the backend sends it to the frontend; The front end uses WebSocket or polling technology to receive update notifications pushed by the back end in real time, thereby achieving real-time updates of the front end.

[0068] above Figure 4 The data dictionary configuration device in the embodiment of the present invention is described in detail from the perspective of modular functional entities, and the electronic device in the embodiment of the present invention is described in detail from the perspective of hardware processing.

[0069] Figure 5 7 is a schematic diagram of the structure of an electronic device provided by an embodiment of the present invention. The electronic device 700 may have relatively large differences due to different configurations or performances, and may include one or more processors (central processing units, CPU) 710 (for example, one or more processors) and a memory 720, and one or more storage media 730 (for example, one or more mass storage devices) storing application programs 733 or data 732. Among them, the memory 720 and the storage medium 730 can be temporary storage or permanent storage. The program stored in the storage medium 730 may include one or more modules (not shown in the figure), and each module may include a series of instruction operations in the electronic device 700. Furthermore, the processor 710 may be configured to communicate with the storage medium 730 to execute a series of instruction operations in the storage medium 730 on the electronic device 700.

[0070] The electronic device 700 may also include one or more power supplies 740, one or more wired or wireless network interfaces 750, one or more input and output interfaces 750, and / or one or more operating systems 731, such as Windows Serve, Mac OS X, Unix, Linux, FreeBSD, etc. It will be appreciated by those skilled in the art that Figure 5 The structure of the electronic device shown does not constitute a limitation on the electronic device, and may include more or less components than shown in the figure, or combine some components, or arrange the components differently.

[0071] The present invention also provides a computer-readable storage medium, which may be a non-volatile computer-readable storage medium or a volatile computer-readable storage medium. Instructions are stored in the computer-readable storage medium. When the instructions are executed on a computer, the computer executes the steps of the data dictionary configuration method.

[0072] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working process of the above-described system, device, or unit can refer to the corresponding process in the aforementioned method embodiment and will not be repeated here.

[0073] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention is essentially or the part that contributes to the prior art or the whole or part of the technical solution can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions to enable a computer device (which can be a personal computer, server, or network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM), random access memory (RAM), disk or optical disk and other media that can store program code.

[0074] As described above, the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features thereof may be replaced by equivalents. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A data dictionary configuration method, characterized in that: include: Acquire corresponding data based on multiple data sources respectively, and preprocess the acquired data to obtain preprocessed data; Build a data dictionary based on the preprocessed data; Based on the constructed data dictionary, batch import the data in the data dictionary into the database; Use the backend interface to obtain and process the data stored in the database so that the data is presented in a tree structure; The front-end page obtains tree structure data based on the back-end interface, passes the obtained tree structure data to the corresponding tree component of the front-end, and dynamically generates a tree selection box.

2. The data dictionary configuration method according to claim 1, characterized in that: The method of acquiring corresponding data based on multiple data sources and preprocessing the acquired data to obtain preprocessed data includes: Acquire corresponding data based on multiple data sources respectively, integrate the data from different data sources using ETL tools and / or data pipelines to obtain integrated data, and store the integrated data in a single storage module; Perform integrity check, consistency check, uniqueness verification, timeliness assessment and desensitization on the data in the storage module to obtain processed data; Based on the processed data, natural language processing technology is used to perform word segmentation and part-of-speech tagging to obtain data after word segmentation and tagging; Based on the data after word segmentation and annotation, the K-means algorithm is used to perform clustering and classification to obtain the clustered and classified data; The Apriori algorithm is used to perform correlation analysis on the data after clustering and classification, and a knowledge graph is constructed based on the correlation analysis.

3. The data dictionary configuration method according to claim 2, characterized in that: The method further comprises: Build a classification model based on random forest; The constructed classification model is trained using the clustered and classified data to obtain a trained classification model; Use the trained classification model to automatically classify the new data obtained.

4. The data dictionary configuration method according to claim 1, characterized in that: The data dictionary also includes forecast data; Construct a prediction model, select feature data that meets preset requirements based on the data dictionary, and train the prediction model by supervised learning or unsupervised learning method using the selected feature data that meets preset requirements to obtain a trained prediction model; The trained prediction model is evaluated by cross-validation, AUC-ROC curve, precision and recall indicators to obtain the current optimal prediction model; Use the current optimal prediction model to predict feature values; Regularly obtain user feedback data, and use the feedback data and updated feature data to train and update the current prediction model.

5. The data dictionary configuration method according to claim 1, characterized in that: The step of importing the data in the data dictionary into the database in batches based on the constructed data dictionary includes: Acquire Excel data based on the constructed data dictionary, and convert the acquired Excel data into a preset format to obtain Excel data that meets the preset format; Use SQL to batch import Excel data that meets the preset format into the database.

6. The data dictionary configuration method according to claim 1, characterized in that: The method of acquiring and processing the data stored in the database by using the back-end interface so that the data is presented in a tree structure includes: Create a RESTful API and obtain the dictionary code through the created RESTful API; According to the dictionary code, use recursive query or CTE to obtain the corresponding data based on the database; A tree structure is constructed based on the corresponding data obtained.

7. The data dictionary configuration method according to claim 1, characterized in that: The method further comprises: By including Canal and / or Debezium to configure the database change monitoring mechanism, when the data dictionary is updated, an update notification is triggered and transmitted to the backend through the message queue; After receiving the update notification, the backend sends it to the frontend; The front end uses WebSocket or polling technology to receive update notifications pushed by the back end in real time, thereby achieving real-time updates of the front end.

8. A data dictionary configuration device, characterized in that: include: A data acquisition module is used to acquire corresponding data based on multiple data sources respectively, and preprocess the acquired data to obtain preprocessed data; Dictionary building module, used to build a data dictionary based on preprocessed data; Import database module, based on the constructed data dictionary, import the data in the data dictionary into the database in batches; Build a tree structure module, which is used to obtain and process the data stored in the database using the back-end interface so that the data is presented in a tree structure; Generate a tree selection box module, which is used for the front-end page to obtain tree structure data based on the back-end interface, and pass the obtained tree structure data to the corresponding tree component of the front-end to dynamically generate a tree selection box.

9. An electronic device, comprising a memory and at least one processor, wherein instructions are stored in the memory; The at least one processor calls the instructions in the memory so that the electronic device executes the various steps of the data dictionary configuration method as described in any one of claims 1-7.

10. A computer-readable storage medium having instructions stored thereon, characterized in that: When the instructions are executed by the processor, the various steps of the data dictionary configuration method as described in any one of claims 1-7 are implemented.

Images