A data warehouse construction method, device, equipment and storage medium

Abstract

The present disclosure provides a method, apparatus, device and storage medium for building a data warehouse, including: collecting at least one corresponding target website based on business needs, and building a data source layer based on the at least one target website; using web crawler technology to collect relevant data in the at least one target website according to data collection fields to obtain first relevant data, and building an original data layer based on the first relevant data; building a data detail table based on the data content in the original data layer, annotating at least one tag based on at least one related entry in the data detail table using a large model, building a tag table based on the at least one tag, and building a data storage layer based on the tag table and the detail table; creating a data query interface based on the data storage layer, creating a data application layer based on an application server, and the data application layer obtaining data in the data storage layer based on the data query interface.

Description

Technical Field

[0001] The present disclosure relates to the field of data collection, and in particular to a method, device, equipment and storage medium for building a data warehouse. Background Art

[0002] With the rapid development of information technology, the digital transformation of the government has become an important way to improve governance capabilities and service levels. However, the data of government policy documents are mostly stored in obscure text form, and there are still many challenges in processing and management, such as data silos, uneven data quality, and increased data security risks. Therefore, it is necessary to provide an efficient, secure, and scalable data warehouse construction and data governance solution to meet the needs of government policy document management. Summary of the invention

[0003] The present disclosure provides a data warehouse construction method, device, equipment and storage medium to at least solve the above technical problems existing in the prior art.

[0004] According to a first aspect of the present disclosure, a method for building a data warehouse is provided, the method comprising:

[0005] Collect at least one corresponding target website based on business needs, and build a data source layer based on the at least one target website;

[0006] Determine a data collection field based on the business requirement, collect relevant data in the at least one target website according to the data collection field using a web crawler technology to obtain first relevant data, and construct an original data layer based on the first relevant data;

[0007] Building a data detail table based on the data content in the original data layer, marking at least one tag using a large model based on at least one related entry in the data detail table, building a tag table based on the at least one tag, building a data storage layer based on the tag table and the detail table, and associating the original data layer with the data storage layer based on the at least one tag;

[0008] Creating a data query interface based on the data storage layer, creating a data application layer based on the application server, wherein the data application layer obtains data in the data storage layer based on the data query interface;

[0009] A data warehouse is built based on the data source layer, the original data layer, the data storage layer and the data application layer.

[0010] In one embodiment, the method further comprises:

[0011] Recording the collection time of the relevant data in the at least one target website, setting a weekly scheduled collection script based on the collection time and the data collection field, and updating the first relevant data based on the weekly scheduled collection script;

[0012] Based on the category of the at least one target website, the first relevant data is classified to obtain at least one data mart, the data in the at least one data mart is associated based on the at least one tag, and the original data layer is constructed based on the associated at least one data mart.

[0013] In one possible implementation, the step of constructing a data list based on the data content in the original data layer includes:

[0014] Cleaning the first related data based on structured query language to remove dirty data in the first related data to obtain second related data;

[0015] Deduplication of data is performed based on the region information and the tag information in the second relevant data to obtain third relevant data;

[0016] The third related data content is classified based on the data category to obtain at least one sub-detail list, and a data detail list is constructed based on the at least one sub-detail list.

[0017] In one embodiment, the method further comprises:

[0018] Acquire dimension information in the third related data, construct a dimension table based on the dimension information, and associate the dimension table with the data detail table based on the at least one tag;

[0019] Performing information statistics on the third related data based on the information quantity, data size, and data category in the third related data, constructing a data summary table based on the information statistics result, and associating the data summary table with the data detail table based on the at least one tag;

[0020] The data storage layer is constructed based on the associated dimension table, the data detail table, the label table and the data summary table.

[0021] In one embodiment, the method further comprises:

[0022] The at least one sub-detail table includes a content detail table for recording text content and a question and answer content table;

[0023] A single foreign key is set based on the contents of the content detail table and the question and answer content table, and the content detail table and the question and answer content table are associated as independent data tables based on the single foreign key;

[0024] A content query interface is created based on the data storage layer, and the data application layer obtains the data in the independent data table based on the content query interface.

[0025] According to a second aspect of the present disclosure, a data warehouse construction device is provided, the device comprising:

[0026] A data source layer construction unit, configured to collect at least one corresponding target website based on business requirements, and to construct a data source layer based on the at least one target website;

[0027] An original data layer construction unit is used to determine a data collection field based on the business requirements, collect relevant data in the at least one target website according to the data collection field using a web crawler technology to obtain first relevant data, and construct an original data layer based on the first relevant data;

[0028] a data storage layer construction unit, configured to construct a data detail table based on the data content in the original data layer, annotate at least one tag using a large model based on at least one related entry in the data detail table, construct a tag table based on the at least one tag, construct a data storage layer based on the tag table and the detail table, and associate the original data layer with the data storage layer based on the at least one tag;

[0029] A data application layer construction unit, used to create a data query interface based on the data storage layer, and to create a data application layer based on an application server, wherein the data application layer obtains data in the data storage layer based on the data query interface;

[0030] The data warehouse construction unit is used to build a data warehouse based on the data source layer, the original data layer, the data storage layer and the data application layer.

[0031] In one embodiment, the device further comprises:

[0032] A data updating unit, configured to record a collection time of the relevant data in the at least one target website, set a weekly scheduled collection script based on the collection time and the data collection field, and update the first relevant data based on the weekly scheduled collection script;

[0033] The original data layer construction unit is further used to classify the first related data based on the category of the at least one target website to obtain at least one data mart, associate the data in the at least one data mart based on the at least one tag, and construct the original data layer based on the at least one associated data mart;

[0034] The data storage layer construction unit is also used to perform data cleaning on the first related data based on a structured query language, remove dirty data in the first related data, and obtain second related data; perform data deduplication based on regional information and tag information in the second related data to obtain third related data; classify the content of the third related data based on data categories to obtain at least one sub-detail table, and construct a data detail table based on the at least one sub-detail table.

[0035] In one possible implementation, the data storage layer construction unit is further used to obtain dimension information in the third related data, construct a dimension table based on the dimension information, and associate the dimension table with the data detail table based on the at least one tag; perform information statistics on the third related data based on the information quantity, data size, and data category in the third related data, construct a data summary table based on the information statistics results, and associate the data summary table with the data detail table based on the at least one tag; construct the data storage layer based on the associated dimension table, the data detail table, the tag table, and the data summary table; the at least one sub-detail table includes a content detail table and a question-and-answer content table for recording text content; a single foreign key is set based on the content in the content detail table and the question-and-answer content table, and the content detail table and the question-and-answer content table are associated as independent data tables based on the single foreign key; a content query interface is created based on the data storage layer, and the data application layer to which it belongs obtains data in the independent data table based on the content query interface.

[0036] According to a third aspect of the present disclosure, there is provided an electronic device, including:

[0037] at least one processor; and

[0038] a memory communicatively connected to the at least one processor; wherein,

[0039] The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method described in the present disclosure.

[0040] According to a fourth aspect of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to cause the computer to execute the method described in the present disclosure.

[0041] The data warehouse construction method, device, equipment and storage medium disclosed in the present invention collect data from multiple original data websites as data sources, respectively establish a data source layer, an original data layer and a data storage layer, use a large model to annotate the data, create a corresponding label table, associate the data between layers, and connect with the data application layer created in the external application server through the corresponding data interface, thereby solving the problems of data islands, uneven data quality, increased data security risks, etc.

[0042] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] The above and other objects, features and advantages of the exemplary embodiments of the present disclosure will become readily understood by reading the detailed description below with reference to the accompanying drawings. In the accompanying drawings, several embodiments of the present disclosure are shown in an exemplary and non-limiting manner, in which:

[0044] In the drawings, the same or corresponding reference numerals represent the same or corresponding parts.

[0045] Figure 1 The following is a schematic diagram showing the implementation process of a data warehouse construction method according to an embodiment of the present disclosure. Figure 1 ;

[0046] Figure 2 The following is a schematic diagram showing the implementation process of a data warehouse construction method according to an embodiment of the present disclosure. Figure 2 ;

[0047] Figure 3 A schematic diagram of a data warehouse according to an embodiment of the present disclosure is shown;

[0048] Figure 4 A schematic diagram of a data warehouse construction device according to an embodiment of the present disclosure is shown;

[0049] Figure 5 A schematic diagram of the structure of an electronic device according to an embodiment of the present disclosure is shown. DETAILED DESCRIPTION

[0050] In order to make the purpose, features, and advantages of the present disclosure more obvious and easy to understand, the technical solutions in the embodiments of the present disclosure will be clearly and completely described below in conjunction with the drawings in the embodiments of the present disclosure. Obviously, the described embodiments are only part of the embodiments of the present disclosure, not all of the embodiments. Based on the embodiments in the present disclosure, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present disclosure.

[0051] Figure 1 The following is a schematic diagram showing the implementation process of a data warehouse construction method according to an embodiment of the present disclosure. Figure 1 ,like Figure 1 As shown, the implementation process of a data warehouse construction method according to an embodiment of the present disclosure includes the following steps:

[0052] Step 101: collect at least one corresponding target website based on business requirements, and build a data source layer based on the at least one target website.

[0053] In the disclosed embodiment, the at least one target website includes: a national government public website, a local government public website, a national enterprise information network, a third-party enterprise information network, an online open source website and other public websites.

[0054] Step 102, determining data collection fields based on the business requirements, using web crawler technology to collect relevant data in the at least one target website according to the data collection fields, obtaining first relevant data, and constructing an original data layer based on the first relevant data.

[0055] In the disclosed embodiment, the collection fields are determined through public discussion in combination with business needs, and complete data is collected as completely as possible to avoid repeated development and collection. The collection time of the relevant data in the at least one target website is recorded, and a weekly scheduled collection script is set based on the collection time and the data collection field. The first relevant data is updated based on the weekly scheduled collection script, wherein the first relevant data is the original data downloaded directly from the data website.

[0056] In an embodiment of the present disclosure, the first relevant data is classified based on the category of the at least one target website to obtain at least one data mart, wherein the category of the target website may be: policy information data, enterprise information data, professional question and answer knowledge data, etc.

[0057] Step 103, construct a data detail table based on the data content in the original data layer, annotate at least one tag using a large model based on at least one related entry in the data detail table, construct a tag table based on the at least one tag, construct a data storage layer based on the tag table and the detail table, and associate the original data layer with the data storage layer based on the at least one tag.

[0058] In an embodiment of the present disclosure, data cleaning is performed on the first related data based on a structured query language to remove dirty data in the first related data, wherein the cleaning content includes removing null values, abnormal value dirty data, and obtaining second related data, i.e., the second related data is the data after data cleaning; data deduplication is performed based on the region information and label information in the second related data, and preferably, the data is preliminarily deduplicated according to the region+title to obtain third related data, i.e., the third related data is the number after data deduplication; the content of the third related data is classified based on the data category to obtain at least one sub-detail table, and a data detail table is constructed based on the at least one sub-detail table.

[0059] In the disclosed embodiment, labeling is performed using large model technology, mainly based on the policy type, the applicable national economic industry, enterprise scale standard, effective time, region, and issuing department for definition and labeling, and a label labeling management system is established to obtain at least one label, and a label table is constructed based on the at least one label. Based on the at least one label, the data in the at least one data mart is associated, and the original data layer is constructed based on the at least one associated data mart.

[0060] In the disclosed embodiment, dimension information in the third related data is obtained, and a dimension table is constructed based on the dimension information, wherein the dimension table may include: a regional dimension table, a label management dimension table, a national economic industry division dimension table, a label dimension table, and an enterprise scale determination dimension table. The dimension table is associated with the data detail table based on the at least one label; information statistics are performed on the third related data based on the information quantity, data size, and data category in the third related data, a data summary table is constructed based on the information statistics results, and the data summary table is associated with the data detail table based on the at least one label; the data storage layer is constructed based on the associated dimension table, the data detail table, the label table, and the data summary table.

[0061] Step 104: Create a data query interface based on the data storage layer, create a data application layer based on the application server, and the data application layer obtains data in the data storage layer based on the data query interface.

[0062] In the disclosed embodiment, the data query interface is an application programming interface (API), and the data warehouse data application mainly provides data to the application end through the API interface. The application layer can provide basic policy query and display functions of different dimensions, policy comparison functions of different dimensions, policy aggregation and statistics functions of different dimensions, and provide underlying data for knowledge base questions and answers, and compare statistical aggregation data using a visual large-screen report form.

[0063] In the disclosed embodiment, at least one sub-detail table includes a content detail table (policy content table) and a question-and-answer content table that record text content; because the query efficiency will be greatly reduced if the characters of the policy content text are too long, a single foreign key is set based on the contents of the content detail table and the question-and-answer content table, and the content detail table and the question-and-answer content table are associated as independent data tables based on the single foreign key; at the same time, a separate API query channel is established, specifically: a content query interface is created based on the data storage layer, and the data application layer obtains the data in the independent data table based on the content query interface.

[0064] Step 105: Building a data warehouse based on the data source layer, the original data layer, the data storage layer and the data application layer.

[0065] In the disclosed embodiment, the data governance platform relies on the constructed data warehouse platform to set data modeling specifications and standards on the data warehouse platform, including table naming, field naming, table structure modeling, data type selection, data default settings, comment descriptions, data storage formats, primary key and foreign key definitions, and cleansing query script specifications.

[0066] Figure 2 The following is a schematic diagram showing the implementation process of a data warehouse construction method according to an embodiment of the present disclosure. Figure 2 ,like Figure 2 As shown, the implementation process of a data warehouse construction method in the embodiment of the present disclosure includes the following steps:

[0067] Step 201, data source definition.

[0068] In the disclosed embodiment, the data sources for data construction mainly come from the following sources: national government public websites, local government public websites, national enterprise information networks, third-party enterprise information networks, and online open source websites. The collection fields are determined through public discussions based on business needs, and complete data is collected as completely as possible to avoid repeated development and collection.

[0069] Step 202: data collection.

[0070] In the disclosed embodiment, web crawler technology is used to collect data from different data sources, and the data sources include local government policy data, national government policy data, open source platform policy data, and first-level open source related website data that does not involve corporate secrets. A weekly scheduled collection script is set, and a collection driver table is constructed to manage the collection script. The collection driver table is used to flexibly expand or delete and adjust the collection script, so as to update the data in time to ensure data timeliness and integrity.

[0071] Step 203: data cleaning.

[0072] In the disclosed embodiment, the collected data is cleaned using programming language (python) scripts and Structured Query Language (SQL). The cleaning content includes removing null values, abnormal values ​​and dirty data, and performing preliminary deduplication of the data according to region + title.

[0073] Step 204: Data warehouse modeling.

[0074] In the present disclosure, Figure 3 As shown, a schematic diagram of a data warehouse in an embodiment of the present disclosure is shown. From a business perspective, the data warehouse construction consists of three data marts, that is, small data warehouses. The three data marts are a policy information library, an enterprise information library, and a professional question-and-answer knowledge library. The three data marts form the entire data warehouse by associating some foreign key fields.

[0075] In the disclosed embodiment, the data warehouse is divided into four layers, namely: a data source layer, a raw data layer, a data storage layer and a data application layer, wherein the data source mainly includes six categories, namely the national government network, local government networks, the national enterprise information network, the third-party open source information network, the government question and answer network and other open source data on the Internet; the raw data layer is composed of the above three data marts; the data storage layer includes four categories of tables: dimension tables, detail tables, label tables and summary tables; the data application layer includes policy query, policy comparison, policy statistics, retrieval-augmented generation (RAG) question and answer and other subsequent expandable applications.

[0076] Step 205: Building a data warehouse.

[0077] In the disclosed embodiment, the data warehouse is constructed mainly by relying on a database relational database management system (Mysql) and a big data tool data warehouse tool (HIVE).

[0078] In the disclosed embodiment, the data warehouse constructs a data source layer which is mainly composed of national government public websites, local government public websites, national enterprise information networks, third-party enterprise information networks, and online open source websites. Taking local government data sources as an example, the data warehouse constructs a data source layer which defines the original collection fields: policy province, city, region, release time, release department, title, detailed content, declaration conditions, additional information, status code, link address, attachment address, field, collection time, number of views, and other 32 fields. While defining the collection information, a collection driver table is set up to manage the collection data script by counting the collection time, completion, data source, whether it needs to be updated, and update frequency.

[0079] In the disclosed embodiment, the collected data is classified according to policy information data, enterprise information data, and professional question and answer knowledge data, and an original data layer is constructed based on the classification results. Furthermore, the original data layer can store: national policy original table, local policy original table, national public enterprise information original table, third-party enterprise information original table, government question and answer original table, online question and answer original table, and professional knowledge base table.

[0080] In the disclosed embodiment, the data storage layer includes a dimension table, a detail table, a label table and a summary table, wherein the dimension table is constructed according to the dimension information collected in the original data layer, and the dimension table includes a regional dimension table, a label management dimension table, a national economic industry division dimension table, a label dimension table, and an enterprise scale determination dimension table. After the data in the data storage layer is cleaned, a detail table is constructed according to the data category, and the detail table includes: a national policy detail table, a local policy detail table, an enterprise information table, and a question and answer detail table. Since the query efficiency will be greatly reduced if the characters of the policy content text are too long, the policy content table and the question and answer content table are split into separate data tables and associated through a unique foreign key. The data in the data storage layer is labeled using large model technology, mainly defining and labeling from the policy type, the national economic industry to which it applies, the enterprise scale standard, the effective time, the region to which it belongs, and the issuing department, and establishing a label labeling management system, that is, establishing the labeling results as a label table, and the patents involving relevant policy label labeling management are not elaborated in this patent. Statistics are performed on the data content in the detail table, including: the number of data, the number of categories, the number of data corresponding to different data branches, the number of data question and answer results, etc., and a summary table is constructed based on the statistical results. The dimension table, detail table, tag table, and summary table are associated based on the tags in the tag table, and the association between the data storage layer and the original data layer is realized.

[0081] In the disclosed embodiment, the data warehouse data application mainly provides data to the application end through the API interface, provides basic policy query and display functions of different dimensions, policy comparison functions of different dimensions, policy aggregation and statistics functions of different dimensions, and provides underlying data for knowledge base questions and answers, and compares statistical aggregation data using visual large-screen reports. At the same time, the establishment of the policy data warehouse of this patent will also provide continuous and effective data support for other subsequent related applications. A data application layer is established based on the above-mentioned application-end functions, which at least includes: policy query, policy comparison, policy statistics, RAG question and answer and other application functions.

[0082] Step 206, data governance.

[0083] In the disclosed embodiment, the data governance platform relies on the data warehouse platform and sets data modeling specifications and standards on the data warehouse platform, including table naming, field naming, table structure modeling, data type selection, data default settings, comment descriptions, data storage formats, primary key and foreign key definitions, and cleansing query script specifications.

[0084] In the disclosed embodiment, data governance also includes data quality management, data security management and data integration interaction management. Specifically: data quality management, define data quality regular detection rules and detection indicators, check and detect metadata, data content, and data format, and configure the detection cycle, save abnormal data separately, trace the source through the data governance platform to find out the cause, and regularly generate quality detection reports, set different detection cycles for different data topics according to the characteristics of different data update frequencies, and monitor quality detection tasks to prompt alarms through emails and phone calls. Thereby ensuring data accuracy and integrity. Data security management, this patent uses data encryption technology for enterprise and policy privacy data such as phone numbers, names, etc., and ciphers numbers to protect data security and prevent data from being accessed without authorization. Permission access control is performed for different user settings to limit data access rights to ensure that only authorized users can access data; at the same time, data access and use are recorded to promptly discover and handle data security issues. Data integration and interaction management, design data integration solutions and interaction methods, and realize the integration and sharing of government policy document data under the premise of ensuring data security and quality.

[0085] Figure 4 A schematic diagram of a data warehouse construction device according to an embodiment of the present disclosure is shown. Figure 4 As shown, a data warehouse construction device in an embodiment of the present disclosure includes:

[0086] The data source layer construction unit 401 is used to collect at least one corresponding target website based on business requirements, and to construct a data source layer based on the at least one target website.

[0087] The original data layer construction unit 402 is used to determine the data collection field based on the business needs, use the web crawler technology to collect relevant data in the at least one target website according to the data collection field, obtain first relevant data, and construct the original data layer based on the first relevant data.

[0088] The original data layer construction unit 402 is further configured to classify the first related data based on the category of the at least one target website to obtain at least one data mart, associate the data in the at least one data mart based on the at least one tag, and construct the original data layer based on the at least one associated data mart.

[0089] The data storage layer construction unit 403 is used to construct a data detail table based on the data content in the original data layer, annotate at least one tag using a large model based on at least one related entry in the data detail table, construct a tag table based on the at least one tag, construct a data storage layer based on the tag table and the detail table, and associate the original data layer with the data storage layer based on the at least one tag.

[0090] The data storage layer construction unit 403 is also used to perform data cleaning on the first related data based on the structured query language, remove dirty data in the first related data, and obtain second related data; perform data deduplication based on the regional information and label information in the second related data to obtain third related data; classify the content of the third related data based on the data category to obtain at least one sub-detail table, and construct a data detail table based on the at least one sub-detail table.

[0091] The data storage layer construction unit 403 is also used to obtain dimension information in the third related data, construct a dimension table based on the dimension information, and associate the dimension table with the data detail table based on the at least one tag; perform information statistics on the third related data based on the information quantity, data size, and data category in the third related data, construct a data summary table based on the information statistics results, and associate the data summary table with the data detail table based on the at least one tag; construct the data storage layer based on the associated dimension table, the data detail table, the tag table, and the data summary table; the at least one sub-detail table includes a content detail table and a question-and-answer content table for recording text content; a single foreign key is set based on the content in the content detail table and the question-and-answer content table, and the content detail table and the question-and-answer content table are associated as independent data tables based on the single foreign key; a content query interface is created based on the data storage layer, and the data application layer to which it belongs obtains data in the independent data table based on the content query interface.

[0092] The data application layer construction unit 404 is used to create a data query interface based on the data storage layer and create a data application layer based on the application server, and the data application layer obtains the data in the data storage layer based on the data query interface.

[0093] The data warehouse construction unit 405 is used to build a data warehouse based on the data source layer, the original data layer, the data storage layer and the data application layer.

[0094] The data updating unit 406 is used to record the collection time of the relevant data in the at least one target website, set a weekly scheduled collection script based on the collection time and the data collection field, and update the first relevant data based on the weekly scheduled collection script.

[0095] In an exemplary embodiment, the data source layer construction unit 401, the original data layer construction unit 402, the data storage layer construction unit 403, the data application layer construction unit 404, the data warehouse construction unit 405 and the data update unit 406 can be implemented by one or more central processing units (CPU, Central Processing Unit), graphics processing units (GPU, Graphics Processing Unit), application specific integrated circuits (ASIC, Application Specific Integrated Circuit), DSP, programmable logic devices (PLD, Programmable Logic Device), complex programmable logic devices (CPLD, Complex Programmable Logic Device), field programmable gate arrays (FPGA, Field-Programmable Gate Array), general processors, controllers, microcontrollers (MCU, Micro Controller Unit), microprocessors (Microprocessors), or other electronic components.

[0096] Regarding the device in the above embodiment, the specific manner in which each module and unit performs operations has been described in detail in the embodiment of the method, and will not be elaborated here.

[0097] According to an embodiment of the present disclosure, the present disclosure also provides an electronic device and a readable storage medium.

[0098] Figure 5 A schematic block diagram of an example electronic device 800 that can be used to implement an embodiment of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or required herein.

[0099] like Figure 5As shown, the device 800 includes a computing unit 801, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 802 or a computer program loaded from a storage unit 808 into a random access memory (RAM) 803. In the RAM 803, various programs and data required for the operation of the device 800 can also be stored. The computing unit 801, the ROM 802, and the RAM 803 are connected to each other via a bus 804. An input / output (I / O) interface 805 is also connected to the bus 804.

[0100] A number of components in the device 800 are connected to the I / O interface 805, including: an input unit 806, such as a keyboard, a mouse, etc.; an output unit 807, such as various types of displays, speakers, etc.; a storage unit 808, such as a disk, an optical disk, etc.; and a communication unit 809, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 809 allows the device 800 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0101] The computing unit 801 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 801 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 801 performs the various methods and processes described above, such as a data warehouse construction method. For example, in some embodiments, a data warehouse construction method may be implemented as a computer software program, which is tangibly contained in a machine-readable medium, such as a storage unit 808. In some embodiments, part or all of the computer program may be loaded and / or installed on the device 800 via ROM 802 and / or communication unit 809. When the computer program is loaded into RAM 803 and executed by the computing unit 801, one or more steps of a data warehouse construction method described above may be performed. Alternatively, in other embodiments, the computing unit 801 may be configured to perform a data warehouse construction method in any other appropriate manner (e.g., by means of firmware).

[0102] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0103] The program code for implementing the method of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that the program code, when executed by the processor or controller, enables the functions / operations specified in the flow chart and / or block diagram to be implemented. The program code may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0104] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0105] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0106] The systems and techniques described herein may be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), and the Internet.

[0107] A computer system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The relationship of client and server is generated by computer programs running on respective computers and having a client-server relationship with each other. The server may be a cloud server, a server of a distributed system, or a server combined with a blockchain.

[0108] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps recorded in this disclosure can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved, and this document does not limit this.

[0109] In addition, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of the features. In the description of the present disclosure, the meaning of "plurality" is two or more, unless otherwise clearly and specifically defined.

[0110] The above is only a specific embodiment of the present disclosure, but the protection scope of the present disclosure is not limited thereto. Any person skilled in the art who is familiar with the technical field can easily think of changes or substitutions within the technical scope disclosed in the present disclosure, which should be included in the protection scope of the present disclosure. Therefore, the protection scope of the present disclosure should be based on the protection scope of the claims.

Claims

1. A method for building a data warehouse, characterized in that: The method comprises: Collect at least one corresponding target website based on business needs, and build a data source layer based on the at least one target website; Determine a data collection field based on the business requirement, collect relevant data in the at least one target website according to the data collection field using a web crawler technology to obtain first relevant data, and construct an original data layer based on the first relevant data; Building a data detail table based on the data content in the original data layer, marking at least one tag using a large model based on at least one related entry in the data detail table, building a tag table based on the at least one tag, building a data storage layer based on the tag table and the detail table, and associating the original data layer with the data storage layer based on the at least one tag; Creating a data query interface based on the data storage layer, creating a data application layer based on the application server, wherein the data application layer obtains data in the data storage layer based on the data query interface; A data warehouse is built based on the data source layer, the original data layer, the data storage layer and the data application layer.

2. The method according to claim 1, characterized in that The method further comprises: Recording the collection time of the relevant data in the at least one target website, setting a weekly scheduled collection script based on the collection time and the data collection field, and updating the first relevant data based on the weekly scheduled collection script; Based on the category of the at least one target website, the first relevant data is classified to obtain at least one data mart, the data in the at least one data mart is associated based on the at least one tag, and the original data layer is constructed based on the associated at least one data mart.

3. The method according to claim 1, characterized in that The step of constructing a data detail table based on the data content in the original data layer includes: Cleaning the first related data based on structured query language to remove dirty data in the first related data to obtain second related data; Deduplication of data is performed based on the region information and the tag information in the second relevant data to obtain third relevant data; The third related data content is classified based on the data category to obtain at least one sub-detail list, and a data detail list is constructed based on the at least one sub-detail list.

4. The method according to claim 3, characterized in that The method further comprises: Acquire dimension information in the third related data, construct a dimension table based on the dimension information, and associate the dimension table with the data detail table based on the at least one tag; Performing information statistics on the third related data based on the information quantity, data size, and data category in the third related data, constructing a data summary table based on the information statistics result, and associating the data summary table with the data detail table based on the at least one tag; The data storage layer is constructed based on the associated dimension table, the data detail table, the label table and the data summary table.

5. The method according to claim 3, characterized in that: The method further comprises: The at least one sub-detail table includes a content detail table for recording text content and a question and answer content table; A single foreign key is set based on the contents of the content detail table and the question and answer content table, and the content detail table and the question and answer content table are associated as independent data tables based on the single foreign key; A content query interface is created based on the data storage layer, and the data application layer obtains the data in the independent data table based on the content query interface.

6. A data warehouse construction device, characterized in that: The device comprises: A data source layer construction unit, configured to collect at least one corresponding target website based on business requirements, and to construct a data source layer based on the at least one target website; An original data layer construction unit is used to determine a data collection field based on the business requirements, collect relevant data in the at least one target website according to the data collection field using a web crawler technology to obtain first relevant data, and construct an original data layer based on the first relevant data; a data storage layer construction unit, configured to construct a data detail table based on the data content in the original data layer, annotate at least one tag using a large model based on at least one related entry in the data detail table, construct a tag table based on the at least one tag, construct a data storage layer based on the tag table and the detail table, and associate the original data layer with the data storage layer based on the at least one tag; A data application layer construction unit, used to create a data query interface based on the data storage layer, and to create a data application layer based on an application server, wherein the data application layer obtains data in the data storage layer based on the data query interface; The data warehouse construction unit is used to build a data warehouse based on the data source layer, the original data layer, the data storage layer and the data application layer.

7. The device according to claim 6, characterized in that The device also includes: A data updating unit, configured to record a collection time of the relevant data in the at least one target website, set a weekly scheduled collection script based on the collection time and the data collection field, and update the first relevant data based on the weekly scheduled collection script; The original data layer construction unit is further used to classify the first related data based on the category of the at least one target website to obtain at least one data mart, associate the data in the at least one data mart based on the at least one tag, and construct the original data layer based on the at least one associated data mart; The data storage layer construction unit is also used to perform data cleaning on the first related data based on a structured query language, remove dirty data in the first related data, and obtain second related data; perform data deduplication based on regional information and tag information in the second related data to obtain third related data; classify the content of the third related data based on data categories to obtain at least one sub-detail table, and construct a data detail table based on the at least one sub-detail table.

8. The device according to claim 7, characterized in that The data storage layer construction unit is further used to obtain dimension information in the third related data, construct a dimension table based on the dimension information, and associate the dimension table with the data detail table based on the at least one tag; perform information statistics on the third related data based on the information quantity, data size, and data category in the third related data, construct a data summary table based on the information statistics result, and associate the data summary table with the data detail table based on the at least one tag; The data storage layer is constructed based on the associated dimension table, the data detail table, the label table and the data summary table; the at least one sub-detail table includes a content detail table and a question and answer content table for recording text content; a single foreign key is set based on the content in the content detail table and the question and answer content table, and the content detail table and the question and answer content table are associated as independent data tables based on the single foreign key; a content query interface is created based on the data storage layer, and the data application layer to which it belongs obtains the data in the independent data table based on the content query interface.

9. An electronic device, characterized in that: include: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 5.

10. A non-transitory computer-readable storage medium storing computer instructions, characterized in that: The computer instructions are used to make a computer execute the method according to any one of claims 1-5.

Images