A government affair based on large model internet of things data sharing and data quality inspection method, system, device and storage medium

Abstract

The scheme discloses a government affair big model-based Internet of Things data sharing and data quality inspection method, system, equipment and storage medium, and a data department inputs a data application composed of natural language through a data sharing module, the data application includes data sharing requirements and data quality inspection requirements, a data sharing model extracts the data sharing requirements in the data application, intelligently generates required data tables, data fields and other contents, and a data quality inspection model intelligently identifies and extracts specific quality inspection requirements of a user, so that quality inspection of government affair data is completed, and the fine operation precision of data quality inspection is improved. The method only needs the data department to input data sharing and quality inspection requirements through a front-end interaction module, so that standard data applications and data quality inspection rules are automatically generated, so that the threshold and cost of data use of the data department are reduced, the rules of data quality inspection are increased according to specific data use requirements of the data department, so that the fine operation precision of data quality inspection is improved.

Description

Technical Field

[0001] This invention relates to the field of government data management, and in particular to a method, system, device, and storage medium for IoT data sharing and data quality inspection based on a large model for government affairs. Background Technology

[0002] Currently, data sharing and data quality inspection technologies for government IoT platforms or other large companies involving numerous internal and external data sources and users have significantly constrained practical applications. Due to the highly dispersed sources, large differences in formats, and inconsistent semantic standards of government IoT data, the data cleaning and standardization process has become complicated, and may even lead to cross-system data association errors. This results in excessively high sharing thresholds and high data usage costs for data-using departments. The sheer volume of data, various data business types, and numerous data table fields mean that even if data-using departments spend a considerable amount of time searching for data, they may not ultimately find the fields they actually need.

[0003] Furthermore, current data quality inspection is based on the overall system from a technical perspective. The rules and business conditions for quality inspection vary for each shared data application from the data-using department. It needs to be considered on a case-by-case basis. The underlying rule language of the quality inspection rules is too complex, and the data-using department cannot write the quality inspection rules itself, resulting in insufficient quality inspection refinement. Moreover, the existing quality inspection process still relies on manual intervention in core environments such as rule base construction and dirty data repair, which seriously reduces the credibility of quality inspection. Summary of the Invention

[0004] The present invention aims to overcome at least one of the defects of the prior art and provide a method, system, device and storage medium for IoT data sharing and data quality inspection based on a large model for government affairs. It is used to solve the technical problems in the prior art where the data-using departments have high data-using thresholds and cannot add data quality inspection rules according to specific data-using needs.

[0005] This invention provides a method for IoT data sharing and data quality inspection based on a large model for government affairs, including:

[0006] S01, The IoT platform acquires data from the device;

[0007] S02. Store the device-side data on the server side;

[0008] S03. Data users of the Internet of Things platform can submit data requests to the government data platform through the data sharing module.

[0009] S04. When the data application is approved, the government data platform transmits the government data to the data quality inspection module. When the data application is not approved, the government data platform returns the processing result to the data sharing module.

[0010] S05. The data quality inspection module performs quality inspection on the government data to obtain quality-inspected government data.

[0011] S06. The data quality inspection module stores the quality inspection government data in the server.

[0012] S07. The quality inspection administrative data is transmitted to the data sharing module via the server.

[0013] S08. The data unit can view the quality inspection administrative data through the data sharing module.

[0014] By generating a unified basic object model for each device, based on a question-and-answer model, the threshold for data use by data-using departments is lowered. Users only need to directly state what kind of government data they need through front-end interaction, and the required data tables, data fields, and other content can be generated. At the same time, in order to enable the data extracted by data-using units to be better applied to business, data validation rules can be added based on specific data usage needs, or rules can be generated based on the large model. In this way, data-using units can also quickly generate data quality inspection rules through natural language, so that the final obtained data can be better applied to business.

[0015] The device-side data mentioned in step S01 includes device attributes, device events, and device service information of two or more sensors, such as water immersion sensors, smoke sensors, gas sensors, displacement sensors, high-definition cameras, and smart valves. Device attributes include device model, latitude and longitude, etc., and device time includes alarms, etc.

[0016] The server-side stored data in step S02 also includes server-side IoT hardware device attributes, event and service information, platform logs, dimensional data, and data resource information obtained from the government data platform. The hardware device attributes include device attributes, device time, and device service information, such as device model, device ID, latitude and longitude. The platform logs include logs of user operations and cloud connection platforms. The dimensional data includes device grouping, administrative region division, etc. It also includes data resource information obtained from the government data platform, namely data table information, specific data fields, etc.

[0017] The data sharing module described in step S03 includes a data sharing front-end interaction and a data sharing intelligent library;

[0018] The data sharing front-end interaction is used to receive data usage requirement descriptions submitted by data-using units. The data usage requirement descriptions can be input into natural language descriptions or large models to obtain intelligent Q&A, which can generate the data tables, data fields and data verification rules required for the requirement, and input the data usage requirement descriptions into the data sharing intelligent library. Data-using units can view the data application results and government data returned by the data sharing intelligent library through the data sharing front-end interaction.

[0019] The data requirements mentioned include data requirements and requirements quality inspection rules;

[0020] The data sharing intelligent library is used to initiate data use requests to the government data platform and receive the data use request results and government data returned by the government data platform.

[0021] The data sharing intelligent library also includes a data sharing model, which is used to generate data tables, data fields, and rules for validating data based on the user's data requirements.

[0022] The specific steps for training the data sharing model include:

[0023] S301. Obtain optimization knowledge from shared data as sample data for the data sharing model;

[0024] S302. The sample data is split into keywords, and the split keywords are mapped to the metadata of the database through vector embedding. The text is converted into a high-dimensional vector, and a correlation coefficient p is introduced for each component of the vector.

[0025] S303, Using p-adjustment distance matching:

[0026] Similarity = |AB| p

[0027] Where A is the vector representation of the query text, B is the vector representation of the table or field, and |A| is calculated using p-based assignment:

[0028] |A| p =p -v(A)

[0029] Where v(A) is the highest degree of p in A;

[0030] S304. Construct a vector index using the HNSW or PQ algorithm, and retrieve the most relevant table structure and example based on the query vector:

[0031] TOP-k = argmin | query text - table field | p

[0032] Where TOP-k represents the top k results selected from the candidate table fields that are most similar to the query text, and argmin represents the parameter value corresponding to the minimum value;

[0033] S305. Combine the recalled table structure, sample, and query using number units into a Prompt template, and input the large model to generate an SQL query script.

[0034] The data quality inspection module in step S05 performs quality inspection on the government data, including data clarity, filtering, verification, standardization, and usability inspection rules, as well as the requirement inspection rules.

[0035] The specific steps for the data quality inspection module to extract the required quality inspection rules include:

[0036] S501. Obtain a sample containing natural language descriptions and their corresponding regular expressions;

[0037] S502. Create a quality inspection dataset. For each record: (Natural Language Description, Regular Expression);

[0038] S503. Use Byte-Pair Encoding (BPE) to segment the samples into word or sub-word units:

[0039]

[0040] Among them, Merge(w i ,w j ) represents the result of the merge operation, w i ,w j Representing two candidate merge units, i,j represent the indices of the candidate merge units, argmax represents the parameter value corresponding to the maximum value, and P reflects w i ,w j The probability of them appearing together in the context;

[0041] S504. Choose either T5 or BART, a Transformer-based self-attention mechanism model, to allow the model to focus on information at different locations when processing sequences.

[0042]

[0043] Where Attention represents the attention mechanism, Q represents the query vector that needs to be focused on, K represents the set of key vectors that are used to calculate similarity with the query vector Q, V represents the set of value vectors corresponding to the key K, softmax is a function that maps real vectors to probability distributions, and d k This represents the dimension of the key vector K;

[0044] S505. Use the cross-entropy loss function to evaluate the difference between the predicted results and the true labels:

[0045]

[0046] Where L represents the degree of difference between the model's predicted result and the true label, N represents the total number of samples in the dataset, i represents the sample index, and y represents the sample index. i p represents the true label of the i-th sample. i It is the probability distribution predicted by the model;

[0047] S506. Clean and optimize the generated regular expressions, check the syntax correctness of the generated regular expressions, and adjust them as needed;

[0048] S507. Use accuracy, precision, recall, and F1 score as evaluation metrics for the results of data extraction quality control rules:

[0049] Accuracy formula:

[0050]

[0051] Where True Positives represents the number of positive samples correctly predicted as positive by the model, TrueNegatives represents the number of negative samples correctly predicted as negative by the model, and Total Predictions represents the total number of predictions made by the model for all samples.

[0052] Precision formula:

[0053]

[0054] Wherein, False Positives represents the number of negative samples that are incorrectly predicted as positive samples;

[0055] Recall formula:

[0056]

[0057] Here, False Negatives represents the number of positive class samples that the model incorrectly predicts as negative class samples;

[0058] F1 score formula:

[0059]

[0060] The F1 score is the harmonic mean of precision and recall, used to comprehensively evaluate the model's performance in predicting the positive class.

[0061] This invention also provides an IoT data sharing and data quality inspection system for government affairs based on a large model, characterized by comprising:

[0062] Acquisition module: Used to acquire data from the device.

[0063] Storage module: used to store the device-side data to the server-side;

[0064] Interaction module: Data-using units can submit data requests to the government data platform through the data sharing module;

[0065] Application module: When a data application is approved, the government data platform transmits the government data to the data quality inspection module; when a data application is rejected, the government data platform returns the processing result to the data sharing module.

[0066] Quality inspection module: The data quality inspection module performs quality inspection on the government data to obtain quality-inspected government data;

[0067] The data entry module: The data quality inspection module enters the quality inspection government data into the data sharing module;

[0068] Transmission module: Transmits the quality inspection administrative data to the data sharing module via the server.

[0069] Viewing Module: Users can view the quality inspection administrative data through the data sharing module using data units.

[0070] The present invention also provides an apparatus comprising a processor, a memory, and a computer program stored in the memory and configured to be executed by the processor, wherein the processor, when executing the computer program, implements a method for IoT data sharing and data quality inspection based on a large model for government affairs as described in any one of claims 1 to 7.

[0071] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, wherein the computer program, when processed and executed, implements the steps of the IoT data sharing and data quality inspection method for government affairs based on a large model as described in any one of claims 1-7.

[0072] This invention provides a method, system, device, and storage medium for IoT data sharing and quality inspection based on a large-scale model for government affairs. Based on a question-and-answer large-scale model, the data sharing model generates the necessary data tables and fields. Users simply need to specify the data they need and its intended use through front-end interaction, and the data sharing model automatically generates standard data application content, thus lowering the threshold for data use by relevant departments. Furthermore, the data quality inspection model identifies and extracts specific quality inspection needs from users. It not only provides general basic data quality inspection standards and rules but also allows for the addition of rules based on specific data usage requirements. Users can quickly generate data quality inspection rules by inputting natural language, thereby improving the precision and accuracy of data quality inspection operations and ultimately achieving better data application in business. Based on large-scale model analysis, a unified object model can be specified first. This unified object model can maximize the inclusion of most IoT platforms, thus solving the problem of inconsistent data sources and allowing for greater openness of device models. Attached Figure Description

[0073] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0074] Figure 1 This is a schematic diagram illustrating the steps of the IoT data sharing and data quality inspection method based on a large model for government affairs.

[0075] Figure 2 This is the handling mechanism of PDCP and JBM in the case of packet loss in Example 2.

[0076] Figure 3 This is a flowchart of the PDCP receiving and processing logic in Example 3.

[0077] Figure 4 This is a flowchart of the optimized processing logic scheme for the PDCP receiving mechanism in Example 3.

[0078] Figure 5 This is a schematic diagram of the steps of the reordering strategy.

[0079] Figure 6 This is a schematic diagram of the IoT data sharing and data quality inspection system based on a large model for government affairs.

[0080] Figure 7 This is a screenshot of the Guangdong Province Internet of Things (IoT) business platform system interface. Detailed Implementation

[0081] To make the objectives, technical solutions, and advantages of the present invention clearer, the embodiments of the present invention will be further described in detail below with reference to the accompanying drawings:

[0082] This invention provides an embodiment of a method and system for IoT data sharing and data quality inspection based on a large model for government affairs. It should be noted that although the logical order is shown in the flowchart, under certain data conditions, the steps shown or described may be completed in a different order than that shown here.

[0083] Example 1: Refer to Figures 1-2

[0084] like Figure 1 As shown, this invention provides a method for IoT data sharing and data quality inspection based on a large model for government affairs, including:

[0085] S01, The IoT platform acquires data from the device;

[0086] S02. Store the device-side data on the server side;

[0087] S03. Data users of the Internet of Things platform can submit data requests to the government data platform through the data sharing module.

[0088] S04. When the data application is approved, the government data platform transmits the government data to the data quality inspection module. When the data application is not approved, the government data platform returns the processing result to the data sharing module.

[0089] S05. The data quality inspection module performs quality inspection on the government data to obtain quality-inspected government data.

[0090] S06. The data quality inspection module stores the quality inspection government data in the server.

[0091] S07. The quality inspection administrative data is transmitted to the data sharing module via the server.

[0092] S08. The data unit can view the quality inspection administrative data through the data sharing module.

[0093] By generating a unified basic object model for each device, based on a question-and-answer model, the threshold for data use by data-using departments is lowered. Users only need to directly state what kind of government data they need through front-end interaction, and the required data tables, data fields, and other content can be generated. At the same time, in order to enable the data extracted by data-using units to be better applied to business, data validation rules can be added based on specific data usage needs, or rules can be generated based on the large model. In this way, data-using units can also quickly generate data quality inspection rules through natural language, so that the final obtained data can be better applied to business.

[0094] The device-side data mentioned in step S01 includes device attributes, device events, and device service information of two or more sensors, such as water immersion sensors, smoke sensors, gas sensors, displacement sensors, high-definition cameras, and smart valves. Device attributes include device model, latitude and longitude, etc., and device time includes alarms, etc.

[0095] The server-side stored data in step S02 also includes server-side IoT hardware device attributes, event and service information, platform logs, dimensional data, and data resource information obtained from the government data platform. The hardware device attributes include device attributes, device time, and device service information, such as device model, device ID, latitude and longitude. The platform logs include logs of user operations and cloud connection platforms. The dimensional data includes device grouping, administrative region division, etc. It also includes data resource information obtained from the government data platform, namely data table information, specific data fields, etc.

[0096] The data sharing module described in step S03 includes a data sharing front-end interaction and a data sharing intelligent library;

[0097] The data sharing front-end interaction is used to receive data usage requirement descriptions submitted by data-using units. The data usage requirement descriptions can be input into natural language descriptions or large models to obtain intelligent Q&A, and can generate data tables, data fields and data verification rules required for the requirement. The data usage requirement descriptions are then input into the data sharing intelligent library. Data-using units can view the data application results and government data returned by the data sharing intelligent library through the data sharing front-end interaction.

[0098] The data requirements mentioned include data requirements and requirements quality inspection rules;

[0099] The data sharing intelligent library is used to initiate data use requests to the government data platform and receive the data use request results and government data returned by the government data platform.

[0100] The data sharing intelligent library also includes a data sharing model, which is used to generate data tables, data fields, and rules for validating data based on the user's data requirements.

[0101] For example, if a user's data request description is: "My organization needs to analyze the alarm data of smoke detectors in Guangzhou over the past 5 years," then the system can generate detailed information such as the device name, device ID, device model, alarm time, and alarm location for the device requesting table_yangan_device_data, as well as rule information for the past 5 years.

[0102] The data quality inspection module in step S05 performs quality inspection on the government data, including data clarity, filtering, verification, standardization, and usability inspection rules, as well as the requirement inspection rules.

[0103] like Figure 2 As shown, the basic technical architecture and process for implementing this solution include the following parts:

[0104] 1) Device side, including various IoT hardware devices such as smoke detectors, cameras, access control systems, etc. Devices contain information such as device attributes, events, and services.

[0105] 2) The server stores data, including device attributes, events, services, and other data, as well as user behavior data, administrative planning, grouping, and other dimensional attribute data of the IoT platform.

[0106] 3) Data Sharing Module: IoT platform users can apply for data from the government data platform through this module. This module, combined with large-scale intelligent models, helps users complete their requests more quickly. Ultimately, users also view the returned results through this module.

[0107] 4) Data quality inspection module: After receiving the returned data, it can perform quality checks on the data and display the results.

[0108] 5) Government data platform (one-network sharing platform) or data platform, which is a unified platform for all IoT data. Each IoT platform needs to apply for and obtain other data from the government data platform.

[0109] Step 1, Device side: Device's own attribute information, device action events, subscribed services. Different devices have different attributes, events, and services. Device attributes include device model, latitude and longitude, etc., and device events include alarms, etc.

[0110] Step 2, IoT platform obtains device information module get_device_mode: IoT platform obtains attribute, event, and service data uploaded by device through the network;

[0111] Step 3, server get_device_user_data

[0112] Information data such as attributes, events, and services of IoT hardware devices, such as device model, device ID, latitude and longitude, platform logs such as user operation logs, cloud docking platform logs, dimensional data such as device grouping, administrative region division, etc., also includes data resource information obtained from the government data platform, namely data table information, specific data fields, etc.

[0113] Step 4, based on the large model's data sharing module LLMS_mode, consists of two parts:

[0114] 1) IoT platform users can initiate data usage requests to the government data platform through the data sharing module. Data usage requests can be submitted manually by selecting and filling in the information or using large-scale intelligent models.

[0115] 2) After inputting the data resource information parameters from step 3 into the large model, the question-answering intelligence is obtained. Based on the user's description of needs, the data tables, data fields, and rules for validating the data can be generated.

[0116] Step 5, the data quality inspection module updates the rule Data Quality_model:

[0117] In addition to basic data cleaning, filtering, validation, and standardization to ensure the accuracy, integrity, consistency, timeliness, and availability of data, the quality inspection rules also include the requirement quality inspection rules generated in step 4.

[0118] Step 6, Government Data Middle Platform:

[0119] Data usage requests initiated from the data sharing module of the IoT platform are transmitted to the government data platform. If the request is approved, the data is returned from the data platform to the IoT platform; if it is not approved, the request result is returned to the IoT platform.

[0120] Step 7, the data quality inspection module verifies the data (Data Quality_model):

[0121] The data returned from the government data platform will be stored in the IoT platform after quality inspection. In addition to basic quality inspection, the application ID will be used to match the newly generated quality inspection rules for refined quality inspection.

[0122] Example 2: Refer to Figure 3 and Figure 4

[0123] like Figure 3 As shown, the data sharing model training is based on the P-adjustment (non-Euclidean distance) Natural Language to SQL (NL2SQL) algorithm principle, and the specific steps include:

[0124] S301. Obtain optimization knowledge from shared data as sample data for the data sharing model;

[0125] S302. The sample data is split into keywords, and the split keywords are mapped to the metadata of the database through vector embedding. The text is converted into a high-dimensional vector, and a correlation coefficient p is introduced for each component of the vector. The higher the correlation, the higher the power of p.

[0126] S303, Using p-adjustment distance matching:

[0127] Similarity = |AB| p

[0128] Where A is the vector representation of the query text, B is the vector representation of the table or field, and |A| is calculated using p-based assignment:

[0129] |A| p =p -v(A)

[0130] Where v(A) is the highest degree of p in A;

[0131] For example, if revenue is p^2·3, income statement is p^2·5, and cost is p^1·5, then...

[0132] |Revenue - Revenue Statement| p =p -2

[0133] |Revenue-Cost| p =p -1

[0134] That is, the income and income statement are more similar;

[0135] S304. Construct a vector index using the HNSW or PQ algorithm, and retrieve the most relevant table structure and example based on the query vector:

[0136] TOP-k = argmin | query text - table field | p

[0137] Where TOP-k represents the top k results selected from the candidate table fields that are most similar to the query text, and argmin represents the parameter value corresponding to the minimum value;

[0138] S305. Combine the recalled table structure, sample, and query using number units into a Prompt template, and input the large model to generate an SQL query script.

[0139] Prompt template example:

[0140] Table structure: CREATE TABLE sales(store_name VARCHAR, sales FLOAT, year INT);

[0141] Example: SELECT store_name FROM sales WHERE year=2023AND sales>1e6;

[0142] User query: Stores with sales exceeding 1 million in 2023 → SELECT store_name FROM salesWHERE year=2023AND sales>1000000.

[0143] Specific optimization knowledge for preparing shared data is used as sample data for training the shared data model, including:

[0144] 1) Basic data description: data overview, data source, data type, data attributes and attribute descriptions, etc.

[0145] 2) Basic processing rules: Formulas involved in derived data calculations, such as equipment failure rate over a certain period of time = number of faulty devices / total number of devices.

[0146] 3) Data statistical characteristics: distribution of data values, use of data, and description of relationships with other related data.

[0147] Compared to directly using large models to generate SQL script queries, using models trained with specific optimization knowledge to generate scripts results in higher accuracy and better meets business needs.

[0148] The following are examples of data sharing model applications:

[0149] 1) Sample data preparation

[0150] a. Basic data description:

[0151] The product table contains: id (primary key), name (product name), type (type), create_date (creation time).

[0152] The device table contains the following parameters: device(id: primary key, name: device name, product_id: associated product ID, status: device status, address: installation address, create_date: creation time).

[0153] b. Basic processing rules:

[0154] Equipment failure rate = Number of devices with a status value of 2 / Total number of devices

[0155] c. Statistical characteristics of the data:

[0156] Device status (0: offline, 1: online, 2: faulty)

[0157] 2) Fine-tune the model:

[0158] The samples were fine-tuned and trained based on the DeepSeek distillation model to obtain a small perceptual model.

[0159] 3) Use small perceptual models

[0160] [Question]: Please provide statistics on the equipment failure rate for the most recent month.

[0161] [Answer]: As Figure 4 As shown

[0162] By injecting sample data related to sensing devices into the model training process, we significantly improved the accuracy of the generated scripts. This application fully leverages the advantages of large language models in semantic parsing and logical reasoning, intelligently identifying key elements in user needs, such as data table structure, statistical rules, and output requirements, and automatically generating SQL query scripts that match the business scenario. This solution not only achieves a seamless connection from business requirements to technical implementation, but also enhances the accuracy of requirements through several aspects of the large model, reducing production illusions and providing users with an efficient and accurate data query solution, significantly lowering the technical threshold and implementation costs.

[0163] Example 3: Reference Figure 5

[0164] The data quality inspection module extracts requirement inspection rules based on a large model, converting pattern descriptions in natural language into corresponding regular expressions. Specific steps include:

[0165] S501. Obtain a sample containing natural language descriptions and their corresponding regular expressions;

[0166] S502. Create a quality inspection dataset. For each record: (natural language description, regular expression), for example, "email must conform to the standard format" -> [a-zA-Z0-9._%+-]+@[a-zA-Z0-9.-]+\.[a-zA-Z]{2,};

[0167] S503 uses Byte-Pair Encoding (BPE) to segment samples into word or sub-word units. Its core idea is to iteratively merge the most frequently occurring character pairs and remove stop words, punctuation marks, etc.

[0168]

[0169] Among them, Merge(w i ,w j ) represents the result of the merge operation, w i ,w j This represents two candidate merge units, where i and j are the indices of the candidate merge units, argmax represents the parameter value corresponding to the maximum value, and P reflects w. i ,w j The probability of them appearing together in the context;

[0170] S504. Choose either T5 or BART, a Transformer-based self-attention mechanism model, to allow the model to focus on information at different locations when processing sequences.

[0171]

[0172] Where Attention represents the attention mechanism, Q represents the query vector that needs to be focused on, K represents the set of key vectors that are used to calculate similarity with the query vector Q, V represents the set of value vectors corresponding to the key K, softmax is a function that maps real vectors to probability distributions, and d k This represents the dimension of the key vector K;

[0173] S505. Use the cross-entropy loss function to evaluate the difference between the predicted results and the true labels:

[0174]

[0175] Where L represents the degree of difference between the model's predicted result and the true label, N represents the total number of samples in the dataset, i represents the sample index, and y represents the sample index. i p represents the true label of the i-th sample. i It is the probability distribution predicted by the model;

[0176] S506. Clean and optimize the generated regular expressions, check the syntax correctness of the generated regular expressions, and adjust them as needed;

[0177] S507. Use accuracy, precision, recall, and F1 score as evaluation metrics for the results of data extraction quality control rules:

[0178] Accuracy formula:

[0179]

[0180] Where True Positives represents the number of positive samples correctly predicted as positive by the model, TrueNegatives represents the number of negative samples correctly predicted as negative by the model, and Total Predictions represents the total number of predictions made by the model for all samples.

[0181] Precision formula:

[0182]

[0183] Wherein, False Positives represents the number of negative samples that are incorrectly predicted as positive samples;

[0184] Recall formula:

[0185]

[0186] Here, False Negatives represents the number of positive class samples that the model incorrectly predicts as negative class samples;

[0187] F1 score formula:

[0188]

[0189] The F1 score is the harmonic mean of precision and recall, used to comprehensively evaluate the model's performance in predicting the positive class.

[0190] Suppose we input a natural language description in units of numbers: "Email addresses must conform to a standard format" as input:

[0191] 1. After segmenting the natural language description, the output is the following token ID sequence (simplified representation): InputTokens=[284,142,789,32098,1289,9876,1234,5678]

[0192] Each number represents the ID of a token;

[0193] 2. Embedding layer: Each token ID is mapped to a fixed-dimensional vector space.

[0194] Let the vocabulary size be V and the embedding dimension be d. model Then the embedding matrix is:

[0195]

[0196] For each tokenx i Its embedding vector is:

[0197]

[0198] Assume d model =512, then each token is represented as a vector of length 512;

[0199] 2. Location coding: Add location coding. The location coding formula is:

[0200]

[0201] For each position pos and dimension i, generate the corresponding position encoding vector. The final input is represented as:

[0202] x i =e i +PE(i)

[0203] 3. Encoder: For query, key, value, matrix Q, K, V, single-head attention is:

[0204]

[0205] Where d k It's a scaling factor; multi-head attention is the concatenation of multiple heads.

[0206] Multihead(Q,K,V)=Concat(head1,…,head h W O

[0207] Residual connections and layer normalization were added after each layer:

[0208] LayerNorm(x+Sublayer(x))

[0209] 4. Decoding: An autoregressive approach is used during generation, meaning only the next token is predicted each time.

[0210] The initial input is <s>(Starting marker):

[0211] y0 = Embedding( <s>)

[0212] Then predict in sequence:

[0213] P(y t |y ＜t ,X)=softmax(W O ·Decoder(y ＜t ,X))

[0214] 5. Output and Token Selection: After passing through the last layer of the decoder, the output is a logits vector.

[0215] z t =W O ·h t ∈R V

[0216] Then, the probability distribution is obtained by applying Softmax:

[0217] p t =softmax(z) t )

[0218] Choose the token with the highest probability:

[0219]

[0220] Repeat this process until the end-of-line character is encountered;

[0221] After all the above steps, the model outputs the following token sequence (abbreviated):

[0222] [\text{`[`},\text{`a`},\text{`-`},\text{`z`},\text{`A`},\text{`-`},\text{`Z`},\text{`0`},\text{`-`},\text{`9 `},\text{`.`},\text{`_`},\text{`%`},\text{`+`},\text{`-`},\text{`+`},\text{`]`},\text{`+`},\text{`@`},\dots]

[0223] The final combination is:

[0224] [a-zA-Z0-9._%+-]+@[a-zA-Z0-9.-]+\.[a-zA-Z]{2,}

[0225] Example 4: Reference Figure 6 and Figure 7

[0226] This invention also provides an IoT data sharing and data quality inspection system for government affairs based on a large model, characterized by comprising:

[0227] Acquisition module: Used to acquire data from the device.

[0228] Storage module: used to store the device-side data to the server-side;

[0229] Interaction module: Data-using units can submit data requests to the government data platform through the data sharing module;

[0230] Application module: When a data application is approved, the government data platform transmits the government data to the data quality inspection module; when a data application is rejected, the government data platform returns the processing result to the data sharing module.

[0231] Quality inspection module: The data quality inspection module performs quality inspection on the government data to obtain quality-inspected government data;

[0232] The data entry module: The data quality inspection module enters the quality inspection government data into the data sharing module;

[0233] Transmission module: Transmits the quality inspection administrative data to the data sharing module via the server.

[0234] Viewing Module: Users can view the quality inspection administrative data through the data sharing module using data units.

[0235] like Figure 6 As shown, the data sharing and data quality inspection processes mentioned in this system mainly involve hardware equipment, IoT platforms, government data platforms (unified data centers), and data-using units, etc., and the process logic is as follows:

[0236] 1. There are two ways to upload device data to IoT devices:

[0237] 1) The device is directly connected to the IoT platform, and the device's data is directly uploaded to the IoT platform;

[0238] 2) Other device platforms connect to the IoT platform via the platform cloud. Device data is first uploaded to the device platform and then transmitted to the IoT platform through the platform.

[0239] 2. Device data is stored on an IoT platform, and each IoT platform then uploads its own data to a globally unified data center, such as a government data platform.

[0240] 3. When users need to access data, it must undergo data sharing and data quality inspection:

[0241] 1) Initiate a data sharing request through the Internet of Things (IoT) platform;

[0242] 2) After the data application is approved and the data undergoes quality inspection and other steps, the data is returned to the IoT platform for the user to use.

[0243] In the original process, users needed to use functions such as resource catalogs to view and find the resources they needed to apply for when initiating a data sharing application. When the amount of data was large, it was quite difficult to find the required data quickly. Even after users found the required data and submitted the application, the data center staff still needed to manually configure and generate the response data based on the approved data application, and fill in various documents and materials. The data quality inspection also used basic and common methods.

[0244] Through the data sharing module and data quality inspection module of this system, the data usage cost for data-using departments is greatly reduced, and data quality inspection can also be generated according to needs, improving the accuracy of quality inspection.

[0245] like Figure 7 As shown, this system has been confirmed for use in the statistical analysis and business management module of version 5.7.0 of the Guangdong Provincial Internet of Things (IoT) Business Platform. The business platform is a business analysis and project management platform for IoT, containing IoT business data. Therefore, quickly finding data and generating rules can better assist business in conducting business analysis. In addition, it is also planned to be applied to the resource catalog module in the IoT sensing platform. Users can use the resource catalog function on the platform to view the affiliation, type, quantity, and other information of device data according to administrative divisions and industry units.< / s> < / s>

Claims

1. A method for IoT data sharing and quality inspection based on a large model for government affairs, characterized in that, include: S01, The IoT platform acquires data from the device; S02. Store the device-side data on the server side; S03. Data users of the Internet of Things platform can submit data requests to the government data platform through the data sharing module. The data sharing module described in step S03 includes a data sharing front-end interaction and a data sharing intelligent library; The data sharing front-end interaction is used to receive the data usage requirement description submitted by the data-using unit and input the data usage requirement description into the data sharing intelligent library. The data-using unit can view the data application results and government data returned by the data sharing intelligent library through the data sharing front-end interaction. The data requirements mentioned include data requirements and requirements quality inspection rules; The data sharing intelligent library is used to initiate data use requests to the government data platform and receive the data use request results and government data returned by the government data platform. The data sharing intelligent library also includes a data sharing model, which is used to generate data tables, data fields, and rules for validating data according to the data requirements. The specific steps for training the data sharing model include: S301. Obtain optimization knowledge from shared data as sample data for the data sharing model; The optimization knowledge includes: basic data description: data overview, data source, data type, data attributes and attribute descriptions; Basic processing rules: Formulas involved in data calculation; Data statistical characteristics: the distribution of data values, the purpose of the data, and the description of the relationship between related data; S302. The sample data is split into keywords, and the split keywords are mapped to the metadata of the database through vector embedding. The text is converted into a high-dimensional vector, and a correlation coefficient p is introduced for each component of the vector. S303, Using p-adjustment distance matching: in, The vector representation of the query text. It is a vector representation of a table field. Calculations are performed using p-based assignment: in, for middle The highest number of times; S304. Construct a vector index using the HNSW or PQ algorithm, and retrieve the most relevant table structure and example based on the query vector: in, This represents the k results selected from the candidate table fields that are most similar to the query text, and argmin represents the parameter value corresponding to the minimum value. S305. Combine the recalled table structure, sample, and query using number units into a Prompt template, input the large model to generate an SQL query script; S04. When the data application is approved, the government data platform transmits the government data to the data quality inspection module. When the data application is not approved, the government data platform returns the processing result to the data sharing module. S05. The data quality inspection module performs quality inspection on the government data to obtain quality-inspected government data. S06. The data quality inspection module stores the quality inspection government data in the server. S07. The quality inspection administrative data is transmitted to the data sharing module via the server. S08. The data unit can view the quality inspection administrative data through the data sharing module.

2. The method for IoT data sharing and data quality inspection based on a large model for government affairs, as described in claim 1, is characterized in that... The device-side data mentioned in step S01 includes device attributes, device events, and device service information of two or more sensors, such as water immersion sensors, smoke sensors, gas sensors, displacement sensors, high-definition cameras, and smart valves.

3. The method for IoT data sharing and data quality inspection based on a large model for government affairs, as described in claim 1, is characterized in that... The server-side stored data in step S02 also includes server-side IoT hardware device attributes, events and service information, platform logs, dimensional data, and data resource information obtained from the government data platform.

4. The method for IoT data sharing and data quality inspection based on a large model for government affairs, as described in claim 3, is characterized in that... The data quality inspection module in step S05 performs quality inspection on the government data, including data clarity, filtering, verification, standardization, and usability inspection rules, as well as the requirement inspection rules.

5. The method for IoT data sharing and data quality inspection based on a large model for government affairs, as described in claim 4, is characterized in that... The specific steps for the data quality inspection module to extract the required quality inspection rules include: S501. Obtain a sample containing natural language descriptions and their corresponding regular expressions; S502. Create a quality inspection dataset, for each record: ; S503. Use Byte-Pair Encoding (BPE) to segment the samples into word or sub-word units: in, This indicates the result of the merge operation. This represents a merge unit that represents two candidates. Indicates the index of the candidate merge unit. This indicates the parameter value corresponding to the maximum value. reflect The probability of them appearing together in the context; S504. Choose either T5 or BART, a Transformer-based self-attention mechanism model, to allow the model to focus on information at different locations when processing sequences. in, This represents the attention mechanism. This represents the query vector that needs to be focused on at present. Indicates the current and query vectors The set of key vectors used for similarity calculation. Represents AND key The corresponding set of value vectors; softmax is a function that maps real number vectors to probability distributions. Represents the key vector The dimension; S505. Use the cross-entropy loss function to evaluate the difference between the predicted results and the true labels: in, This indicates the degree of difference between the model's predicted results and the true labels. This represents the total number of samples in the dataset. Indicates the sample index. Indicates the first The true label of each sample It is the probability distribution predicted by the model; S506. Clean and optimize the generated regular expressions, check the syntax correctness of the generated regular expressions, and adjust them as needed; S507. Use accuracy, precision, recall, and F1 score as evaluation metrics for the results of data extraction quality control rules: Accuracy formula: in, This indicates the number of positive samples that the model correctly predicted as positive. This indicates the number of negative samples that the model correctly predicted as negative. This represents the total number of predictions made by the model for all samples; Precision formula: in, This indicates the number of negative samples that were incorrectly predicted as positive. Recall formula: in, This indicates the number of positive class samples that the model incorrectly predicted as negative class samples; F1 score formula: The F1 score is the harmonic mean of precision and recall, used to comprehensively evaluate the model's performance in predicting the positive class.

6. A data sharing and data quality inspection system for government affairs based on a large model, as described in claim 1, characterized in that: include: Acquisition module: Used to acquire data from the device. Storage module: used to store the device-side data to the server-side; Interaction module: Data-using units can submit data requests to the government data platform through the data sharing module; Application module: When a data application is approved, the government data platform transmits the government data to the data quality inspection module; when a data application is rejected, the government data platform returns the processing result to the data sharing module. Quality inspection module: The data quality inspection module performs quality inspection on the government data to obtain quality-inspected government data; The data entry module: The data quality inspection module enters the quality inspection government data into the data sharing module; Transmission module: Transmits the quality inspection administrative data to the data sharing module via the server. Viewing Module: Users can view the quality inspection administrative data through the data sharing module using data units.

7. A device, characterized in that, The method includes a processor, a memory, and a computer program stored in the memory and configured to be executed by the processor, wherein the processor executes the computer program to implement a method for IoT data sharing and data quality inspection based on a large model for government affairs, as described in any one of claims 1 to 6.

8. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is processed and executed, it implements the steps of the IoT data sharing and data quality inspection method for government affairs based on a large model as described in any one of claims 1-5.

Images