Structured data stratified sampling method, system and equipment based on security data sandbox and medium

Abstract

The invention provides a structured data stratified sampling method, system and device based on a security data sandbox and a medium, and belongs to the technical field of data analys.The method comprises the steps that the security data sandbox is used for being in butt joint with a data source of a data provider, and metadata information of the data source is determined; determining a layering field in response to the layering request, determining layering and a sampling strategy of each layer according to the layering field in the security data sandbox, and allowing the sampling strategy of each layer to be independently modified; stratified sampling is carried out on the data source in the security data sandbox according to the sampling strategy of each layer, and then sampling data of each layer is merged; and training a data analysis model in the security data sandbox by using the merged sampling data, and analyzing the total data of the data source by using the data analysis model. Through stratified sampling, the accuracy and efficiency of data analysis are improved, and the data processing cost is reduced; and the sampling strategy of each layer is allowed to be independently modified, so that the flexibility and adaptability are improved.

Description

Technical Field

[0001] The present application belongs to the field of data analysis technology, and specifically relates to a method, system, device and medium for structured data stratified sampling based on a secure data sandbox. Background Art

[0002] With the advancement of digitalization, the amount of data accumulated across various industries has exploded. In many fields, such as finance, healthcare, and e-commerce, data has become a crucial basis for business decision-making and innovation. However, faced with this massive amount of data, extracting valuable information efficiently and accurately has become a pressing challenge.

[0003] Sampling technology is a method for selecting representative samples from large amounts of data, and it plays a vital role in data analysis. Stratified sampling is a key branch of sampling technology. By dividing the overall sample into different strata based on attribute characteristics and then randomly selecting sample units from each stratum, it can effectively improve the representativeness of the sample and reduce the time and cost of data processing. However, existing stratified sampling methods have many problems in practical applications. First, in terms of sample allocation, the rationality of the sample data in each stratum cannot be ensured, resulting in samples from some strata not accurately reflecting the characteristics of that stratum, which in turn affects the representativeness of the overall sample. Second, there is a lack of effective means to control sampling errors, making it difficult to guarantee the accuracy of sampling results. Third, there is the problem of low sampling efficiency. When processing massive amounts of data, it consumes a lot of time and resources and cannot meet the needs of rapid decision-making.

[0004] Furthermore, security issues are becoming increasingly prominent during data analysis. Enterprises must ensure data security when conducting data analysis. However, traditional stratified sampling methods fail to fully consider data security factors and fail to provide enterprises with a secure and efficient data analysis environment. Therefore, there is an urgent need to develop a stratified sampling method that can improve sample representativeness and abstraction efficiency while ensuring data security. Summary of the Invention

[0005] In a first aspect, an embodiment of the present application provides a method for stratified sampling of structured data based on a secure data sandbox, comprising the following steps: S1. Use the secure data sandbox to connect to the data provider's data source and determine the metadata information of the data source; S2. Respond to the stratification request to determine the stratification fields, determine the stratification and sampling strategy for each stratum based on the stratification fields within the secure data sandbox, and allow the sampling strategy for each stratum to be modified separately; S3. Perform stratified sampling of data sources within the secure data sandbox according to the sampling strategy for each layer, and then merge the sampled data for each layer; S4. Use the combined sampled data to train a data analysis model within a secure data sandbox, and then use the data analysis model to analyze the full data source. Stratified sampling improves the accuracy and efficiency of data analysis and reduces data processing costs. Allowing the sampling strategy for each stratum to be modified individually increases the flexibility and adaptability of the method, ensuring data security and privacy during processing.

[0006] Furthermore, the specific steps of step S1 are as follows: S11. Obtain data source information in the secure data sandbox and determine the data source type; If the data source is a database, proceed to step S12; If the data source is a file type, go to step S14; S12. Perform a connection test on a database-type data source in a secure data sandbox; If the test passes, proceed to step S15; If the test fails, proceed to step S13; S13. Modify the data source information and return to step S12; S14. Parse the data source of the file type; S15. Obtain metadata information and save it to the secure data sandbox database; the metadata information includes fields. By differentiating database and file type data sources, the secure data sandbox is compatible with multiple data source formats, improving versatility and adaptability.

[0007] Furthermore, the specific steps of step S14 are as follows: S141. The file name of the data source of the file type is used as the library name; S142. Determine the file type of the file data source; If it is an Excel file, the sheet name is used as the table name; If it is a CSV file, use the file name as the table name; S143. Use the column name of the file type data source as the field name; The metadata information in step S15 also includes the library, table, and number of data rows. By clarifying the determination method of the library name, table name, and field name, a basis is provided for accurately obtaining the metadata information of the file type data source, ensuring the accuracy of the metadata information.

[0008] Furthermore, the specific steps of step S2 are as follows: S21. Respond to the hierarchical request and obtain the selected hierarchical field; S22. Query the data source in the secure data sandbox based on the selected layer field to determine the number of layers M and the amount of data C per layer i, and feedback to the data provider; S23. respond and receive data provider according to the data volume of each layer C i Set sampling parameters; If the sampling parameter is the number of sampling samples N, proceed to step S24; If the sampling parameter is the sampling ratio R, proceed to step S25; S24. Query the full amount of data S of the data source in the secure data sandbox, calculate the sampling ratio R using the number of sampled samples N / full amount of data S, and proceed to step S3; S25. Obtain the data volume C of each layer in the secure data sandbox i And the sampling ratio R calculates the number of sampling samples N in each layer i , for the number of sampling samples N i Mark the layers that do not meet the requirements; S26. Modify the sampling strategy for each marked stratum individually, setting a new number of samples or sampling ratio until the requirements are met. By involving data providers in setting sampling parameters and performing corresponding calculations and adjustments based on different parameters, flexible sampling strategy settings are achieved, improving the accuracy of sample data.

[0009] Furthermore, the specific steps of step S26 are as follows: S261. Determine new sample sampling strategy for marker stratification; If we set a new sampling number N i , proceed to step S262; If a new sampling ratio R is set i , proceed to step S263; S262. Use the new sampling number N i And the data volume C of this layer i Calculate the sampling ratio R of this layer i =Number of samples in this layer N i / The amount of data in this layer C i , proceed to step S3; S263. Use the new sampling ratio R i And the data volume C of this layer i Calculate the number of samples N for this layer i ; S264. Determine the number of samples N for this layer i Whether there is demand; If yes, go to step S3; If not, the layer is marked and the process returns to step S261. By continuously adjusting the sampling strategy, it is ensured that the sampling results of each layer can meet the actual needs, thereby improving the quality of the sample.

[0010] Furthermore, the specific steps of step S3 are as follows: S31. According to the amount of data per layer C i and the sampling ratio R of each layer i Calculate the final number of samples N for each layer i = Data volume per layer C i × sampling ratio of each layer R i ; S32. In the secure data sandbox, the number of samples N at each layer is sampled. i Randomly sample each layer of the data source to obtain sampling data for each layer; S33. Combine the sampling data of each layer to obtain the sampling results. By calculating the number of samples in each layer and performing random sampling and data merging, the rationality of the sampling process is ensured and the reliability of the sampling results is improved.

[0011] Furthermore, the specific steps of step S4 are as follows: S41. Upload the sampling data in the sampling results to the debugging environment of the secure data sandbox; S42. The data user obtains sample data from the debugging environment of the secure data sandbox and uses the sample data to train the data analysis model until the data analysis model meets the requirements; S43. Publish the data analysis model to the operating environment of the secure data sandbox and use the data analysis model to analyze the full amount of data from the data source; S44. Respond to requests from data providers to view full data analysis results, and grant data users access to the full data analysis results. By conducting model training and full data analysis in the debugging and runtime environments of the secure data sandbox, and providing feedback to data providers, effective data utilization is achieved and the efficiency of data analysis is improved.

[0012] In a second aspect, an embodiment of the present application further provides a system for stratified sampling of structured data based on a secure data sandbox, comprising: The data source access module is used to connect to the data provider's data source using the secure data sandbox and determine the metadata information of the data source; A layer strategy setting module is used to determine the layer fields in response to the layer request, determine the layers and the sampling strategy for each layer based on the layer fields within the secure data sandbox, and allow the sampling strategy for each layer to be modified separately; The sample allocation and extraction module is used to perform stratified sampling of data sources within the secure data sandbox according to the sampling strategy of each layer, and then merge the sampled data of each layer; The data analysis and feedback module is used to use the merged sample data to train the data analysis model in the secure data sandbox, and then use the data analysis model to analyze the full data of the data source.

[0013] In a third aspect, an embodiment of the present application further provides an electronic device comprising a memory, a processor, and a computer program stored on the memory and executable on the processor. When the processor executes the program, the steps of the structured data stratified sampling method based on the secure data sandbox as described in the first aspect are implemented.

[0014] In a fourth aspect, an embodiment of the present application further provides a storage medium on which a computer program is stored. When the computer program is executed by a processor, the steps of the structured data stratified sampling method based on the secure data sandbox as described in the first aspect are implemented.

[0015] It can be seen from the above technical solutions that this application has the following advantages: The structured data stratified sampling method, system, device and medium based on the security data sandbox provided in this application perform stratified sampling by modifying the sampling strategy, thereby improving the representativeness of the sample data in the data security sandbox and providing the accuracy of the data user model; in the data security sandbox environment, various complex scenarios and situations may be encountered. Through this stratified sampling, the population can be stratified according to different characteristics and attributes, so as to simulate and test complex scenarios more flexibly; through stratified sampling in the data security sandbox, the development of sandbox functions can be made more efficient, while also reducing the cost of subsequent maintenance and upgrades. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] In order to more clearly illustrate the technical solution of the present application, the following is a brief introduction to the drawings required for the description. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0017] Figure 1 Schematic diagram of the process of the structured data stratified sampling method based on the secure data sandbox of the present invention.

[0018] Figure 2 Schematic diagram of the process of the structured data stratified sampling system based on the secure data sandbox of the present invention. DETAILED DESCRIPTION

[0019] The specific steps of the structured data stratified sampling method based on the secure data sandbox will be described in detail below, and various embodiments of the present disclosure will be described more fully. The present disclosure can have various embodiments, and adjustments and changes can be made therein. However, it should be understood that there is no intention to limit the various embodiments of the present disclosure to the specific embodiments disclosed herein, but rather that the present disclosure should be understood to cover all adjustments, equivalents, and / or alternatives that fall within the spirit and scope of the various embodiments of the present disclosure.

[0020] For example, driven by today's digital technologies, the amount of data accumulated across various industries is expanding at an astonishing rate. Across sectors like finance, healthcare, and e-commerce, data has become a critical enabler for business decision-making and innovation. However, faced with this massive amount of data, efficiently and accurately sifting through it for truly valuable information remains a challenge.

[0021] Sampling technology, as a method for selecting representative samples from massive amounts of data, plays a crucial role in data analysis. Stratified sampling, a key branch of sampling technology, significantly improves sample representativeness by dividing the overall sample into different strata based on attribute characteristics and then randomly selecting sample units from each stratum, effectively reducing the time and cost of data processing. However, in practical applications, existing stratified sampling methods have exposed many shortcomings.

[0022] First, during the sample allocation process, existing methods struggle to ensure the rationality of sample data across all strata. This results in samples from some strata failing to accurately reflect their true characteristics, negatively impacting the representativeness of the overall sample. Secondly, the lack of effective means to control sampling errors makes it difficult to fully guarantee the accuracy of sampling results. Furthermore, there's the issue of low sampling efficiency, particularly when processing massive amounts of data. This consumes significant time and resources, making it difficult to meet the demands of businesses for rapid decision-making.

[0023] Furthermore, data security issues have become increasingly prominent during data analysis. Enterprises must ensure data security when conducting data analysis. However, traditional stratified sampling methods, originally designed without adequate consideration of this critical factor, fail to provide enterprises with a secure and efficient data analysis environment. Therefore, there is an urgent need to develop a new stratified sampling method that not only improves sample representativeness and efficiency, but also effectively ensures data security.

[0024] To address the above issues, this embodiment provides a structured data stratified sampling method based on a secure data sandbox. Through stratified sampling, the accuracy and efficiency of data analysis are improved, and the cost of data processing is reduced. The sampling strategy of each layer can be modified separately, which increases the flexibility and adaptability of the method and ensures the security and privacy protection of data during the processing process.

[0025] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0026] See also Figure 1 FIG. 1 is a flowchart of a structured data stratified sampling method based on a secure data sandbox in a specific embodiment, the method comprising the following steps: S1. Use the secure data sandbox to connect to the data provider's data source and determine the metadata information of the data source; It should be noted that by connecting to multiple data sources through the secure data sandbox, centralized management and unified scheduling of data sources can be achieved; by obtaining metadata information of the data source, a data basis is provided for subsequent stratified sampling, ensuring the accuracy and reliability of the sampling process; S2. Respond to the stratification request to determine the stratification fields, determine the stratification and sampling strategy for each stratum based on the stratification fields within the secure data sandbox, and allow the sampling strategy for each stratum to be modified separately; It should be noted that the stratification fields are flexibly selected according to data characteristics and user needs to ensure the rationality of stratification; users are allowed to modify the sampling strategy of each layer separately, which enhances the adaptability and flexibility of the system and can better meet the sampling needs in different scenarios; S3. Perform stratified sampling of data sources within the secure data sandbox according to the sampling strategy for each layer, and then merge the sampled data for each layer; It should be noted that stratified sampling is carried out according to the optimized sampling strategy to improve the efficiency of sampling; the sampling data of each layer are merged to form a complete sampling result, which is convenient for subsequent data analysis and processing; S4. Use the combined sample data to train a data analysis model in the secure data sandbox, and then use the data analysis model to analyze the full data source; It should be noted that the data analysis model is trained by sampling data to verify the accuracy of the model; the trained model is applied to the full amount of data to obtain comprehensive analysis results and provide data support for subsequent decision-making.

[0027] This embodiment optimizes the entire process from data source access to data analysis, improving the efficiency and security of data processing.

[0028] Furthermore, as a refinement and extension of the specific implementation of the above embodiment, in order to fully illustrate the specific implementation process in this embodiment, another structured data stratified sampling method based on a secure data sandbox is provided, which includes the following steps: S1. Use the secure data sandbox to connect to the data provider's data source and determine the metadata information of the data source. For example, the data provider provides a database containing customer information and an Excel file containing sales records. The specific steps of step S1 are as follows: S11. Obtain data source information in the secure data sandbox and determine the data source type; If the data source is a database, proceed to step S12; If the data source is a file type, go to step S14; For example, the data source information is obtained in the secure data sandbox and two data sources are found: a MySQL database containing customer information and an Excel file containing sales records; Determine the data source type: if it is a MySQL database, go to step S12; if it is an Excel file, go to step S14; S12. Perform a connection test on a database-type data source in a secure data sandbox; If the test passes, proceed to step S15; If the test fails, proceed to step S13; S13. Modify the data source information and return to step S12; For example, try to connect to the MySQL database in the secure data sandbox to test whether the connection is successful; If the connection fails, for example, the database address is incorrect or the permissions are insufficient, proceed to step S13 to modify the data source information, such as correcting the database address or adjusting the permission settings, and then return to step S12 to retest.

[0029] If the connection is successful, go to step S15; S14. Parse the data source of the file type; For example, the Excel file is parsed and the file name "Sales_Data.xlsx" is used as the library name; if the file type is Excel, each sheet page name, such as "2024_Q1" and "2024_Q2", is used as the table name; and each sheet page column name, such as "Order Number", "Customer ID", and "Sales Amount", is used as the field name; S15 obtains metadata information and saves it to the database of the secure data sandbox; the metadata information includes fields; For example, for a MySQL database, metadata information is obtained, including the database name, table name, field name, and number of data rows; For Excel files, obtain metadata information, including the library name as the file name, the table name as the sheet name, the field name as the column name, and the number of data rows; Save these metadata information in the database of the secure data sandbox to facilitate subsequent operations; It should be noted that, after step S15, the following steps are also included: S16. Regularly verify metadata consistency through the secure data sandbox and trigger an alarm when changes to the data source structure are detected; It should be noted that, compared to ETL tools, the secure data sandbox has the ability to dynamically monitor the consistency of data source structure and metadata; S2. Respond to the stratification request and determine the stratification fields. Within the secure data sandbox, determine the stratification and sampling strategy for each stratum based on the stratification fields. The sampling strategy for each stratum can be modified individually. For example, let's assume that a user needs to perform stratified sampling of customer data based on the "region" field. The specific steps in step S2 are as follows: S21. Respond to the hierarchical request and obtain the selected hierarchical field; For example, the user selects "region" as the hierarchical field in the interactive interface; S22. Query the data source in the secure data sandbox based on the selected layer field to determine the number of layers M and the amount of data C per layer i , and feedback to the data provider; For example, by querying the data source based on the "Region" field, three regions are found: Region A, Region B, and Region C. The data volume for each region is: 1,000 records for Region A, 1,500 records for Region B, and 2,000 records for Region C. This information is fed back to the data provider. S23. respond and receive data provider according to the data volume of each layer C i Set sampling parameters; If the sampling parameter is the number of sampling samples N, proceed to step S24; If the sampling parameter is the sampling ratio R, proceed to step S25; For example, the data provider decides to sample 100 records from region A, 150 records from region B, and 200 records from region C based on the amount of data in each layer; these sampling parameters are received through the secure data sandbox; S24. Query the full amount of data S of the data source in the secure data sandbox, calculate the sampling ratio R using the number of sampled samples N / full amount of data S, and proceed to step S3; For example, query the full data set S = 4500 records in the secure data sandbox and calculate the sampling ratio for each region: Region A: 100 / 4500≈0.0222 Region B: 150 / 4500≈0.0333 Region C: 200 / 4500≈0.0444 S25. Obtain the data volume C of each layer in the secure data sandbox i And the sampling ratio R calculates the number of sampling samples N in each layer i , for the number of sampling samples N i Mark the layers that do not meet the requirements; For example, the number of sampling samples for each layer is calculated based on the data volume and sampling ratio of each layer: Region A: 1000×0.0222≈100 Region B: 1500×0.0333≈150 Region C: 2000×0.0444≈200 Assuming that the amount of data in region C is large and 200 samples cannot meet the demand, the stratification of region C is marked. S26. Modify the sampling strategy for each marked layer individually, setting a new number of samples or sampling ratio until the requirements are met; For example, due to the large amount of data in region C, 200 samples still cannot meet the needs. After marking, more detailed analysis is required. The data provider decides to adjust the sampling ratio of region C to 0.05; Recalculate the number of samples in area C according to the new sampling ratio: Region C: 2000×0.05=100 If the new number of samples still does not meet the demand, continue to adjust the sampling strategy until the demand is met; It should be noted that in the secure data sandbox, the calculation of the sampling ratio R is set to satisfy ∑(C i ×R i )≤sandbox memory threshold, achieving mandatory minimum sample size guarantee; S3. Perform stratified sampling of data sources within the secure data sandbox according to the sampling strategy for each layer, and then merge the sampled data for each layer; the specific steps of step S3 are as follows: S31. According to the amount of data per layer C i and the sampling ratio R of each layer i Calculate the final number of samples N for each layer i = Data volume per layer C i × sampling ratio of each layer R i ; For example, calculate the final number of samples for each layer: Region A: 1000 × 0.0222 = 100 Region B: 1500 × 0.0333 = 150 Region C: 2000×0.05=100 S32. In the security data sandbox, the number of samples N of each layer is sampled. i Randomly sample each layer of the data source to obtain sampling data of each layer; For example, 100 records are randomly selected from region A, 150 records are randomly selected from region B, and 100 records are randomly selected from region C; It should be noted that the sampling process is completely carried out in the encrypted memory area of the secure data sandbox, and the original data cache is immediately erased after the sampling is completed; S33. Merge the sampling data of each layer to obtain the sampling results; For example, the sampling data of three regions A, B, and C are merged to form a sampling result containing 350 records; It should be noted that step S3 also includes the following steps: S34. Calculate the hash value of the sampling strategy and store it in conjunction with the sampling result; S4. Use the combined sample data to train a data analysis model in the secure data sandbox, and then use the data analysis model to analyze the full amount of data from the data source. The specific steps of step S4 are as follows: S41. Upload the sampling data in the sampling results to the debugging environment of the secure data sandbox; Exemplarily, the merged 350 sample data are uploaded to the debugging environment of the secure data sandbox; S42. The data user obtains sample data from the debugging environment of the secure data sandbox and uses the sample data to train the data analysis model until the data analysis model meets the requirements; For example, a data user obtains sample data from a debugging environment and uses the sample data to train a data analysis model for predicting customer purchasing behavior. Assume that the model training results show that the accuracy rate reaches 90%, which meets the requirements; S43. Publish the data analysis model to the operating environment of the secure data sandbox and use the data analysis model to analyze the full amount of data from the data source; For example, the trained model is published to the operating environment of a secure data sandbox. The model is used to analyze the entire data set, i.e., 4,500 records, to predict the purchasing behavior of each customer. S44. Respond to the data provider's request to view the full data analysis results, and grant the data user access to the full data analysis results; the access rights include at least one of downloading the desensitized aggregated results, viewing the visualization report online, or exporting model parameters; For example, data providers can view the full data analysis results, such as the predicted purchase behavior distribution; data users can download or use these analysis results according to authorization to provide a basis for subsequent marketing strategies; It should be noted that the debugging environment of the secure data sandbox is physically isolated from the operating environment, and the parameters of the data analysis model are transmitted through an encrypted channel for transmission; Step S4 also includes the following steps: S45. Compare the accuracy difference between the data analysis model and the full data analysis results and the actual results, and trigger resampling when the difference exceeds the threshold.

[0030] In an embodiment of the present invention, based on step S14, a possible embodiment will be given below to illustrate its specific implementation scheme in a non-limiting manner.

[0031] The specific steps of step S14 are as follows: S141. The file name of the data source of the file type is used as the library name; S142. Determine the file type of the file data source; If it is an Excel file, the sheet name is used as the table name; If it is a CSV file, use the file name as the table name; S143. Use the column name of the file type data source as the field name; The metadata information in step S15 also includes the library, table, and number of data rows; For example, the data provider provides an Excel file named "Sales_Data.xlsx" containing two sheets: "2024_Q1" and "2024_Q2". The file name "Sales_Data.xlsx" is used as the library name, the file type is determined to be Excel, the sheet "2024_Q1" is used as the table name, and the sheet "2024_Q2" is used as the table name. The column names of the table "2024_Q1" are set to "Order Number", "Customer ID", and "Sales Amount" as the field names, and the column names of the table "2024_Q2" are set to "Order Number", "Customer ID", and "Sales Amount" as the field names. It should be noted that step S14 also includes the following steps: S144. Automatically generate hashed field identifiers for files with missing column names.

[0032] In an embodiment of the present invention, based on step S26, a possible embodiment will be given below to illustrate its specific implementation scheme in a non-limiting manner.

[0033] The specific steps of step S26 are as follows: S261. Determine new sample sampling strategy for marker stratification; If we set a new sampling number N i , proceed to step S262; If a new sampling ratio R is set i , proceed to step S263; S262. Use the new sampling number N i And the data volume C of this layer i Calculate the sampling ratio R of this layer i =Number of samples in this layer N i / The amount of data in this layer C i , proceed to step S3; S263. Use the new sampling ratio R i And the data volume C of this layer i Calculate the number of samples N for this layer i ; S264. Determine the number of samples N for this layer i Whether there is demand; If yes, go to step S3; If not, mark the layer and return to step S261; For example, if in step S25, the number of sampling samples in region C, 200, still cannot meet the demand, the sampling strategy needs to be adjusted; The data provider decided to adjust the sampling ratio of region C to 0.05; Use the new sampling number N i =100 and data volume C i =2000 Calculate the new sampling ratio R i : R i =N i / C i =100 / 2000=0.05; Use the new sampling ratio R i = 0.05 and the data volume C i =2000 Calculate the new sample size N i : N i =C i ×R i =2000×0.05=100; If the new sampling number of 100 still does not meet the requirement, mark the layer.

[0034] It should be noted that the secure data sandbox environment is implemented through the following steps: A trusted execution environment is built using the Intel SGX framework, and all sampling policy modification operations are recorded through the blockchain.

[0035] It should be understood that the size of the serial numbers of the steps in the above embodiments does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0036] like Figure 2 As shown, the following is an embodiment of the system for stratified sampling of structured data based on a secure data sandbox provided by an embodiment of the present disclosure. The system and the method for stratified sampling of structured data based on a secure data sandbox in the above-mentioned embodiments belong to the same inventive concept. For details not fully described in the embodiment of the system for stratified sampling of structured data based on a secure data sandbox, please refer to the embodiment of the method for stratified sampling of structured data based on a secure data sandbox.

[0037] The system includes: The data source access module is used to connect to the data provider's data source using the secure data sandbox and determine the metadata information of the data source; A layer strategy setting module is used to determine the layer fields in response to the layer request, determine the layers and the sampling strategy for each layer based on the layer fields within the secure data sandbox, and allow the sampling strategy for each layer to be modified separately; The sample allocation and extraction module is used to perform stratified sampling of data sources within the secure data sandbox according to the sampling strategy of each layer, and then merge the sampled data of each layer; The data analysis and feedback module is used to use the merged sample data to train the data analysis model in the secure data sandbox, and then use the data analysis model to analyze the full data of the data source.

[0038] This embodiment realizes the automation of the stratified sampling method and improves the stability of the system through the coordinated work of the data source access module, the stratified strategy setting module, the sample allocation and extraction module, and the data analysis and feedback module.

[0039] The method for stratified sampling of structured data based on a secure data sandbox provided in an embodiment of the present application can be applied to electronic devices. Those skilled in the art will appreciate that the electronic device structure involved in the embodiments of the present invention does not constitute a limitation on the electronic device, and the electronic device may include more or fewer components than shown, or combine certain components, or arrange components differently. In an embodiment of the present invention, electronic devices include but are not limited to laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the embodiments of the present application described and / or required herein.

[0040] The electronic device may include a processor, an external memory interface, an internal memory, a universal serial bus (USB) interface, a charging management module, a power management module, a battery, a wireless communication module, an audio module, a speaker, a microphone, a sensor module, a button, a camera, a display, and a SIM card interface, etc.

[0041] It is understood that the structures illustrated in the embodiments of the present application do not constitute specific limitations on the electronic device. In other embodiments of the present application, the electronic device may include more or fewer components than shown, or combine or separate certain components, or arrange the components differently. The illustrated components may be implemented in hardware, software, or a combination of software and hardware.

[0042] A processor may include one or more processing units, such as a central processing unit (CPU), an application processor (AP), a modem processor, a graphics processing unit (GPU), an image signal processor (ISP), a controller, a memory, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural-network processing unit (NPU). Different processing units may be independent devices or integrated into one or more processors.

[0043] The processor can be the nerve center and command center of the electronic device. The controller can generate operation control signals based on the instruction opcode and timing signal to complete the control of instruction fetching and execution.

[0044] The processor may also include a memory for storing instructions and data. In some embodiments, the memory in the processor is a cache memory. This memory can store instructions or data that the processor has just used or is reusing. If the processor needs to use the instruction or data again, it can directly call it from the memory. This avoids repeated accesses, reduces processor latency, and thus improves system efficiency.

[0045] The above-mentioned electronic device implements the method of structured data stratified sampling based on the secure data sandbox of the present application, which uses the secure data sandbox to connect to the data source of the data provider and determine the metadata information of the data source; determines the stratification field in response to the stratification request, determines the stratification and sampling strategy of each layer according to the stratification field in the secure data sandbox, and allows the sampling strategy of each layer to be modified separately; performs stratified sampling of the data source in the policy secure data sandbox according to the sampling of each layer, and then merges the sampled data of each layer; uses the merged sampled data to train the data analysis model in the secure data sandbox, and then uses the data analysis model to analyze the full amount of data of the data source. The technical solution improves the accuracy and efficiency of data analysis and reduces the data processing cost through stratified sampling; allows the sampling strategy of each layer to be modified separately, increases the flexibility and adaptability of the method, and ensures the security and privacy protection of the data during the processing process.

[0046] The storage medium provided in the present application stores a program product that can implement a method for stratified sampling of structured data based on a secure data sandbox.

[0047] The methods for structured data stratified sampling based on secure data sandbox include: Use the secure data sandbox to connect to the data provider's data source and determine the metadata information of the data source; Respond to stratification requests to determine stratification fields, determine stratification and sampling strategies for each stratum based on the stratification fields within the secure data sandbox, and allow the sampling strategies for each stratum to be modified individually; Perform stratified sampling of data sources within the secure data sandbox according to the sampling strategy for each layer, and then merge the sampled data for each layer; Use the merged sample data to train the data analysis model in the secure data sandbox, and then use the data analysis model to analyze the full data of the data source.

[0048] In some possible implementations, the method of structured data stratified sampling based on a secure data sandbox disclosed herein can be implemented in the form of a program product, which includes program code. When the program product is run on a terminal device, the program code is used to enable the terminal device to execute the steps described in the above "Exemplary Method" section of this specification according to various exemplary implementations of the present disclosure.

[0049] The storage medium of the present disclosure can adopt any combination of one or more readable media. The readable medium can be a readable signal medium or a readable storage medium. The readable storage medium can be, for example, but not limited to, a system, device or component of electricity, magnetism, light, electromagnetic, infrared, or semiconductor, or any combination thereof. More specific examples (non-exhaustive list) of readable storage media include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof.

[0050] The above description of the disclosed embodiments is intended to enable one skilled in the art to implement or use the present invention. Various modifications to these embodiments will be readily apparent to one skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention is not limited to the embodiments shown herein but is intended to conform to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A method for stratified sampling of structured data based on a secure data sandbox, characterized in that: The steps include: S1. Use the secure data sandbox to connect to the data provider's data source and determine the metadata information of the data source; S2. Respond to the stratification request to determine the stratification fields, determine the stratification and sampling strategy for each stratum based on the stratification fields within the secure data sandbox, and allow the sampling strategy for each stratum to be modified separately; S3. Perform stratified sampling of data sources within the secure data sandbox according to the sampling strategy for each layer, and then merge the sampled data for each layer; S4. Use the merged sample data to train the data analysis model in the secure data sandbox, and then use the data analysis model to analyze the full data of the data source.

2. The method for stratified sampling of structured data based on a secure data sandbox according to claim 1, characterized in that: The specific steps of step S1 are as follows: S11. Obtain data source information in the secure data sandbox and determine the data source type; If the data source is a database, proceed to step S12; If the data source is a file type, go to step S14; S12. Perform a connection test on a database-type data source in a secure data sandbox; If the test passes, proceed to step S15; If the test fails, proceed to step S13; S13. Modify the data source information and return to step S12; S14. Parsing the data source of the file type; S15. Obtain metadata information and save it to the database of the secure data sandbox; the metadata information includes fields.

3. The method for stratified sampling of structured data based on a secure data sandbox according to claim 2, characterized in that: The specific steps of step S14 are as follows: S141. The file name of the data source of the file type is used as the library name; S142. Determine the file type of the file data source; If it is an Excel file, the sheet name is used as the table name; If it is a CSV file, use the file name as the table name; S143. Use the column name of the file type data source as the field name; The metadata information in step S15 also includes the library, table, and number of data rows.

4. The method for stratified sampling of structured data based on a secure data sandbox according to claim 2, characterized in that: The specific steps of step S2 are as follows: S21. Respond to the hierarchical request and obtain the selected hierarchical field; S22. Query the data source in the secure data sandbox based on the selected layer field to determine the number of layers M and the amount of data C per layer i , and feedback to the data provider; S23. respond and receive data provider according to the data volume of each layer C i Set sampling parameters; If the sampling parameter is the number of sampling samples N, proceed to step S24; If the sampling parameter is the sampling ratio R, proceed to step S25; S24. Query the full amount of data S of the data source in the secure data sandbox, calculate the sampling ratio R using the number of sampled samples N / full amount of data S, and proceed to step S3; S25. Obtain the data volume C of each layer in the secure data sandbox i And the sampling ratio R calculates the number of sampling samples N in each layer i , for the number of sampling samples N i Mark the layers that do not meet the requirements; S26. Modify the sample sampling strategy for the marked stratum separately, and set a new sampling number or sampling ratio until the requirements are met.

5. The method for stratified sampling of structured data based on a secure data sandbox according to claim 4, characterized in that: The specific steps of step S26 are as follows: S261. Determine new sample sampling strategy for marker stratification; If we set a new sampling number N i , proceed to step S262; If a new sampling ratio R is set i , proceed to step S263; S262. Use the new sampling number N i And the data volume C of this layer i Calculate the sampling ratio R of this layer i =Number of samples in this layer N i / The amount of data in this layer C i , proceed to step S3; S263. Use the new sampling ratio R i And the data volume C of this layer i Calculate the number of samples N for this layer i ; S264. Determine the number of samples N for this layer i Whether there is demand; If yes, go to step S3; If not, mark the layer and return to step S261.

6. The method for stratified sampling of structured data based on a secure data sandbox according to claim 4, characterized in that: The specific steps of step S3 are as follows: S31. According to the amount of data per layer C i and the sampling ratio R of each layer i Calculate the final number of samples N for each layer i = Data volume per layer C i × sampling ratio of each layer R i ; S32. In the security data sandbox, the number of samples N of each layer is sampled. i Randomly sample each layer of the data source to obtain sampling data of each layer; S33. Merge the sampling data of each layer to obtain the sampling result.

7. The method for stratified sampling of structured data based on a secure data sandbox according to claim 6, characterized in that: The specific steps of step S4 are as follows: S41. Upload the sampling data in the sampling results to the debugging environment of the secure data sandbox; S42. The data user obtains sample data from the debugging environment of the secure data sandbox and uses the sample data to train the data analysis model until the data analysis model meets the requirements; S43. Publish the data analysis model to the operating environment of the secure data sandbox and use the data analysis model to analyze the full amount of data from the data source; S44. Respond to the data provider's request to view the full data analysis results, and grant the data user the right to use the full data analysis results.

8. A system for stratified sampling of structured data based on a secure data sandbox, characterized in that: include: The data source access module is used to connect to the data provider's data source using the secure data sandbox and determine the metadata information of the data source; A layer strategy setting module is used to determine the layer fields in response to the layer request, determine the layers and the sampling strategy for each layer based on the layer fields within the secure data sandbox, and allow the sampling strategy for each layer to be modified separately; The sample allocation and extraction module is used to perform stratified sampling of data sources within the secure data sandbox according to the sampling strategy of each layer, and then merge the sampled data of each layer; The data analysis and feedback module is used to use the merged sample data to train the data analysis model in the secure data sandbox, and then use the data analysis model to analyze the full data of the data source.

9. An electronic device, characterized in that: The method comprises a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the method implements the steps of the structured data stratified sampling method based on the secure data sandbox as described in any one of claims 1 to 7.

10. A storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method for structured data stratified sampling based on a secure data sandbox according to any one of claims 1 to 7 are implemented.

Images