Sample expansion method and device
By correcting the simulation data based on the positive distribution diffusion formula, a simulation data set with higher richness is generated, which solves the problem of insufficient typhoon disaster data, improves the comprehensiveness and accuracy of the data set, and supports more effective loss prediction.
Patent Information
- Application Number
- CN202411130716.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-16
- Publication Date
- 2024-11-26
AI Technical Summary
How to effectively increase the richness and diversity of data sets to support loss prediction when facing limited typhoon disaster data.
By obtaining the sample data set, the step size is determined based on the extreme value of each dimension and the total sample size, the first simulated data set is generated, and based on the difference between the sample data and the first simulated data, the positive distribution diffusion formula is used for correction. Generate a second simulated dataset, thereby improving the richness and comprehensiveness of the sample dataset.
By reasonably inferring and generating additional simulation data based on existing samples, the second simulation data set retains the statistical characteristics of the original data, improves the richness and diversity of the sample data set, and supports more accurate typhoon disaster losses predict.
Smart Images

Figure CN119025893A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of data processing technology, and in particular to a sample expansion method and device. Background Art
[0002] Typhoon disasters are one of the most serious marine disasters in my country, and their loss prediction is crucial for disaster prevention and mitigation. Effective loss prediction needs to be based on historical typhoon disaster data, but such data are often limited, which brings challenges to the prediction work.
[0003] Therefore, when faced with limited typhoon disaster data, in order to solve the problem of insufficient data volume, a sample expansion method is urgently needed to increase the richness and diversity of the data set. Summary of the invention
[0004] This application proposes a sample expansion method and device. The specific scheme is as follows:
[0005] In one aspect, an embodiment of the present application provides a sample expansion method, including:
[0006] Acquire a sample data set, wherein the sample data set includes a plurality of sample data, and the sample data includes feature values of a plurality of dimensions;
[0007] Determine a step size based on an extreme value of the eigenvalue in each dimension and a total sample size of the sample data set, so as to generate a first simulated data set based on each sample data and the step size, wherein the first simulated data set includes a plurality of first simulated data, and the first simulated data includes eigenvalues of a plurality of dimensions;
[0008] Based on the difference between the sample data and the first simulation data, the first simulation data is corrected to obtain a second simulation data set.
[0009] In a possible implementation of an embodiment of an aspect of the present application, based on the difference between the sample data and the first simulation data, the first simulation data is corrected to obtain a second simulation data set, including:
[0010] Based on the variance corresponding to each dimension, determine the normal distribution diffusion formula corresponding to each dimension;
[0011] Determine the correction value corresponding to each dimension in the first simulation data based on the normal distribution diffusion formula corresponding to each dimension and the difference between the sample data and the characteristic value in the first simulation data under each dimension;
[0012] The first simulation data is corrected based on the correction value to obtain a second simulation data set.
[0013] In a possible implementation of an embodiment of one aspect of the present application, it also includes:
[0014] Based on the extreme value of the eigenvalue and the number of samples under each dimension, the variance corresponding to each dimension is determined.
[0015] In a possible implementation of an embodiment of an aspect of the present application, based on the difference between the sample data and the first simulation data, the first simulation data is corrected to obtain a second simulation data set, including:
[0016] Based on the preset window length and sliding step size, sub-sample data in each dimension are sequentially extracted from the sample data set, and correspondingly, simulated data to be corrected in each dimension are sequentially extracted from the first simulated data set;
[0017] Based on the difference between each simulated data to be corrected and the corresponding sub-sample data, each simulated data to be corrected is corrected to obtain a second simulated data set.
[0018] Another aspect of the present application provides a sample expansion device, comprising:
[0019] An acquisition module is used to acquire a sample data set, wherein the sample data set includes a plurality of sample data, and the sample data includes feature values of a plurality of dimensions;
[0020] A generating module, configured to determine a step length based on an extreme value of a characteristic value in each dimension and a total sample size of the sample data set, so as to generate a first simulated data set based on each sample data and the step length, wherein the first simulated data set includes a plurality of first simulated data, and the first simulated data includes characteristic values of a plurality of dimensions;
[0021] The correction module is used to correct the first simulation data based on the difference between the sample data and the first simulation data to obtain a second simulation data set.
[0022] In a possible implementation of an embodiment of one aspect of the present application, the correction module is used to:
[0023] Based on the variance corresponding to each dimension, determine the normal distribution diffusion formula corresponding to each dimension;
[0024] Determine the correction value corresponding to each dimension in the first simulation data based on the normal distribution diffusion formula corresponding to each dimension and the difference between the sample data and the characteristic value in the first simulation data under each dimension;
[0025] The first simulation data is corrected based on the correction value to obtain a second simulation data set.
[0026] In a possible implementation manner of an embodiment of one aspect of the present application, a determination module is further included, which is used to:
[0027] Based on the extreme value of the eigenvalue and the number of samples under each dimension, the variance corresponding to each dimension is determined.
[0028] In a possible implementation of an embodiment of one aspect of the present application, the correction module is used to:
[0029] Based on the preset window length and sliding step size, sub-sample data in each dimension are sequentially extracted from the sample data set, and correspondingly, simulated data to be corrected in each dimension are sequentially extracted from the first simulated data set;
[0030] Based on the difference between each simulated data to be corrected and the corresponding sub-sample data, each simulated data to be corrected is corrected to obtain a second simulated data set.
[0031] Another aspect of the present application provides a computer device, including a processor and a memory;
[0032] The processor runs a program corresponding to the executable program code by reading the executable program code stored in the memory, so as to implement the method as described in the above embodiment.
[0033] Another aspect of the present application provides a computer-readable storage medium having a computer program stored thereon, wherein the program implements the method of the above embodiment when executed by a processor.
[0034] Additional aspects and advantages of the present application will be given in part in the description below, and in part will become apparent from the description below, or will be learned through the practice of the present application. BRIEF DESCRIPTION OF THE DRAWINGS
[0035] The above and / or additional aspects and advantages of the present application will become apparent and easily understood from the following description of the embodiments in conjunction with the accompanying drawings, in which:
[0036] Figure 1 A flow chart of a sample expansion method provided in an embodiment of the present application;
[0037] Figure 2 A schematic diagram of the structure of a sample expansion device provided in an embodiment of the present application. DETAILED DESCRIPTION
[0038] The embodiments of the present application are described in detail below, and examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are intended to be used to explain the present application, and should not be construed as limiting the present application.
[0039] The sample expansion method of an embodiment of the present application is described below with reference to the accompanying drawings.
[0040] Figure 1 A flow chart of a sample expansion method provided in an embodiment of the present application.
[0041] The sample expansion method of the embodiment of the present application is executed by the sample expansion device (hereinafter referred to as the expansion device) provided in the embodiment of the present application. The device can be configured in a computer device to improve the richness and diversity of samples.
[0042] like Figure 1 As shown, the sample expansion method includes:
[0043] Step 101: Obtain a sample data set, wherein the sample data set includes a plurality of sample data, and the sample data includes feature values of a plurality of dimensions.
[0044] For example, in the scenario of predicting the economic losses caused by typhoon disasters, multiple dimensions may include wind speed level, central air pressure, local fiscal revenue, etc., and this application does not impose any restrictions on this. The sample data set can be obtained from information published by relevant agencies such as meteorological statistics.
[0045] For example, some sample data in the sample data set are shown in Table 1:
[0046] Table 1
[0047]
[0048] Step 102, determine the step size based on the extreme value of the eigenvalue under each dimension and the total sample size of the sample data set, so as to generate a first simulated data set based on each sample data and the step size, wherein the first simulated data set includes multiple first simulated data, and the first simulated data includes eigenvalues of multiple dimensions.
[0049] The generation process of the eigenvalues in each dimension in the first simulated data set can be seen in the following formula:
[0050]
[0051] u i =X min +i×t
[0052] Among them, t is the step size, X max is the maximum value, X min is the minimum value, n is the total number of samples in the sample data set, i≤n. For example, the maximum value of central air pressure is a, the minimum value is b, and the total number of samples is n. Then the characteristic value of the central air pressure dimension in the first simulation data set is
[0053] Step 103: based on the difference between the sample data and the first simulation data, correct the first simulation data to obtain a second simulation data set.
[0054] In the present application, the variance corresponding to each dimension can be set in the system in advance, so as to determine the normal distribution diffusion formula corresponding to each dimension based on the variance corresponding to each dimension. Afterwards, based on the normal distribution diffusion formula corresponding to each dimension and the difference between the sample data and the characteristic value in the first simulation data under each dimension, the correction value corresponding to each dimension in the first simulation data is determined, and then, the first simulation data is corrected based on the correction value to obtain the second simulation data set. Thus, through the properties of the normal distribution, an additional second simulation data set is reasonably inferred and generated based on the existing samples, and the second simulation data set retains the statistical characteristics of the original data. Thereby improving the richness and comprehensiveness of the sample data set.
[0055] For example, the process of determining the correction value corresponding to each dimension in the first simulation data can be referred to the following formula:
[0056]
[0057] Where h is the variance.
[0058] Correction value q ij Perform normalization to obtain the normalized correction value:
[0059]
[0060]
[0061] Afterwards, the first simulation data is corrected based on the correction value to obtain the second simulation data set. The process can be seen in the following formula:
[0062]
[0063] Optionally, based on the extreme value of the eigenvalue in each dimension and the number of samples, the variance corresponding to each dimension is determined.
[0064] For example, the process of determining the variance corresponding to each dimension can be seen in the following formula:
[0065]
[0066] Wherein, L=abs(ab), a and b are the maximum and minimum values of the corresponding dimension, respectively, and N is the number of samples.
[0067] Optionally, based on a preset window length and sliding step size, subsample data in each dimension are sequentially extracted from the sample data set, and correspondingly, simulated data to be corrected in each dimension are sequentially extracted from the first simulated data set, and then, based on the difference between each simulated data to be corrected and the corresponding subsample data, each simulated data to be corrected is corrected to obtain a second simulated data set. The influence of redundant data is avoided, thereby improving the accuracy of the second simulated data set.
[0068] For example, assuming that each sample data set is (x1, x2, x3, x4, x5, x6, x7) in the central air pressure dimension, the window length is 3, and the sliding step is 2, then the sub-sample data extracted from the sample data set in the central air pressure dimension are (x1, x2, x3), (x3, x4, x5), (x5, x6, x7). Assuming that the first simulation data set is (y1, y2, y3, y4, y5, y6, y7) in the central air pressure dimension, then the simulation data to be corrected in the central air pressure dimension are (y1, y2, y3), (y3, y4, y5), (y5, y6, y7) extracted from the first simulation data set in the central air pressure dimension. According to the acquisition order of the simulation data to be corrected and the sub-sample data, determine the sub-sample data corresponding to the simulation data to be corrected. Thus, (y1, y2, y3) corresponds to (x1, x2, x3), (y3, y4, y5) corresponds to (x3, x4, x5), and (y5, y6, y7) corresponds to (x5, x6, x7). Afterwards, (y1, y2, y3) is corrected based on the difference between (y1, y2, y3) and (x1, x2, x3); (y3, y4, y5) is corrected based on the difference between (y3, y4, y5) and (x3, x4, x5); (y5, y6, y7) is corrected based on the difference between (y5, y6, y7) and (x5, x6, x7), thereby obtaining a second simulated data set.
[0069] In the present application, a sample data set is obtained, wherein the sample data set includes multiple sample data, and the sample data includes characteristic values of multiple dimensions. Then, the step size is determined based on the extreme value of the characteristic value in each dimension and the total sample size of the sample data set, so as to generate a first simulated data set based on each sample data and the step size, wherein the first simulated data set includes multiple first simulated data, and the first simulated data includes characteristic values of multiple dimensions. Then, based on the difference between the sample data and the first simulated data, the first simulated data is corrected to obtain a second simulated data set. Thus, by reasonably inferring and generating an additional second simulated data set based on the existing samples, the second simulated data set retains the statistical characteristics of the original data. Thereby, the richness and comprehensiveness of the sample data set are improved.
[0070] In order to implement the above embodiment, the embodiment of the present application further proposes a sample expansion device. Figure 2 A schematic diagram of the structure of a sample expansion device provided in an embodiment of the present application.
[0071] like Figure 2 As shown, the sample expansion device 200 includes:
[0072] An acquisition module 210 is used to acquire a sample data set, wherein the sample data set includes a plurality of sample data, and the sample data includes feature values of a plurality of dimensions;
[0073] A generating module 220 is used to determine a step size based on an extreme value of the characteristic value in each dimension and a total sample size of the sample data set, so as to generate a first simulated data set based on each sample data and the step size, wherein the first simulated data set includes a plurality of first simulated data, and the first simulated data includes characteristic values of a plurality of dimensions;
[0074] The correction module 230 is used to correct the first simulation data based on the difference between the sample data and the first simulation data to obtain a second simulation data set.
[0075] In a possible implementation of another aspect of the present application, the correction module 230 is used to:
[0076] Based on the variance corresponding to each dimension, determine the normal distribution diffusion formula corresponding to each dimension;
[0077] Determine the correction value corresponding to each dimension in the first simulation data based on the normal distribution diffusion formula corresponding to each dimension and the difference between the sample data and the characteristic value in the first simulation data under each dimension;
[0078] The first simulation data is corrected based on the correction value to obtain a second simulation data set.
[0079] In a possible implementation manner of another aspect of the present application, a determination module is further included, which is used to:
[0080] Based on the extreme value of the eigenvalue and the number of samples under each dimension, the variance corresponding to each dimension is determined.
[0081] In a possible implementation of another aspect of the present application, the correction module 230 is used to:
[0082] Based on the preset window length and sliding step size, sub-sample data in each dimension are sequentially extracted from the sample data set, and correspondingly, simulated data to be corrected in each dimension are sequentially extracted from the first simulated data set;
[0083] Based on the difference between each simulated data to be corrected and the corresponding sub-sample data, each simulated data to be corrected is corrected to obtain a second simulated data set.
[0084] It should be noted that the above explanation of the sample expansion method embodiment is also applicable to the sample expansion device of this embodiment, so it will not be repeated here.
[0085] In the present application, a sample data set is obtained, wherein the sample data set includes multiple sample data, and the sample data includes characteristic values of multiple dimensions. Then, the step size is determined based on the extreme value of the characteristic value in each dimension and the total sample size of the sample data set, so as to generate a first simulated data set based on each sample data and the step size, wherein the first simulated data set includes multiple first simulated data, and the first simulated data includes characteristic values of multiple dimensions. Then, based on the difference between the sample data and the first simulated data, the first simulated data is corrected to obtain a second simulated data set. Thus, by reasonably inferring and generating an additional second simulated data set based on the existing samples, the second simulated data set retains the statistical characteristics of the original data. Thereby, the richness and comprehensiveness of the sample data set are improved.
[0086] In order to implement the above embodiment, the embodiment of the present application also proposes a computer device, including a processor and a memory;
[0087] The processor runs a program corresponding to the executable program code by reading the executable program code stored in the memory, so as to implement the sample expansion method as described in the above embodiment.
[0088] In order to implement the above embodiment, the embodiment of the present application further proposes a computer-readable storage medium, on which a computer program is stored. When the program is executed by a processor, the sample expansion method as in the above embodiment is implemented.
[0089] Although the embodiments of the present application have been shown and described above, it can be understood that the above embodiments are exemplary and cannot be understood as limitations on the present application. Ordinary technicians in the field can change, modify, replace and modify the above embodiments within the scope of the present application.
Claims
1. A sample expansion method, characterized in that: include: Acquire a sample data set, wherein the sample data set includes a plurality of sample data, and the sample data includes feature values of a plurality of dimensions; Determine a step size based on the extreme value of the characteristic value under each dimension and the total sample size of the sample data set, so as to generate a first simulated data set based on each sample data and the step size, wherein the first simulated data set includes a plurality of first simulated data, and the first simulated data includes characteristic values of a plurality of dimensions; Based on the difference between the sample data and the first simulation data, the first simulation data is corrected to obtain a second simulation data set.
2. The method according to claim 1, characterized in that The step of correcting the first simulation data based on the difference between the sample data and the first simulation data to obtain a second simulation data set includes: Based on the variance corresponding to each dimension, determine the normal distribution diffusion formula corresponding to each dimension; Determine a correction value corresponding to each dimension in the first simulation data based on a normal distribution diffusion formula corresponding to each dimension and a difference between the sample data and the characteristic value in the first simulation data under each dimension; The first simulation data is corrected based on the correction value to obtain a second simulation data set.
3. The method according to claim 1, characterized in that Also includes: Based on the extreme value of the eigenvalue under each dimension and the number of samples, the variance corresponding to each dimension is determined.
4. The method according to claim 1, characterized in that The step of correcting the first simulation data based on the difference between the sample data and the first simulation data to obtain a second simulation data set includes: Based on a preset window length and a sliding step size, subsample data under each dimension is sequentially extracted from the sample data set, and correspondingly, simulated data to be corrected under each dimension is sequentially extracted from the first simulated data set; Based on the difference between each of the simulated data to be corrected and the corresponding sub-sample data, each of the simulated data to be corrected is corrected to obtain a second simulated data set.
5. A sample expansion device, characterized in that: include: An acquisition module, used to acquire a sample data set, wherein the sample data set includes a plurality of sample data, and the sample data includes feature values of a plurality of dimensions; A generating module, configured to determine a step length based on an extreme value of the characteristic value under each dimension and a total sample size of the sample data set, so as to generate a first simulated data set based on each sample data and the step length, wherein the first simulated data set includes a plurality of first simulated data, and the first simulated data includes characteristic values of a plurality of dimensions; The correction module is used to correct the first simulation data based on the difference between the sample data and the first simulation data to obtain a second simulation data set.
6. The device according to claim 5, characterized in that The correction module is used to: Based on the variance corresponding to each dimension, determine the normal distribution diffusion formula corresponding to each dimension; Determine a correction value corresponding to each dimension in the first simulation data based on a normal distribution diffusion formula corresponding to each dimension and a difference between the sample data and the characteristic value in the first simulation data under each dimension; The first simulation data is corrected based on the correction value to obtain a second simulation data set.
7. The device according to claim 5, characterized in that Also included is a determination module for: Based on the extreme value of the eigenvalue under each dimension and the number of samples, the variance corresponding to each dimension is determined.
8. The device according to claim 5, characterized in that The correction module is used to: Based on a preset window length and a sliding step size, subsample data under each dimension is sequentially extracted from the sample data set, and correspondingly, simulated data to be corrected under each dimension is sequentially extracted from the first simulated data set; Based on the difference between each of the simulated data to be corrected and the corresponding sub-sample data, each of the simulated data to be corrected is corrected to obtain a second simulated data set.
9. An electronic device, characterized in that: include: A processor, and a memory communicatively connected to the processor; The memory stores computer-executable instructions; The processor executes the computer-executable instructions stored in the memory to implement the method according to any one of claims 1 to 4.
10. A computer program product, characterized in that The invention comprises a computer program, which implements the method according to any one of claims 1 to 4 when being executed by a processor.
