Data quality assessment method based on data processing

Abstract

The present invention provides a data quality assessment method based on data processing, comprising: step 1, obtaining damage target-damage tool attribute data; step 2, preprocessing the damage target-damage tool attribute data based on an ETL tool to obtain structured damage target data and damage tool data, and loading them into a data warehouse; step 3, based on the damage target data and the damage tool data, using a logistic regression assessment model based on quantum particle swarm optimization to assess the damage probability as a first quality assessment result; step 4, comparing the first quality assessment result with an assessment system predetermined based on the damage target-damage tool attributes to obtain a second assessment result output, wherein the assessment system includes damage levels under different damage probabilities.

Description

Technical Field

[0001] The present invention relates to the technical field of data processing, and in particular to a data quality assessment method based on data processing. Background Art

[0002] With the continuous development of the information industry, how to process massive amounts of data and complete the evaluation of data quality has become a new challenge. The diversity of raw data makes it difficult to access subsequent application modules and data analysis in a unified format. In the process of data processing, in order to ensure data reliability, it is necessary to clean the data, detect and remove errors and inconsistent data parts from the data, so as to improve data availability, form structured data and synchronize it to the data warehouse, and then perform subsequent processing, analysis or application based on this data. Summary of the invention

[0003] The first aspect of the present invention provides a data quality assessment method based on data processing, which is applicable to data quality estimation of a damage target, comprising the following steps:

[0004] Step 1, obtaining the damage target-damage tool attribute data;

[0005] Step 2: Preprocess the damage target-damage tool attribute data based on the ETL tool to obtain structured damage target data and damage tool data, and load them into the data warehouse;

[0006] Step 3: Based on the damage target data and the damage tool data, a logistic regression evaluation model based on quantum particle swarm optimization is used to evaluate the damage probability as the first quality evaluation result;

[0007] Step 4: Compare the first quality assessment result with a predetermined assessment system based on the damage target-damage tool attributes to obtain a second assessment result output, wherein the assessment system includes damage levels under different damage probabilities.

[0008] A second aspect of the present invention provides a data quality assessment system based on data processing, comprising:

[0009] one or more processors;

[0010] A memory stores operable instructions, wherein when the instructions are executed by the one or more processors, the one or more processors are caused to perform operations, wherein the operations include the process of the aforementioned data quality assessment method.

[0011] A third aspect of the present invention provides a computer-readable medium storing software, wherein the software includes instructions that can be executed by one or more computers, and the instructions, through such execution, cause the one or more computers to perform operations, and the operations include the process of the aforementioned data quality assessment method.

[0012] According to the above technical scheme, the data quality assessment method based on data processing proposed in the present invention is based on the damage target-damage tool attribute, and adopts a logistic regression assessment model based on quantum particle swarm optimization to evaluate the damage probability, and compares it with the damage level of a predetermined evaluation system to determine the damage level output, thereby realizing the estimation of the damage probability and level of flying damage targets, and providing scientific data support for warhead development, actual strike damage and survival design of flying targets.

[0013] At the same time, the estimation model proposed in the present invention integrates the quantum particle swarm algorithm and the logistic regression algorithm. The combination of the quantum particle swarm algorithm helps the logistic regression algorithm to solve the problem of decreased accuracy due to excessive feature space, replaces the traditional gradient descent algorithm, helps improve the global data processing capability, reduces the problem of decreased model accuracy caused by sparse data and irregular distribution, and improves the sparse data effect of the logistic regression algorithm.

[0014] It should be understood that all combinations of the aforementioned concepts and the additional concepts described in more detail below can be considered as part of the inventive subject matter of the present disclosure as long as such concepts are not mutually inconsistent. In addition, all combinations of the claimed subject matter are considered as part of the inventive subject matter of the present disclosure.

[0015] The foregoing and other aspects, embodiments and features of the present invention can be more fully understood from the following description in conjunction with the accompanying drawings. Other additional aspects of the present invention, such as the features and / or beneficial effects of the exemplary embodiments, will be apparent from the following description or learned from the practice of the specific embodiments according to the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] The drawings are not intended to be drawn to scale. In the drawings, each identical or nearly identical component shown in various figures may be represented by the same reference numeral. For clarity, not every component is labeled in every figure. Embodiments of various aspects of the present invention will now be described by way of example and with reference to the accompanying drawings, in which:

[0017] Figure 1 It is a flowchart of a data quality assessment method based on data processing according to an exemplary embodiment of the present invention. DETAILED DESCRIPTION

[0018] In order to better understand the technical content of the present invention, specific embodiments are given and described as follows in conjunction with the accompanying drawings.

[0019] Various aspects of the present invention are described in this disclosure with reference to the accompanying drawings, in which many illustrative embodiments are shown. The embodiments of the present disclosure are not necessarily intended to include all aspects of the present invention. It should be understood that the various concepts and embodiments introduced above, as well as those described in more detail below, can be implemented in any of many ways, because the concepts and embodiments disclosed by the present invention are not limited to any implementation. In addition, some aspects disclosed by the present invention can be used alone or in any appropriate combination with other aspects disclosed by the present invention.

[0020] Combination Figure 1 The data quality assessment method based on data processing of the illustrated embodiment includes the following processes: Step 1, obtaining damage target-damage tool attribute data; Step 2, preprocessing the damage target-damage tool attribute data based on the ETL tool to obtain structured damage target data and damage tool data, and loading them into the data warehouse; Step 3, based on the damage target data and the damage tool data, using a logistic regression assessment model based on quantum particle swarm optimization to assess the damage probability as the first quality assessment result; Step 4, comparing the first quality assessment result with a predetermined assessment system based on the damage target-damage tool attributes to obtain a second assessment result output, wherein the assessment system includes damage levels under different damage probabilities.

[0021] Therefore, the damage probability of each structural component of the damage target is obtained through the logistic regression algorithm QPSO based on particle quantum swarm optimization, and the damage probability of the damage target is obtained through weighted calculation, which is then compared with the predetermined damage levels under different damage probabilities. According to the damage probability falling into the corresponding probability interval, the corresponding damage level output is obtained.

[0022] In an optional embodiment, the aforementioned evaluation system can be determined based on different damage targets. For a certain type of fighter, different damage levels are set according to the distribution range of its damage probability. The greater the damage probability, the higher the damage level.

[0023] In the embodiment of the present invention, the damage target is a flying damage target, such as various types of fighters, drones, jammers, tankers, transport aircraft, etc. The damage tool is a flying warhead, especially a guided missile, such as a surface-to-air missile, an air-to-air missile, etc.

[0024] In an embodiment of the present invention, the damage target-damage tool attribute data is preprocessed based on the ETL tool, including:

[0025] Binarize the type, structural components, volume, mass, and flight speed of flying damage targets;

[0026] The damage tool flight speed, fragment flight speed, fragment dispersion angle, and ignition / detonation damage mode data of the damage tool are binarized.

[0027] ETL tools can use existing data processing tools to process and transform element data.

[0028] For example, different types are classified and labeled according to predetermined standards and given specific type identifiers. Structural components refer to cabin sections, cabin codes, wings, engine bays, tail fins, cockpits, bomb bays, etc.

[0029] In an embodiment of the present invention, the logistic regression evaluation model based on quantum particle swarm optimization includes a logistic regression evaluation algorithm layer based on quantum particle swarm optimization and a fully connected layer, that is, an underlying model and an output module. The underlying model, that is, the logistic regression evaluation algorithm layer based on quantum particle swarm optimization, is used to evaluate and output the damage probability of each structural component of the damage target, and then perform weighted calculation in the fully connected layer to obtain the damage probability of the damage target.

[0030] In an embodiment of the present invention, weighted calculation is performed at the fully connected layer to obtain the probability of damaging the target, including:

[0031] The probability of damaging the target Fc is calculated as follows:

[0032] Fc=Fc1(C1)+Fc2(C2)+Fc3(C3)…+Fc n (C n )

[0033] Among them, Fc n represents the damage weighting coefficient of the nth structural component, C n represents the damage probability of the nth structural component; n = 1, 2, 3, ..., N, N represents the total number of structural components of the flight damage target, Fc1, Fc2, Fc3, ..., Fc n The values ​​of are all in [0,1], and the sum of all damage weighting coefficients is 1.

[0034] Among them, for the structural components of flight-type damage targets, the damage weighting coefficient of critical components is set greater than the damage weighting coefficient of non-critical components.

[0035] The aforementioned critical components can be pre-set according to different types of flying damage targets.

[0036] As an optional embodiment, the aforementioned quantum particle swarm optimization-based logistic regression evaluation algorithm layer uses a multinomial logistic regression algorithm (QPSO) improved based on quantum particle swarm optimization to estimate the damage probability of each structural component, specifically including the following process:

[0037] Step 3.1, set the particle swarm parameters: particle swarm size N, particle dimension D, expansion and contraction control coefficient α, maximum number of iterations max_iter, and numerical optimization Newton method iteration number t;

[0038] Step 3.2, population initialization: given a data set X = {x1, x2, ..., x n}, where n is the number of samples in the data set, and each sample is the attribute data of a flight damage target, with multiple variable attributes; for any sample sequence k, k = 1, 2, 3, .., n, then (x k1 ,x k2 ,…,x kd ), indicating that any sample data has d variable attributes; generating a particle population with a population size of N and a particle dimension of D;

[0039] Step 3.3: Logistic regression uses the prediction results of the linear regression model to approximate the logarithmic probability of the true mark, expressed as:

[0040]

[0041] Among them, ω is a d×1 vector group, b is a constant;

[0042] The objective function of logistic regression is set as a convex function of any order. In order to obtain the optimal values ​​of ω and b, equation (1) is transformed to obtain:

[0043]

[0044] Then we have:

[0045]

[0046] Let β = (ω; b), Substituting into (3) we get:

[0047]

[0048] based on but:

[0049]

[0050] Calculate the parameters ω and β according to the maximum likelihood estimation method;

[0051] The data set value range is {(x i,y i )},i∈{1,2,…,n},y i ∈{0,1}, the logistic regression model maximizes the log-likelihood expression:

[0052]

[0053] Substituting (5) into (6), we obtain:

[0054]

[0055] By finding the β value, that is, the ω and b value, the log-likelihood (7) is maximized;

[0056] At this time, LL(β) is a high-order continuous differentiable convex function about β. According to the classical numerical optimization iterative algorithm, the optimal solution of β can be obtained by Newton's method. * Maximize the LL(β) value in formula (7);

[0057] The optimal solution β is determined from this * It is expressed as:

[0058] β * =argmaxLL(β) (8)

[0059] When the t+1th iteration begins, β is expressed as follows:

[0060]

[0061] The first and second order partial derivative expressions of β are:

[0062]

[0063]

[0064] In particular, the optimal solution β * ;

[0065] Step 3.4, combined with the iterative optimization of particle swarm parameters, solve the optimal parameters of logistic regression, specifically including substituting the particle swarm position into the log-likelihood formula (7) and calculating the log-likelihood value and the optimal solution, repeating the particle update operation until the number of iterations reaches the maximum number of iterations max_iter and then stopping, and outputting the final maximum likelihood value and the corresponding optimal solution parameter results, thereby determining the logistic regression evaluation algorithm layer model based on quantum particle swarm optimization.

[0066] As an optional example, in step 3.4, the particle swarm position is substituted into the log-likelihood formula (7) and the log-likelihood value and the optimal solution are calculated. The particle update operation is repeated until the number of iterations reaches the maximum number of iterations max_iter and then stops. The final maximum likelihood value and the corresponding optimal solution parameter result are output, including the following process:

[0067] Particle position In a d-dimensional search space, the particle quantum swarm algorithm consists of N particles representing potential problem solutions. The position of the i-th particle at time t is:

[0068] A i (t) = {a i,1 (t),a i,2 (t),…,α i,j (t),…,a i,D (t)},i∈{1,2,…,N},j∈{1,2,…,D} (12)

[0069] The optimal position Pbest of individual particles is expressed as:

[0070] P i (t) = {p i,1 (t),p i,2 (t),…,p i,j (t),…,p i,D (t)} (13)

[0071] The global optimal position Gbest is expressed as:

[0072] G(t)={g1(t),g2(t),…,g j (t),…,g D (t)} (14)

[0073] The optimal average position loc(t) of all individual particles is defined as:

[0074]

[0075] according to

[0076] Confirm the individual optimal position Pbest of the particle;

[0077] Where LL() is the log-likelihood value of formula (7);

[0078] Then the global optimal position Gbest transformation is expressed as:

[0079] Gbest = argmaxLL(P i (t)),i∈{1,2,..,N} (17)

[0080] When the particle group is at the global optimal position Gbest, the global optimal position value is updated as:

[0081] G(t)=P Gbest (t),Gbest∈{1,2,…,N} (18)

[0082] Therefore, when updating particles, the individual optimal position Pbest and the global optimal position Gbest of the particles are obtained; then the maximum log-likelihood formula (7) is substituted to update the fitness value of each particle, and the individual position fitness of each particle is compared with the fitness of the global optimal position. If the current individual optimal position fitness is better than the global optimal position fitness, the particle position is updated. According to formula (18), the updating process of the particle swarm is as follows:

[0083] Individual particle positions are updated as follows:

[0084] P i,j (t) = r(t) × P i,j (t)+(1-r(t))×G(t) (19)

[0085] Among them, r(t) is a random number with a value range of [0,1];

[0086] The particle swarm position is updated as follows:

[0087]

[0088] Among them, r i,j (t) is a random number in the range [0,1], α is the contraction and expansion coefficient, and ± is randomly expressed as + or - with equal probability for calculation.

[0089] According to an embodiment disclosed in the present invention, a data quality assessment system based on data processing is also proposed, comprising:

[0090] one or more processors;

[0091] The memory stores operable instructions, and when the instructions are executed by one or more processors, the one or more processors execute operations, and the operations include the process of the data quality assessment method of the aforementioned embodiment.

[0092] According to the embodiments disclosed in the present invention, a computer-readable medium storing software is also proposed. The software includes instructions that can be executed by one or more computers. The instructions enable one or more computers to perform operations through such execution. The operations include the process of the data quality assessment method of the aforementioned embodiment.

[0093] Although the present invention has been disclosed as above with preferred embodiments, it is not intended to limit the present invention. A person with ordinary knowledge in the technical field to which the present invention belongs may make various changes and modifications without departing from the spirit and scope of the present invention. Therefore, the protection scope of the present invention shall be determined by the definition of the claims.

Claims

1. A data quality assessment method based on data processing, characterized in that: include: Step 1, obtaining the damage target-damage tool attribute data; Step 2: Preprocess the damage target-damage tool attribute data based on the ETL tool to obtain structured damage target data and damage tool data, and load them into the data warehouse; wherein the damage target is a flying damage target, and the damage tool is a flying warhead, and the damage target attribute data includes the type, volume, mass, and flight speed of the flying damage target; and the damage tool attribute data includes the damage tool flight speed, fragment flight speed, fragment scattering angle, and ignition / detonation damage mode data; Step 3: Based on the damage target data and the damage tool data, a logistic regression evaluation model based on quantum particle swarm optimization is used to evaluate the damage probability as the first quality evaluation result; Step 4: Compare the first quality assessment result with a predetermined assessment system based on the damage target-damage tool attribute to obtain a second assessment result output, wherein the assessment system includes damage levels under different damage probabilities; The logistic regression evaluation model based on quantum particle swarm optimization includes a logistic regression evaluation algorithm layer based on quantum particle swarm optimization and a fully connected layer. The logistic regression evaluation algorithm layer based on quantum particle swarm optimization evaluates and outputs the damage probability of each structural component of the damage target, and then performs weighted calculation in the fully connected layer to obtain the damage probability of the damage target. Among them, the logistic regression evaluation algorithm layer based on quantum particle swarm optimization uses a multi-logistic regression algorithm improved based on quantum particle swarm optimization to estimate the damage probability of each structural component, which specifically includes the following process: Step 3.1, set the particle swarm parameters: particle swarm size N, particle dimension D, expansion and contraction control coefficient α, maximum number of iterations max_iter, and numerical optimization Newton method iteration number t; Step 3.2, population initialization: given a data set X = {x1, x2, ..., x n }, where x is the number of samples in the data set, and each sample is the attribute data of a flight damage target, with multiple variable attributes; for any sample sequence k, k = 1, 2, 3, .., n, then (x k1 ,x k2 ,…,x kd ), indicating that any sample data has d variable attributes; generating a particle population with a population size of N and a particle dimension of D; Step 3.3: Logistic regression uses the prediction results of the linear regression model to approximate the logarithmic probability of the true mark, expressed as: Among them, ω is a d×1 vector group, b is a constant; The objective function of logistic regression is set as a convex function of any order. In order to obtain the optimal values ​​of ω and b, equation (1) is transformed to obtain: Then we have: Let β = (ω; b), Substituting into (3) we get: based on but: Calculate the parameters ω and β according to the maximum likelihood estimation method; The data set value range is {(x i ,y i )},i∈{1,2,…,n},y i ∈{0,1}, the logistic regression model maximizes the log-likelihood expression: Substituting (5) into (6), we obtain: By finding the β value, that is, the ω and b value, the log-likelihood (7) is maximized; At this time, LL(β) is a high-order continuous differentiable convex function about β. According to the classical numerical optimization iterative algorithm, the Newton method is used to find the optimal solution of β * Maximize the LL(β) value in formula (7); The optimal solution β is determined from this * It is expressed as: β * =arg maxLL(β) (8) When the t+1th iteration begins, β is expressed as follows: The first and second order partial derivative expressions of β are: Thus, the optimal solution β is obtained * ; Step 3.4, combined with the iterative optimization of particle swarm parameters, solve the optimal parameters of logistic regression, specifically including substituting the particle swarm position into the log-likelihood formula (7) and calculating the log-likelihood value and the optimal solution, repeating the particle update operation until the number of iterations reaches the maximum number of iterations max_iter and then stopping, and outputting the final maximum likelihood value and the corresponding optimal solution parameter results, thereby determining the logistic regression evaluation algorithm layer model based on quantum particle swarm optimization; Furthermore, in step 3.4, the particle swarm position is substituted into the log-likelihood formula (7) and the log-likelihood value and the optimal solution are calculated. The particle update operation is repeated until the number of iterations reaches the maximum number of iterations max_iter and then stops. The final maximum likelihood value and the corresponding optimal solution parameter result are output, including the following process: Particle position In a d-dimensional search space, the particle quantum swarm algorithm consists of N particles representing potential problem solutions. The position of the i-th particle at time t is: A i (t)={a i,1 (t),a i,2 (t),…,a i,j (t),…,a i,D (t)},i∈{1,2,…,N},j∈{1,2,…,D} (12) The optimal position Pbest of individual particles is expressed as: P i (t)={p i,1 (t),p i,2 (t),…,p i,j (t),…,p i,D (t)} (13) The global optimal position Gbest is expressed as: G(t)={g1(t),g2(t),…,g j (t),…,g D (t)} (14) The optimal average position loc(t) of all individual particles is defined as: according to Confirm the individual optimal position Pbest of the particle; Where LL() is the log-likelihood value of formula (7); Then the global optimal position Gbest transformation is expressed as: Gbest=arg maxLL(P i (t)),i∈{1,2,..,N} (17) When the particle group is at the global optimal position Gbest, the global optimal position value is updated as: G(t)=P Gbest (t),Gbest∈{1,2,…,N} (18) Therefore, when updating particles, the individual optimal position Pbest and the global optimal position Gbest of the particles are obtained; then the maximum log-likelihood formula (7) is substituted to update the fitness value of each particle, and the individual position fitness of each particle is compared with the fitness of the global optimal position. If the current individual optimal position fitness is better than the global optimal position fitness, the particle position is updated. According to formula (18), the updating process of the particle swarm is as follows: Individual particle positions are updated as follows: P i,j (t)=r(t)×P i,j (t)+(1-r(t))×G(t) (19) Among them, r(t) is a random number with a value range of [0,1]; The particle swarm position is updated as follows: Among them, r i,j (t) is a random number in the range [0,1], α is the contraction and expansion coefficient, and ± is randomly expressed as + or - with equal probability for calculation.

2. The data quality assessment method based on data processing according to claim 1, characterized in that: The weighted calculation at the fully connected layer to obtain the probability of damaging the target includes: The probability of damaging the target Fc is calculated as follows: Fc=Fc1(C1)+Fc2(C2)+Fc3(C3)…+Fc n (C n ) Among them, Fc n represents the damage weighting coefficient of the nth structural component, C n represents the damage probability of the nth structural component; n = 1, 2, 3, ..., N, N represents the total number of structural components of the flight damage target, Fc1, Fc2, Fc3, ..., Fc n The values ​​of are all in [0,1], and the sum of all damage weighting coefficients is 1.

3. The data quality assessment method based on data processing according to claim 2 is characterized in that: For the structural components of flight damage targets, the damage weighting coefficient of critical components is set greater than the damage weighting coefficient of non-critical components.

4. A data quality assessment system based on data processing, characterized in that: include: one or more processors; A memory storing operable instructions, wherein when the instructions are executed by the one or more processors, the one or more processors are caused to perform operations, wherein the operations include the process of the data quality assessment method as described in any one of claims 1 to 3.

5. A computer-readable medium storing software, characterized in that: The software includes instructions that can be executed by one or more computers, and the instructions, through such execution, enable the one or more computers to perform operations, and the operations include the process of the data quality assessment method described in any one of claims 1-3.

Images