Data acquisition method, device, electronic device and computer readable medium

Abstract

The embodiments of the present disclosure disclose a data acquisition method, device, electronic device, and computer-readable medium. A specific implementation of the method includes: acquiring data acquisition information; acquiring a target data set from a target storage database; performing at least one data interception operation on the target data set to obtain at least one data subset; for each data subset, performing a first determination step: determining data importance information and data impact information; generating a data quality score; in response to determining that the data quality score is greater than or equal to a first value, determining the data subset as a candidate data subset; screening out candidate data subsets whose corresponding data quality scores are in the front target number to obtain a candidate data subset group; generating an actual data set to be acquired; and sending the actual data set to be acquired to a data storage device. This implementation can accurately acquire the required high-quality data set on the premise of determining that the data will not be leaked.

Description

Technical Field

[0001] Embodiments of the present disclosure relate to the field of computer technology, and in particular to a data acquisition method, device, electronic device, and computer-readable medium. Background Art

[0002] At present, in the era of big data, data sets have become valuable assets for various enterprises. Acquiring effective data sets has become the key development direction of current enterprises. The usual way to obtain data sets is to directly collect or purchase relevant data sets.

[0003] However, the inventors have found that when the above method is adopted, the following technical problem often occurs:

[0004] The volume of the required data set is generally large. The direct collection method takes a long time to collect, and the collected data is often incomplete, resulting in poor quality of the collected data set. In addition, through the purchase method, the purchased data set may have a large deviation from the required data set, resulting in poor quality of the obtained data set, which greatly affects the accuracy of subsequent business.

[0005] In the process of adopting technical solutions to solve the above technical problem 1, the following technical problem 2 is often accompanied: how to accurately generate a data quality score for the above data subset. For the above technical problem 2, the conventional solution is generally: directly input the data subset and other related information directly into the data quality score generation model to generate the data quality score. However, the above conventional solution still has the following problem: it is impossible to effectively evaluate the output results of the data quality score generation model, resulting in the output data quality score being not accurate enough.

[0006] The above information disclosed in this Background section is only for enhancement of understanding of the background of the inventive concept and therefore it may contain information that does not form the prior art that is already known in this country to a person of ordinary skill in the art. Summary of the invention

[0007] The content of this disclosure is used to introduce concepts in a brief form, which will be described in detail in the detailed implementation section below. The content of this disclosure is not intended to identify the key features or essential features of the technical solution claimed for protection, nor is it intended to limit the scope of the technical solution claimed for protection.

[0008] Some embodiments of the present disclosure propose data acquisition methods, devices, electronic devices, and computer-readable media to solve one or more of the technical problems mentioned in the above background technology section.

[0009] In a first aspect, some embodiments of the present disclosure provide a data acquisition method, comprising: acquiring data acquisition information input into a target data acquisition platform, wherein the data acquisition information comprises: data volume information, data range information and data domain information; acquiring a target data set from a target storage database according to the data domain information; performing at least one data interception operation on the target data set according to the data volume information and the data range information to obtain at least one data subset; for each data subset in the at least one data subset, performing the following first determination step: determining data importance information and data influence degree information corresponding to the data subset; determining the data importance information and data influence degree information corresponding to the data subset according to the above data domain information; performing the following first determination step on the target data set according to the above data volume information and the data range information; performing the following first determination step on the target data set ... The data importance information and the data influence information are used to generate a data quality score for the data subset using a data quality score generation model; in response to determining that the data quality score is greater than or equal to a first value, the data subset is determined as a candidate data subset; from the obtained at least one candidate data subset, a number of candidate data subsets whose corresponding data quality scores are in the front target number are screened to obtain a candidate data subset group; based on the candidate data subset group, an actual data set to be obtained for the data acquisition information is generated; and the actual data set to be obtained is sent to the data storage device connected to the target data acquisition platform and inserted by the user through asynchronous transmission.

[0010] In a second aspect, some embodiments of the present disclosure provide a data acquisition device, comprising: a first acquisition unit, configured to acquire data acquisition information input into a target data acquisition platform, wherein the data acquisition information includes: data volume information, data range information and data domain information; a second acquisition unit, configured to acquire a target data set from a target storage database according to the data domain information; a capture unit, configured to perform at least one data capture operation on the target data set according to the data volume information and the data range information to obtain at least one data subset; an execution unit, configured to perform the following first determination step for each data subset in the at least one data subset: determine the data importance information and data influence degree corresponding to the data subset information; generating a data quality score for the data subset using a data quality score generation model according to the data importance information and the data impact information; determining the data subset as a candidate data subset in response to determining that the data quality score is greater than or equal to a first value; a screening unit configured to screen out candidate data subsets whose corresponding data quality scores are in the front target number from at least one obtained candidate data subset to obtain a candidate data subset group; a generating unit configured to generate an actual data set to be obtained for the data acquisition information according to the candidate data subset group; a sending unit configured to send the actual data set to be obtained to the data storage device connected to the target data acquisition platform and inserted by the user through asynchronous transmission.

[0011] In a third aspect, some embodiments of the present disclosure provide an electronic device comprising: one or more processors; a storage device on which one or more programs are stored, and when the one or more programs are executed by the one or more processors, the one or more processors implement the method described in any implementation manner in the first aspect.

[0012] In a fourth aspect, some embodiments of the present disclosure provide a computer-readable medium having a computer program stored thereon, wherein when the program is executed by a processor, the method described in any implementation manner in the first aspect is implemented.

[0013] The above-mentioned embodiments of the present disclosure have the following beneficial effects: through the data acquisition method of some embodiments of the present disclosure, the required high-quality data set can be accurately obtained under the premise of determining that the data will not be leaked. Specifically, the reason why the relevant data set is not of high quality is that the volume of the required data set is generally large, the direct collection method has a long collection time, and the collected data is often incomplete, resulting in poor quality of the collected data set. In addition, through the purchase method, the purchased data set may have a large deviation from the required data set, resulting in the quality of the obtained data set is also poor, which greatly affects the accuracy of subsequent business. Based on this, the data acquisition method of some embodiments of the present disclosure first obtains the data acquisition information input in the target data acquisition platform. Among them, the above-mentioned data acquisition information includes: data volume information, data range information and data field information. Here, the data volume information, data range information and data field information in the acquired data acquisition information are used for subsequent high-quality data set acquisition. Then, according to the above-mentioned data field information, the target data set is obtained from the target storage database as a basic data set to obtain subsequent high-quality data sets. Then, according to the above data volume information and the above data range information, at least one data interception operation is performed on the above target data set to obtain at least one data subset, so as to accurately obtain a high-quality data set by multiple data interceptions. Then, for each data subset in the above at least one data subset, the following first determination step is performed: the first step is to determine the data importance information and data impact information corresponding to the above data subset. Here, by determining the data importance information and data impact information as relevant indicator information for determining data quality, the numerical quality corresponding to each data subset is determined, so as to facilitate the subsequent determination of an accurate high-quality data set. The second step is to accurately generate a data quality score for the above data subset based on the above data importance information and the above data impact information, using a data quality score generation model, so as to characterize the data quality of the data subset. The third step is to determine the above data subset as a candidate data subset in response to determining that the data quality score is greater than or equal to the first value. Furthermore, a target number of candidate data subsets with corresponding data quality scores are selected from the obtained at least one candidate data subset to obtain a candidate data subset group, so as to preliminarily select a target number of candidate data subsets with relatively high quality, so as to make a more accurate determination of the suitability of the subsequent data set. Furthermore, based on the above candidate data subset group, an actual data set to be acquired for the above data acquisition information can be accurately generated as an accurate and high-quality data set. Finally, the above actual data set to be acquired is sent to the data storage device connected to the above target data acquisition platform and inserted by the user through asynchronous transmission.In summary, by acquiring data information including data volume information, data range information and data domain information, as well as determining the data importance information, data impact information and subsequent data quality score corresponding to the data subset, the required high-quality data set can be accurately obtained under the premise of ensuring that the data will not be leaked. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] The above and other features, advantages and aspects of the embodiments of the present disclosure will become more apparent with reference to the following detailed description in conjunction with the accompanying drawings. Throughout the accompanying drawings, the same or similar reference numerals represent the same or similar elements. It should be understood that the drawings are schematic and that components and elements are not necessarily drawn to scale.

[0015] Figure 1 is a flow chart of some embodiments of the data acquisition method according to the present disclosure;

[0016] Figure 2 is a schematic diagram of the structure of some embodiments of the data acquisition device according to the present disclosure;

[0017] Figure 3 It is a schematic diagram of the structure of an electronic device suitable for implementing some embodiments of the present disclosure. DETAILED DESCRIPTION

[0018] Embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although certain embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be construed as being limited to the embodiments set forth herein. On the contrary, these embodiments are provided to provide a more thorough and complete understanding of the present disclosure. It should be understood that the drawings and embodiments of the present disclosure are only for exemplary purposes and are not intended to limit the scope of protection of the present disclosure.

[0019] It should also be noted that, for ease of description, only the parts related to the invention are shown in the drawings. In the absence of conflict, the embodiments and features in the embodiments of the present disclosure can be combined with each other.

[0020] It should be noted that the concepts such as "first" and "second" mentioned in the present disclosure are only used to distinguish different devices, modules or units, and are not used to limit the order or interdependence of the functions performed by these devices, modules or units.

[0021] It should be noted that the modifications of "one" and "plurality" mentioned in the present disclosure are illustrative rather than restrictive, and those skilled in the art should understand that unless otherwise clearly indicated in the context, it should be understood as "one or more".

[0022] The names of the messages or information exchanged between multiple devices in the embodiments of the present disclosure are only used for illustrative purposes and are not used to limit the scope of these messages or information.

[0023] The present disclosure will be described in detail below with reference to the accompanying drawings and in conjunction with embodiments.

[0024] refer to Figure 1 , shows a process 100 of some embodiments of the data acquisition method according to the present disclosure. The data acquisition method comprises the following steps:

[0025] Step 101, obtaining data acquisition information input in a target data acquisition platform.

[0026] In some embodiments, the execution subject of the above-mentioned data acquisition method can obtain the data acquisition information input in the target data acquisition platform through a wired connection method or a wireless connection method. Among them, the above-mentioned data acquisition information includes: data volume information, data range information and data field information. The target data acquisition platform can be a platform for acquiring the required data set. In practice, the target data acquisition platform can be a platform deployed on the target terminal. The data acquisition information can be relevant information for acquiring the data set. The data volume information can be the data volume size of the desired data set. The data range information can be the data content range involved in the data set. For example, for the data in the required data set is a numerical type, the data range can be the numerical size range of the data. The data field information can be the data field corresponding to the required data set. For example, the data field information can be e-commerce field information.

[0027] Step 102: Acquire the target data set from the target storage database according to the above data domain information.

[0028] In some embodiments, the execution subject may obtain the target data set from the target storage database according to the data domain information, wherein the target storage database may be a database storing data sets of various data domains and various data range information.

[0029] In some optional implementations of some embodiments, the step of acquiring the target data set from the target storage database according to the data domain information may include the following steps:

[0030] The first step is to generate first coded information for the data acquisition information using a first coding model according to the data domain information. The first coding model may be a neural network model for encoding information. The first coded information may be information in vector form. The first coding model may be a convolutional neural network model with 5 layers connected in series.

[0031] As an example, first, the execution subject may perform information word embedding processing on the data domain information to obtain a word embedding vector, and then input the word embedding vector into the first encoding model to generate the first encoding information.

[0032] In the second step, second coding information corresponding to the first coding information is determined from the pre-established second coding information set as the second target coding information. The second coding information set is generated based on the second coding model. The second coding information in the second coding information set can be information in vector form. Each second coding information in the second coding information set has corresponding first coding information. In practice, the second coding model can be a convolutional neural network model with 6 layers connected in series.

[0033] It should be noted that the second coding information set can be generated by the following steps:

[0034] Step 1: Get the full data domain information set.

[0035] Step 2: for each piece of full data field information in the full data field information set, use the second coding model to generate second coding information for the full data field information.

[0036] The third step is to input the second target encoding information into the first decoding model to generate decoding information. The decoding information can represent representative information of the data domain information. In practice, the decoding information can be an information identifier corresponding to the data domain information. For example, the decoding information can be "563782".

[0037] The fourth step is to generate a first data query statement for the above decoded information.

[0038] As an example, first, the execution subject may obtain a data query statement expression, and then add the decoded information to the data query statement expression to generate a data query statement as the first data query statement.

[0039] The fifth step is to obtain the target data set from the target storage database using the first data query statement.

[0040] As an example, the first data query is performed to obtain a target data set from the target storage database.

[0041] In some optional implementations of some embodiments, the step of acquiring the target data set from the target storage database according to the data domain information may include the following steps:

[0042] The first step is to obtain the data field identifier corresponding to the above data field information. The data field identifier may be information representing the data field information. For example, the data field identifier may be "0876".

[0043] The second step is to obtain a random number for the above data field identifier.

[0044] The third step is to encode the random number and the data field identifier respectively to generate random encoding information and data field encoding information, wherein the random encoding information and the data field encoding information can be information in the form of vectors.

[0045] As an example, the execution subject may use a data encoding model to encode the random number and the data domain identifier respectively to generate random encoding information and data domain encoding information. For example, the data encoding model may be a Bret pre-training model.

[0046] Step 4: Randomly select an information splicing method from the information splicing method set as the target information splicing method. The information splicing method in the information splicing method set may be a method for splicing at least two pieces of information. The information splicing method may be, but is not limited to, at least one of the following: a left-right information splicing method, and a top-bottom information splicing method.

[0047] Step 5: According to the target information splicing method, the random coded information and the data field coded information are spliced ​​to obtain spliced ​​information. For example, the target information splicing method can be a left-right information splicing method. That is, using the target information splicing method, the spliced ​​information can be [random coded information, data field coded information].

[0048] In step 6, the splicing information is input into a plurality of serially connected downsampling layers included in the third coding model to generate a feature extraction vector. The downsampling layer may be a convolutional neural network model. The vector dimension corresponding to the feature extraction vector is smaller than the vector dimension corresponding to the splicing information.

[0049] In the seventh step, the feature extraction vector is input into the vector element random shuffle model included in the third coding model to generate a shuffled feature vector as the third coding information. The number of vector elements included in the third coding information is the target number. The size of the vector elements included in the third coding information is in the target numerical range and is an integer. The vector element random shuffle model can be a model for randomly arranging each element in the vector. The target number can be a preset value. For example, the target number can be 5. The target numerical range can be a preset numerical range. For example, the target numerical range can be [0,6].

[0050] In step 8, the third encoding information is sent to a decoding end to obtain the target data set obtained from the target storage database sent by the decoding end. The decoding end may be a terminal where a decoding model is deployed.

[0051] Optionally, the target dataset is obtained by the following steps:

[0052] The first step is to receive the method information corresponding to the target information splicing method, the random number, the random shuffle order information corresponding to the random shuffle model of the vector elements, and the third encoding information. The random shuffle order information can represent the order information of the random shuffle of each element in the vector.

[0053] The second step is to input the third encoded information into a vector element restoration model included in the second decoding model deployed by the decoding end to generate the restored vector information. The vector element restoration model is a fully connected layer built based on the random scrambled order information. The vector element restoration model can be a fully connected layer that performs element restoration on the third encoded information.

[0054] The third step is to input the restored vector information into a plurality of serially connected upsampling layers included in the second decoding model to obtain upsampled feature information. The vector dimension corresponding to the upsampled feature information is the same as the vector dimension corresponding to the concatenated information. The vector dimension corresponding to the upsampled feature information is higher than the vector dimension corresponding to the restored vector information.

[0055] The fourth step is to extract information from the up-sampled feature information according to the method information to generate candidate random numbers and candidate data domain identifiers.

[0056] In a fifth step, in response to determining that the candidate random number is the same as the random number, a second data query statement for the candidate data field identifier is generated.

[0057] As an example, the execution entity may add the candidate data domain identifier to the data query statement expression to generate a data query statement as the second data query statement.

[0058] Step 6: According to the second data query statement, the target data set is obtained from the target storage database.

[0059] As an example, the execution subject may execute a second data query statement to obtain a target data set from the target storage database.

[0060] Step 103: Perform at least one data interception operation on the target data set according to the data volume information and the data range information to obtain at least one data subset.

[0061] In some embodiments, the execution subject may perform at least one data interception operation on the target data set according to the data volume information and the data range information to obtain at least one data subset, wherein the data volume information corresponding to each data subset in the at least one data subset is the same as the data volume information and is within the data range information.

[0062] As an example, the execution subject may perform at least one data random interception operation on the target data set according to the data volume information and the data range information to obtain at least one data subset.

[0063] Step 104: for each data subset in the at least one data subset, perform the following first determination step:

[0064] Step 1041, determining the data importance information and data impact information corresponding to the above data subset.

[0065] In some embodiments, the execution entity may determine the data importance information and data impact information corresponding to the data subset. The data importance information may represent the importance of each data in the data subset. In practice, the data importance information may be a numerical value between 0 and 100. The higher the numerical value, the higher the importance of each data in the data subset. The data impact information may represent the impact of each data in the data subset. In practice, the data impact information may be a numerical value between 0 and 100. The higher the numerical value, the higher the impact of each data in the data subset.

[0066] As an example, first, the execution subject may perform data encoding on each data in the data subset to generate encoding information and obtain the encoding information subset. Then, each encoding information in the encoding information subset is information fused to generate fused information. Next, the acquired data purpose information is information encoded to generate data purpose encoding information. Next, the fused information and data purpose encoding information are input into the multi-layer serially connected fully connected layer to generate data importance information for the data subset. Finally, the fused information and data purpose encoding information are input into the multi-layer serially connected LSTM model to generate data impact information.

[0067] Step 1042: Generate a data quality score for the data subset using a data quality score generation model based on the data importance information and the data impact information.

[0068] In some embodiments, the execution subject may generate a data quality score for the data subset using a data quality score generation model based on the data importance information and the data impact information. The data quality score generation model may be a neural network model for generating data quality scores. In practice, the data quality score may characterize the quality of the data corresponding to the data subset. The higher the data quality score, the better the data quality corresponding to the data subset.

[0069] As an example, first, the execution subject may encode each data in the data subset to generate encoded data and obtain the encoded data subset. Then, each encoded data in the encoded data subset is spliced ​​to generate spliced ​​encoded data. Next, the spliced ​​encoded data, the data importance information, and the data impact information are input into the data quality score generation model to generate a data quality score for the data subset.

[0070] Taking into account the problems of the above conventional solutions, facing the above technical problem 2: the output results of the data quality score generation model cannot be effectively evaluated, resulting in the output data quality score being not accurate enough. Combined with the advantages / technical status, we can decide to adopt the following solution.

[0071] In some optional implementations of some embodiments, based on the data importance information and the data impact information, a data quality score generation model is used to generate a data quality score for the data subset, including:

[0072] The first step is to perform data encoding processing on each data in the above data subset to generate encoding information and obtain an encoding information subset.

[0073] The second step is to classify the attributes of each data in the above data subset to obtain an attribute data set group.

[0074] The third step is to perform data overall coding processing on each attribute data set in the above attribute data set group to generate data overall coding information and obtain a data overall coding information group.

[0075] In the fourth step, the above-mentioned encoded information subset is input into the data subset overall semantic feature information extraction model included in the above-mentioned data quality score generation model to generate the overall semantic feature information of the data subset. Among them, the data subset overall semantic feature information extraction model can be a neural network model that generates the overall semantic feature information of the data subset. Among them, the data subset overall semantic feature information extraction model includes: a data domain feature extraction model, a data attribute feature extraction model, and a data attribute distribution feature extraction model. Among them, the data domain feature extraction model can be a neural network model that generates feature semantic information of the data domain corresponding to the data subset. The data domain feature extraction model can be a generative and adversarial neural network model. The data attribute feature extraction model can be a neural network model that generates data attribute feature information. The data attribute feature information can be the overall attribute feature semantic information corresponding to each attribute set of the data subset. The data attribute feature extraction model can be an attribute feature extraction model based on a knowledge graph. In practice, the data attribute feature extraction model can be a multi-layer parallel connected convolutional neural network model + a multi-head attention mechanism model. The data attribute distribution feature extraction model can be a neural network model that extracts attribute distribution feature information corresponding to each attribute information corresponding to the data subset. In practice, the data attribute distribution feature extraction model can be a multi-head attention mechanism model.

[0076] The fifth step is to splice the above-mentioned coding information subset, the above-mentioned data subset overall semantic feature information and the above-mentioned data overall coding information group to generate splicing information.

[0077] In the sixth step, the splicing information is input into the data importance information generation model included in the data quality score generation model to generate initial data importance information. The data importance information generation model may be a neural network model for generating data importance information. In practice, the data importance information generation model may be a first convolutional neural network model with multiple layers connected in series.

[0078] In the seventh step, the above-mentioned splicing information is input into the data impact degree information generation model included in the above-mentioned data quality score generation model to generate initial data impact degree information. Among them, the data impact degree information generation model can be a neural network model for generating data impact degree information. In practice, the data impact degree information generation model can be a second convolutional neural network model with multiple layers connected in series.

[0079] The eighth step is to determine the first information difference between the initial data importance information and the data importance information, and to determine the second information difference between the initial data influence degree information and the data influence degree information.

[0080] In the ninth step, in response to determining that the first information difference is less than the first predetermined value and the second information difference is less than the second predetermined value, the splicing information is input into the first data quality scoring model included in the data quality scoring generation model to generate a data quality score. The first data quality scoring model can be a neural network model for generating a data quality score. In practice, the data quality scoring model can include an encoding model of a third convolutional neural network connected in series with multiple layers + a decoding model of a fourth convolutional neural network connected in series with multiple layers + a fully connected layer connected in series with multiple layers. Parameter sharing between the data quality scoring model, the data importance information generation model, and the data impact information generation model before the target number of layers. For example, the target number can be 5 layers.

[0081] Step 10: In response to determining that the first information difference is greater than or equal to the first predetermined value, and / or the second information difference is greater than or equal to the second predetermined value, the data importance information, the data impact information and the splicing information are input into the second data quality scoring model to generate a data quality score, wherein the second data quality scoring model includes a multi-layer serial fifth convolutional neural network encoding model + a multi-layer serial sixth convolutional neural network decoding model + a multi-layer serial full connection layer. The encoding model of the multi-layer serial fifth convolutional neural network has more network layers than the encoding model of the multi-layer serial third convolutional neural network. The encoding model of the multi-layer serial sixth convolutional neural network has more network layers than the encoding model of the multi-layer serial fourth convolutional neural network.

[0082] The above-mentioned contents in some optional implementation methods of some embodiments, as an inventive point of the present disclosure, solve the technical problem mentioned in the background technology that "the output result of the data quality score generation model cannot be effectively evaluated, resulting in the output data quality score being not accurate enough". Based on this, the present disclosure uses the data impact information generation model and the data importance information generation model as a verification guarantee for the output result of the data quality score generation model according to the data importance information and the above-mentioned data impact information. On the basis of the first data quality scoring model and the second data quality scoring model, a more accurate data quality score is generated.

[0083] Step 1043: In response to determining that the data quality score is greater than or equal to the first value, determine the data subset as a candidate data subset.

[0084] In some embodiments, in response to determining that the data quality score is greater than or equal to a first value, the execution entity may determine the data subset as a candidate data subset. The first value may be a preset value. For example, the first value may be 70.

[0085] In some optional implementations of some embodiments, after step 1043, the steps further include:

[0086] In the first step, in response to determining that the data quality score corresponding to the data subset is less than the first value and the number of data adjustments corresponding to the data subset is less than a predetermined number, the following second determination step is performed:

[0087] Sub-step 1, adjusting each data in the data subset to generate an adjusted data subset. The predetermined number of times may be a preset number of times. For example, the predetermined number of times may be 5 times.

[0088] As an example, the execution subject may perform fine-tuning on each data in the data subset under the target distribution condition to generate an adjusted data subset. For example, the target distribution may be a normal distribution.

[0089] Sub-step 2: Based on the data importance information and data impact information corresponding to the adjusted data subset, the above data quality score generation model is used to generate a data quality score for the adjusted data subset as the adjusted data quality score. For specific implementation methods, see Generation of Data Quality Score.

[0090] Sub-step 3, in response to determining that the adjusted data quality score is greater than or equal to the above-mentioned first value, or the number of data adjustments is equal to the above-mentioned predetermined number, determining the adjusted data subset as a candidate data subset, and determining the adjusted data quality score as the data quality score.

[0091] In the second step, in response to determining that the candidate data quality score is less than the first value, the adjusted data subset is used as the data subset, the adjusted data quality score is used as the data quality score, and the second determination step is performed again.

[0092] Optionally, the data acquisition information further includes: data acquisition purpose information and data occupied size. The data acquisition purpose information may be information on the purpose of acquiring the data subset. For example, the data acquisition purpose information may be information for image recognition purposes. The data occupied size may be the size of bytes occupied by the data subset.

[0093] Optionally, the above step of adjusting each data in the data subset to generate an adjusted data subset may include the following steps:

[0094] The first step is to determine the data attribute set corresponding to the data subset. The data attribute in the data attribute set can be the attribute name corresponding to the data subset. For example, the data attribute can be but is not limited to at least one of the following: data type, data source, data size.

[0095] The second step is to determine, for each data attribute in the data attribute set, the data completeness information corresponding to the data attribute for the data subset. The data completeness information may be the data completeness of each data in the data attribute set corresponding to the data attribute. The data attribute set may be a data set related to the data attribute in the data subset. The data completeness may characterize the data missingness of the data set.

[0096] The third step is to remove data attributes whose corresponding data integrity information is less than a second value from the above data attribute set to obtain a data attribute set after removal. The second value may be a preset value. For example, the second value may be 0.7.

[0097] The fourth step is to determine the association degree information between each post-removal data attribute in the post-removal data attribute set and the data acquisition target information according to the post-removal data subset corresponding to the post-removal data attribute set.

[0098] As an example, first, the execution subject may perform data encoding fusion on each data in the removed data subset corresponding to the removed data attribute to generate encoding fusion information. Then, the data acquisition purpose information is encoded to generate data acquisition encoding purpose information. Finally, the encoding fusion information and the data acquisition encoding purpose information are input into the correlation degree information generation model to generate the correlation degree information. The correlation degree information generation model may be a convolutional neural network model + a fully connected layer.

[0099] The fifth step is to sort the above-mentioned removed data attribute set according to the association degree information corresponding to each removed data attribute to obtain a first data attribute sequence.

[0100] As an example, the execution subject may sort the removed data attribute set according to the order of association degree information from large to small to obtain a first data attribute sequence.

[0101] The sixth step is to determine the size occupied by the attribute data corresponding to each data attribute in the data attribute set after the removal.

[0102] In the seventh step, the data attribute set after the removal is sorted according to the size occupied by the attribute data corresponding to each data attribute, so as to obtain a second data attribute sequence.

[0103] As an example, the execution subject may sort the removed data attribute set in descending order of the size occupied by the attribute data to obtain a first data attribute sequence.

[0104] In the eighth step, an adjusted data subset for the data subset is generated according to the first data attribute sequence, the second data attribute sequence and the data occupied size.

[0105] Optionally, the generating of the adjusted data subset for the data subset according to the first data attribute sequence, the second data attribute sequence and the data occupied size may include the following steps:

[0106] In the first step, the data attributes in the second data attribute sequence are combined with the data occupied size as a data limit to obtain at least one second data attribute subsequence, wherein the attribute data occupied size corresponding to the attribute data set corresponding to the second data attribute subsequence is not greater than the data occupied size.

[0107] In the second step, according to the attribute order corresponding to the first data attribute sequence, weights are assigned to the first data attributes in the first data attribute sequence to generate an attribute weight sequence, wherein the attribute weights are generated based on the attention mechanism model and are generated for the data acquisition purpose information. The attention mechanism model may be a multi-head attention mechanism model in Transformer.

[0108] As an example, the execution subject may generate an attribute vector for each first data attribute in the first data attribute sequence to generate an attribute vector and obtain an attribute vector sequence. Then, according to the attribute order, each attribute vector in the attribute vector sequence is combined to generate a combined attribute vector. Then, the combined attribute vector is input into the attention mechanism model to generate an attribute weight sequence.

[0109] The third step is to determine whether each second data attribute subsequence in the at least one second data attribute subsequence has a corresponding data quality score, that is, to determine whether the second data attribute subsequence has a data quality score determined in history.

[0110] In the fourth step, in response to the determination of existence, the second data attribute subsequence corresponding to the determined data quality score is removed from the at least one second data attribute subsequence to obtain a second data attribute subsequence set, wherein the second data attribute subsequence in the second data attribute subsequence set is an attribute sequence for which the corresponding data quality score has not been determined.

[0111] Step 5: For each second data attribute subsequence in the second data attribute subsequence set, perform the following third determination step:

[0112] Sub-step 1: Determine the attribute weight sub-sequence corresponding to the second data attribute sub-sequence according to the attribute weight sequence, wherein the second attribute in the second data attribute sub-sequence has a one-to-one correspondence with the attribute weight in the attribute weight sub-sequence.

[0113] Sub-step 2: Add the attribute weights in the attribute weight subsequence to generate an added weight.

[0114] The sixth step is to set the data level corresponding to each second data attribute subsequence in the second data attribute subsequence set according to the obtained addition weight set.

[0115] As an example, the execution subject may set the data level corresponding to each second data attribute subsequence in the second data attribute subsequence set according to the obtained set of added weights according to the data level association table. The data level association table may be a table that represents the association between the data level and the corresponding value range.

[0116] In the seventh step, the second data attribute subsequence with the highest corresponding data level in the second data attribute subsequence set is determined as a candidate second data attribute subsequence.

[0117] In the eighth step, data fine-tuning is performed on each data in the attribute data set corresponding to the candidate second data attribute subsequence to generate a fine-tuned attribute data set as the adjusted data subset.

[0118] Step 105 , selecting a number of candidate data subsets whose corresponding data quality scores are within the top target number from the at least one candidate data subset obtained, to obtain a candidate data subset group.

[0119] In some embodiments, the execution subject may select a target number of candidate data subsets with corresponding data quality scores from the obtained at least one candidate data subset to obtain a candidate data subset group. The target number may be a preset value. For example, the target number may be 3.

[0120] Step 106: Generate an actual data set to be acquired for the data acquisition information according to the candidate data subset group.

[0121] In some embodiments, the execution entity may generate an actual data set to be acquired for the data acquisition information based on the candidate data subset group.

[0122] As an example, the execution entity may randomly select a candidate data subset from the candidate data subset group to generate an actual data set to be acquired for the data acquisition information.

[0123] In some optional implementations of some embodiments, generating an actual data set to be acquired for the data acquisition information according to the candidate data subset group includes the following steps:

[0124] The first step is to obtain the data acquisition purpose information included in the data acquisition information.

[0125] The second step is to obtain at least one model corresponding to the above data acquisition target information, wherein each model in the at least one model can be a pre-trained neural network model. For example, the data acquisition target information is image recognition target information. The model in the at least one model can be an image recognition model. The model structures of the models in the at least one model are different.

[0126] In the third step, for each candidate data subset in the above candidate data subset group, perform the following generation steps:

[0127] The first sub-step is to input the candidate data subset into each model of at least one model to generate a model output result, thereby obtaining at least one model output result.

[0128] The second sub-step is to determine the model output weight corresponding to the at least one model. The model output weight represents the model importance of the corresponding model. The model output weight can be a value between 0 and 1. The model output weight can be a weight obtained through multiple model trainings that represents the accuracy of the model.

[0129] The third sub-step is to perform weighted summation processing on the at least one model output result and the at least one model output weight to obtain a weighted value.

[0130] The fourth step is to perform weighted summation on the weighted values ​​in the obtained weighted value group and the data quality scores in the data quality score group to generate a weighted sum value group.

[0131] The fifth step is to select the candidate data subset with the highest corresponding weighted sum value from the candidate data subset group as the actual data set to be obtained.

[0132] Step 107 : sending the actual data set to be acquired to the data storage device connected to the target data acquisition platform and inserted by the user through asynchronous transmission.

[0133] In some embodiments, the execution subject may send the actual data set to be acquired to a data storage device connected to the target data acquisition platform and inserted by the user through asynchronous transmission. The data storage device may be a device for storing data sets. For example, the data storage device may be a mobile hard disk.

[0134] The above-mentioned embodiments of the present disclosure have the following beneficial effects: through the data acquisition method of some embodiments of the present disclosure, the required high-quality data set can be accurately obtained under the premise of determining that the data will not be leaked. Specifically, the reason why the relevant data set is not of high quality is that the volume of the required data set is generally large, the direct collection method has a long collection time, and the collected data is often incomplete, resulting in poor quality of the collected data set. In addition, through the purchase method, the purchased data set may have a large deviation from the required data set, resulting in the quality of the obtained data set is also poor, which greatly affects the accuracy of subsequent business. Based on this, the data acquisition method of some embodiments of the present disclosure first obtains the data acquisition information input in the target data acquisition platform. Among them, the above-mentioned data acquisition information includes: data volume information, data range information and data field information. Here, the data volume information, data range information and data field information in the acquired data acquisition information are used for subsequent high-quality data set acquisition. Then, according to the above-mentioned data field information, the target data set is obtained from the target storage database as a basic data set to obtain subsequent high-quality data sets. Then, according to the above data volume information and the above data range information, at least one data interception operation is performed on the above target data set to obtain at least one data subset, so as to accurately obtain a high-quality data set by multiple data interceptions. Then, for each data subset in the above at least one data subset, the following first determination step is performed: the first step is to determine the data importance information and data impact information corresponding to the above data subset. Here, by determining the data importance information and data impact information as relevant indicator information for determining data quality, the numerical quality corresponding to each data subset is determined, so as to facilitate the subsequent determination of an accurate high-quality data set. The second step is to accurately generate a data quality score for the above data subset based on the above data importance information and the above data impact information, using a data quality score generation model, so as to characterize the data quality of the data subset. The third step is to determine the above data subset as a candidate data subset in response to determining that the data quality score is greater than or equal to the first value. Furthermore, a target number of candidate data subsets with corresponding data quality scores are selected from the obtained at least one candidate data subset to obtain a candidate data subset group, so as to preliminarily select a target number of candidate data subsets with relatively high quality, so as to make a more accurate determination of the suitability of the subsequent data set. Furthermore, based on the above candidate data subset group, an actual data set to be acquired for the above data acquisition information can be accurately generated as an accurate and high-quality data set. Finally, the above actual data set to be acquired is sent to the data storage device connected to the above target data acquisition platform and inserted by the user through asynchronous transmission.In summary, by acquiring data information including data volume information, data range information and data domain information, as well as determining the data importance information, data impact information and subsequent data quality score corresponding to the data subset, the required high-quality data set can be accurately obtained under the premise of ensuring that the data will not be leaked.

[0135] Further references Figure 2 As an implementation of the methods shown in the above figures, the present disclosure provides some embodiments of a data acquisition device. These device embodiments are similar to Figure 1 Corresponding to the method embodiments shown, the data acquisition device can be specifically applied to various electronic devices.

[0136] like Figure 2 As shown, a data acquisition device 200 includes: a first acquisition unit 201, a second acquisition unit 202, a capture unit 203, an execution unit 204, a screening unit 205, a generation unit 206 and a sending unit 207. The first acquisition unit 201 is configured to acquire data acquisition information input in a target data acquisition platform, wherein the data acquisition information includes: data volume information, data range information and data domain information; the second acquisition unit 202 is configured to acquire a target data set from a target storage database according to the data domain information; the capture unit 203 is configured to perform at least one data capture operation on the target data set according to the data volume information and the data range information to obtain at least one data subset; the execution unit 204 is configured to perform the following first determination step for each data subset in the at least one data subset: determine the data importance information and data influence information corresponding to the data subset; and determine the data importance information and data influence information according to the data importance information. The degree information and the above-mentioned data influence degree information are used to generate a data quality score for the above-mentioned data subset using a data quality score generation model; in response to determining that the above-mentioned data quality score is greater than or equal to the first value, the above-mentioned data subset is determined as a candidate data subset; the screening unit 205 is configured to screen out candidate data subsets whose corresponding data quality scores are in the front target number from the at least one obtained candidate data subset to obtain a candidate data subset group; the generating unit 206 is configured to generate an actual data set to be obtained for the above-mentioned data acquisition information according to the above-mentioned candidate data subset group; the sending unit 207 is configured to send the above-mentioned actual data set to be obtained to the data storage device connected to the above-mentioned target data acquisition platform and inserted by the user through asynchronous transmission.

[0137] It is understandable that the units described in the data acquisition device 200 are similar to those described in the reference Figure 1Therefore, the operations, features and beneficial effects described above for the method are also applicable to the data acquisition device 200 and the units included therein, and will not be described in detail here.

[0138] Reference below Figure 3 , which shows a structural schematic diagram of an electronic device (eg, an electronic device) 300 suitable for implementing some embodiments of the present disclosure. Figure 3 The electronic device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present disclosure.

[0139] like Figure 3 As shown, the electronic device 300 may include a processing device (e.g., a central processing unit, a graphics processing unit, etc.) 301, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 302 or a program loaded from a storage device 308 into a random access memory (RAM) 303. In the RAM 303, various programs and data required for the operation of the electronic device 300 are also stored. The processing device 301, the ROM 302, and the RAM 303 are connected to each other via a bus 304. An input / output (I / O) interface 305 is also connected to the bus 304.

[0140] Typically, the following devices may be connected to the I / O interface 305: input devices 306 including, for example, a touch screen, a touch pad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, etc.; output devices 307 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; storage devices 308 including, for example, a magnetic tape, a hard disk, etc.; and communication devices 309. The communication devices 309 may allow the electronic device 300 to communicate with other devices wirelessly or by wire to exchange data. Although Figure 3 The electronic device 300 is shown with various devices, but it should be understood that it is not required to implement or possess all the devices shown. More or fewer devices may be implemented or possessed instead. Figure 3 Each block shown in the figure may represent one device, or may represent multiple devices as required.

[0141] In particular, according to some embodiments of the present disclosure, the process described above with reference to the flowchart can be implemented as a computer software program. For example, some embodiments of the present disclosure include a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In some such embodiments, the computer program can be downloaded and installed from the network through the communication device 309, or installed from the storage device 308, or installed from the ROM 302. When the computer program is executed by the processing device 301, the above-mentioned functions defined in the method of some embodiments of the present disclosure are executed.

[0142] It should be noted that the computer-readable medium in some embodiments of the present disclosure may be a computer-readable signal medium or a computer-readable storage medium or any combination of the two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In some embodiments of the present disclosure, the computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, device or device. In some embodiments of the present disclosure, the computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, in which a computer-readable program code is carried. This propagated data signal may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. The computer readable signal medium may also be any computer readable medium other than a computer readable storage medium, which may send, propagate or transmit a program for use by or in conjunction with an instruction execution system, apparatus or device. The program code contained on the computer readable medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (radio frequency), etc., or any suitable combination of the above.

[0143] In some embodiments, the client and the server may communicate using any currently known or future developed network protocol such as HTTP (HyperText Transfer Protocol), and may be interconnected with any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network ("LAN"), a wide area network ("WAN"), an internet (e.g., the Internet), and a peer-to-peer network (e.g., an ad hoc peer-to-peer network), as well as any currently known or future developed network.

[0144] The computer-readable medium may be included in the electronic device, or may exist independently without being incorporated into the electronic device. The computer-readable medium carries one or more programs. When the one or more programs are executed by the electronic device, the electronic device: obtains data acquisition information input in the target data acquisition platform, wherein the data acquisition information includes: data volume information, data range information and data domain information; obtains the target data set from the target storage database according to the data domain information; performs at least one data interception operation on the target data set according to the data volume information and the data range information to obtain at least one data subset; for each data subset in the at least one data subset, performs the following first determination step: determines the data importance information and data influence information corresponding to the data subset; generates a data quality score for the data subset according to the data importance information and the data influence information using a data quality score generation model; in response to determining that the data quality score is greater than or equal to a first value, determines the data subset as a candidate data subset; selects candidate data subsets whose corresponding data quality scores are in the first target number from the obtained at least one candidate data subset to obtain a candidate data subset group; generates an actual data set to be acquired for the data acquisition information according to the candidate data subset group;

[0145] The actual data set to be acquired is sent to the data storage device connected to the target data acquisition platform and inserted by the user through asynchronous transmission.

[0146] Computer program code for performing the operations of some embodiments of the present disclosure may be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, C++, and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0147] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present disclosure. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some implementations as replacements, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0148] The units described in some embodiments of the present disclosure may be implemented by software or by hardware. The units described may also be provided in a processor, for example, may be described as: a processor including a first acquisition unit, a second acquisition unit, an interception unit, an execution unit, a screening unit, a generation unit, and a sending unit. The names of these units do not, in some cases, constitute limitations on the units themselves, for example, the first acquisition unit may also be described as a "unit for acquiring data acquisition information input into a target data acquisition platform".

[0149] The functions described above herein may be performed at least in part by one or more hardware logic components. For example, without limitation, exemplary types of hardware logic components that may be used include: field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), and the like.

[0150] The above descriptions are only some preferred embodiments of the present disclosure and an explanation of the technical principles used. Those skilled in the art should understand that the scope of the invention involved in the embodiments of the present disclosure is not limited to the technical solutions formed by a specific combination of the above-mentioned technical features, but should also cover other technical solutions formed by any combination of the above-mentioned technical features or their equivalent features without departing from the above-mentioned inventive concept. For example, the above-mentioned features are replaced with the technical features with similar functions disclosed in the embodiments of the present disclosure (but not limited to) and the technical solutions formed.

Claims

1. A data acquisition method, comprising: Acquire data acquisition information inputted in the target data acquisition platform, wherein the data acquisition information includes: data volume information, data range information and data domain information; According to the data domain information, obtaining a target data set from a target storage database; According to the data volume information and the data range information, performing at least one data interception operation on the target data set to obtain at least one data subset; For each data subset in the at least one data subset, the following first determination step is performed: Determine the data importance information and data impact information corresponding to the data subset; Generate a data quality score for the data subset using a data quality score generation model according to the data importance information and the data impact information; In response to determining that the data quality score is greater than or equal to a first value, determining the data subset as a candidate data subset; In response to determining that the data quality score corresponding to the data subset is less than the first value and the number of data adjustments corresponding to the data subset is less than a predetermined number, performing the following second determination step: Adjusting each data in the data subset to generate an adjusted data subset; According to the data importance information and the data impact information corresponding to the adjusted data subset, the data quality score generation model is used to generate a data quality score for the adjusted data subset as the adjusted data quality score; In response to determining that the adjusted data quality score is greater than or equal to the first value, or the number of data adjustments is equal to the predetermined number, determining the adjusted data subset as a candidate data subset, and determining the adjusted data quality score as the data quality score; In response to determining that the candidate data quality score is less than the first value, using the adjusted data subset as the data subset, using the adjusted data quality score as the data quality score, and performing the second determining step again; Selecting a number of candidate data subsets whose corresponding data quality scores are in the front target number from the at least one candidate data subset obtained, and obtaining a candidate data subset group; Generating an actual data set to be acquired for the data acquisition information according to the candidate data subset group; The actual data set to be acquired is sent to the data storage device connected to the target data acquisition platform and inserted by the user through asynchronous transmission.

2. The method according to claim 1, wherein: The acquiring the target data set from the target storage database according to the data domain information includes: Generate first encoding information for the data acquisition information using a first encoding model according to the data domain information; Determine, from a pre-established second coding information set, second coding information that has a corresponding relationship with the first coding information as second target coding information, wherein the second coding information set is generated based on a second coding model; Inputting the second target encoded information into a first decoding model to generate decoded information; generating a first data query statement for the decoded information; The target data set is acquired from the target storage database using the first data query statement.

3. The method according to claim 1, wherein: The data acquisition information also includes: data acquisition purpose information and data occupied size; and The step of adjusting each data in the data subset to generate an adjusted data subset includes: Determining a data attribute set corresponding to the data subset; For each data attribute in the data attribute set, determining data completeness information corresponding to the data attribute and for the data subset; Removing data attributes whose corresponding data integrity information is less than a second value from the data attribute set to obtain a data attribute set after removal; Determining, according to the removed data subset corresponding to the removed data attribute set, information on the degree of association between each removed data attribute in the removed data attribute set and the data acquisition target information; sorting the removed data attribute set according to the association degree information corresponding to each removed data attribute to obtain a first data attribute sequence; Determine the size occupied by the attribute data corresponding to each data attribute in the removed data attribute set; According to the size occupied by the attribute data corresponding to each data attribute, the removed data attribute set is sorted to obtain a second data attribute sequence; An adjusted data subset for the data subset is generated according to the first data attribute sequence, the second data attribute sequence and the data occupied size.

4. The method according to claim 3, wherein: The step of generating an adjusted data subset for the data subset according to the first data attribute sequence, the second data attribute sequence and the data occupied size includes: Taking the data occupied size as the data limit, combining the data attributes in the second data attribute sequence to obtain at least one second data attribute subsequence, wherein the attribute data occupied size corresponding to the attribute data set corresponding to the second data attribute subsequence is not greater than the data occupied size; According to the attribute order corresponding to the first data attribute sequence, weighting each first data attribute in the first data attribute sequence to generate an attribute weight sequence, wherein the attribute weight is generated based on the attention mechanism model and is generated for the data acquisition purpose information; determining whether each second data attribute subsequence in the at least one second data attribute subsequence determines a corresponding data quality score; In response to determining that it exists, removing the second data attribute subsequence corresponding to the determined data quality score from the at least one second data attribute subsequence to obtain a second data attribute subsequence set; For each second data attribute subsequence in the second data attribute subsequence set, performing the following third determination step: Determining, according to the attribute weight sequence, an attribute weight subsequence corresponding to the second data attribute subsequence; Adding each attribute weight in the attribute weight subsequence to generate an added weight; According to the obtained set of added weights, setting a data level corresponding to each second data attribute subsequence in the second data attribute subsequence set; Determine the second data attribute subsequence with the highest corresponding data level in the second data attribute subsequence set as a candidate second data attribute subsequence; Data fine-tuning is performed on each data in the attribute data set corresponding to the candidate second data attribute subsequence to generate a fine-tuned attribute data set as the adjusted data subset.

5. The method according to claim 1, wherein: The acquiring the target data set from the target storage database according to the data domain information includes: Obtaining a data field identifier corresponding to the data field information; Obtaining a random number for the data field identifier; Encoding the random number and the data field identifier respectively to generate random coding information and data field coding information; Randomly select an information splicing method from the information splicing method set as a target information splicing method; According to the target information splicing method, the random coded information and the data field coded information are spliced ​​to obtain spliced ​​information; Inputting the splicing information into a plurality of serially connected downsampling layers included in a third encoding model to generate a feature extraction vector; Inputting the feature extraction vector into a vector element random scrambling model included in the third coding model to generate a scrambled feature vector as third coding information, wherein the number of vector elements included in the third coding information is a target number, and the size of the vector elements included in the third coding information is in a target numerical range and is an integer; The third encoding information is sent to a decoding end to obtain a target data set obtained in the target storage database sent by the decoding end.

6. The method according to claim 5, wherein: The target data set is obtained by the following steps: Receive method information corresponding to the target information splicing method, the random number, random shuffle order information corresponding to the vector element random shuffle model, and the third encoding information; Inputting the third encoded information into a vector element restoration model included in a second decoding model deployed at the decoding end to generate restored vector information, wherein the vector element restoration model is a fully connected layer constructed based on the randomly shuffled order information; Inputting the restored vector information into a plurality of serially connected upsampling layers included in the second decoding model to obtain upsampling feature information, wherein the dimension of the vector corresponding to the upsampling feature information is the same as the dimension of the vector corresponding to the splicing information; According to the method information, extracting information from the up-sampled feature information to generate candidate random numbers and candidate data domain identifiers; In response to determining that the candidate random number is the same as the random number, generating a second data query statement for the candidate data field identifier; According to the second data query statement, a target data set is acquired from the target storage database.

7. A data acquisition device, comprising: A first acquisition unit is configured to acquire data acquisition information input in a target data acquisition platform, wherein the data acquisition information includes: data volume information, data range information and data domain information; A second acquisition unit is configured to acquire a target data set from a target storage database according to the data domain information; a capture unit configured to perform at least one data capture operation on the target data set according to the data volume information and the data range information to obtain at least one data subset; The execution unit is configured to perform the following first determination step for each data subset in the at least one data subset: determine the data importance information and data impact information corresponding to the data subset; generate a data quality score for the data subset using a data quality score generation model based on the data importance information and the data impact information; in response to determining that the data quality score is greater than or equal to a first value, determine the data subset as a candidate data subset; in response to determining that the data quality score corresponding to the data subset is less than the first value and the number of data adjustments corresponding to the data subset is less than a predetermined number, perform the following second determination step: perform a adjusting to generate an adjusted data subset; using the data quality score generation model to generate a data quality score for the adjusted data subset as the adjusted data quality score according to the data importance information and the data impact information corresponding to the adjusted data subset; in response to determining that the adjusted data quality score is greater than or equal to the first value, or the number of data adjustments is equal to the predetermined number of times, determining the adjusted data subset as a candidate data subset, and determining the adjusted data quality score as the data quality score; in response to determining that the candidate data quality score is less than the first value, determining the adjusted data subset as a data subset, determining the adjusted data quality score as the data quality score, and performing the second determining step again; A screening unit is configured to screen out a number of candidate data subsets whose corresponding data quality scores are in the front target number from the obtained at least one candidate data subset to obtain a candidate data subset group; A generating unit, configured to generate an actual data set to be acquired for the data acquisition information according to the candidate data subset group; The sending unit is configured to send the actual data set to be acquired to the data storage device connected to the target data acquisition platform and inserted by the user through asynchronous transmission.

8. An electronic device, comprising: one or more processors; a storage device having one or more programs stored thereon, When the one or more programs are executed by the one or more processors, the one or more processors implement the method according to any one of claims 1 to 6.

9. A computer readable medium having a computer program stored thereon, wherein: When the program is executed by a processor, the method according to any one of claims 1 to 6 is implemented.

Images