Data acquisition method and device, computer equipment and readable storage medium

Abstract

The embodiment of the invention provides a data acquisition method and device, computer equipment and a readable storage medium. The method comprises the steps that data demand information is acquired; the data demand information carries data feature information; determining a plurality of target training sample data from the initial training sample data according to the data feature information and label information corresponding to the initial training sample data; querying a preset database, and obtaining target storage corresponding to each piece of target training sample data; and sending a data acquisition request to each target storage to acquire target training sample data. According to the data acquisition mode provided by the embodiment of the invention, different training data can be integrated, and the appropriate training sample data can be automatically requested and acquired based on the demand of the user for the training data, so that the effect of subsequent model training is ensured.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of artificial intelligence technology, and specifically to a data acquisition method, apparatus, computer equipment, and readable storage medium. Background Art

[0002] With the development of AI (Artificial Intelligence) technology, more and more companies will choose to develop AI projects, such as training neural network models for various purposes to undertake various businesses.

[0003] Among them, the development of AI projects often requires reliance on massive amounts of sample data. Technical personnel often download and use sample data from professional websites. As a result, as the number of projects increases, the requirements for enterprise platform storage are also increasing. In addition, under normal circumstances, the sample data used is often less relevant to the enterprise business, affecting the effect of AI project development. Summary of the invention

[0004] The embodiments of the present application provide a data acquisition method, apparatus, computer device and readable storage medium, which aim to solve the problem in the prior art of high storage requirements for enterprise platforms and impact on model training effects when acquiring training data.

[0005] On the one hand, an embodiment of the present application provides a data acquisition method, including:

[0006] Acquire data demand information; the data demand information carries data feature information;

[0007] According to the data feature information and the label information corresponding to the preset initial training sample data, a number of target training sample data are determined from the initial training sample data;

[0008] Querying a preset database to obtain target storage corresponding to each of the target training sample data;

[0009] A data acquisition request is sent to each of the target storages to acquire the target training sample data.

[0010] As a feasible embodiment of the present application, after the step of sending a data acquisition request to each of the target storage gateways to acquire the target training sample data, the method further includes:

[0011] Training a target network model based on the target training sample data, and determining effect evaluation information corresponding to the target network model;

[0012] Based on the effect evaluation information, the label information corresponding to the target training sample data is updated to obtain updated label information;

[0013] The association relationship between the updated label information and the target training sample data is stored in a preset database.

[0014] As a feasible embodiment of the present application, the training to obtain a target network model based on the target training sample data includes:

[0015] An initial network model is obtained by training first training sample data carrying data labels in the target training sample data;

[0016] Processing the second training sample data in the target training sample data according to the initial network model to obtain a prediction label corresponding to the second training sample data;

[0017] The initial network model is trained according to the second training sample data, the predicted label corresponding to the second training sample data, the first training sample data and the data label carried by the first training sample data to obtain a target network model.

[0018] As a feasible embodiment of the present application, the determining of a number of target training sample data from the initial training sample data according to the data feature information and the label information corresponding to the preset initial training sample data includes:

[0019] Calculate the relevance of each of the initial training sample data to the data demand information according to the similarity between the label information corresponding to the preset initial training sample data and the data feature information;

[0020] Based on the magnitude relationship of the association degrees, a number of target training sample data are determined from the initial training sample data.

[0021] As a feasible embodiment of the present application, the step of calculating the association degree between each of the initial training sample data and the data demand information according to the similarity between the label information corresponding to the preset initial training sample data and the data feature information includes:

[0022] Obtain the effect level of the label information corresponding to the initial training sample data;

[0023] The similarity between the label information and the data feature information is weighted according to the weight corresponding to the effect level to obtain the correlation between the initial training sample data and the data demand information.

[0024] As a feasible embodiment of the present application, the step of determining a plurality of target training sample data from the initial training sample data according to the data feature information and the label information corresponding to the preset initial training sample data includes:

[0025] According to the domain scene information carried by the data demand information, determining the associated storage associated with the domain scene information from the preset initial storage;

[0026] The training sample data in the associated storage is determined as initial training sample data.

[0027] As a feasible embodiment of the present application, the sending of a data acquisition request to each of the target storages to acquire the target training sample data includes:

[0028] Sending a data request to a target gateway corresponding to each of the target storages based on a reverse proxy service;

[0029] Receive target training sample data returned by each target gateway in response to the data request.

[0030] On the other hand, an embodiment of the present application further provides a data acquisition device, including:

[0031] An acquisition module, used to acquire data demand information; the data demand information carries data feature information;

[0032] A determination module, configured to determine a number of target training sample data from the initial training sample data according to the data feature information and the label information corresponding to the preset initial training sample data;

[0033] A query module is used to query a preset database to obtain target storage corresponding to each target training sample data;

[0034] The request module is used to send a data acquisition request to each of the target storages to acquire the target training sample data.

[0035] On the other hand, an embodiment of the present application further provides a computer device, which includes a processor, a memory, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the above-mentioned data acquisition method.

[0036] On the other hand, an embodiment of the present application further provides a computer-readable storage medium, on which a computer program is stored, and the computer program is executed by a processor to implement the above-mentioned data acquisition method.

[0037] The data acquisition method provided by the present application, after obtaining the user's data demand information, uses the data feature information carried in the data demand information and the label information corresponding to the initial training sample data to determine the target training sample data that meets the data demand information from the initial training sample data, and then after querying the storage corresponding to the target training sample data, sends a data acquisition request to the target storage to obtain the target training sample data. The data acquisition method provided by the embodiment of the present application can integrate different training data, and automatically request to obtain appropriate training sample data based on the user's demand for training data, thereby ensuring the effect of subsequent model training. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative work.

[0039] Figure 1 A schematic diagram of a data acquisition method according to an embodiment of the present invention;

[0040] Figure 2 A schematic diagram of a process flow for updating labels of training sample data provided in an embodiment of the present application;

[0041] Figure 3 A schematic diagram of a process flow of obtaining a target network model through semi-supervised training provided in an embodiment of the present application;

[0042] Figure 4 A schematic diagram of a process flow for determining target training sample data based on similarity provided in an embodiment of the present application;

[0043] Figure 5 A schematic diagram of a step flow of calculating a correlation degree based on weighted calculation provided in an embodiment of the present application;

[0044] Figure 6 A schematic diagram of a process flow for screening initial training sample data provided in an embodiment of the present application;

[0045] Figure 7 A schematic diagram of a process flow for obtaining target training sample data based on a reverse proxy service provided in an embodiment of the present application;

[0046] Figure 8 A schematic diagram of the structure of a data acquisition device provided in an embodiment of the present application;

[0047] Fig. 9A schematic diagram of the structure of a computer device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0048] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative work are within the scope of the present invention.

[0049] In the embodiments of the present application, the word "exemplary" is used to mean "used as an example, illustration or description". Any embodiment described as "exemplary" in the embodiments of the present application is not necessarily to be construed as being more preferred or advantageous than other embodiments. The following description is given to enable any technician in the field to implement and use the present invention. In the following description, details are listed for the purpose of explanation. It should be understood that a person of ordinary skill in the art can recognize that the present invention can be implemented without using these specific details. In other examples, well-known structures and processes will not be elaborated in detail to avoid unnecessary details that make the description of the present invention obscure. Therefore, the present invention is not intended to be limited to the embodiments shown, but is consistent with the widest range of principles and features disclosed in the embodiments of the present application.

[0050] To facilitate understanding of the data acquisition method provided in the embodiment of the present application, the implementation scenario diagram of the method is first described. Typically, an enterprise will develop an AI platform system to complete the research and development of AI projects in the enterprise. The training sample data used by different AI projects are usually stored in the storage system of the AI ​​platform system. As the number of AI projects increases, the amount of training sample data stored in the storage system increases, and the requirements for the storage system become higher and higher.

[0051] In order to reasonably utilize the training sample data stored in the AI ​​platform system, an embodiment of the present application provides a data acquisition method, which can respond to user needs and automatically filter out training sample data that meets user needs from the storage system. In this way, when a new AI project needs to be developed, appropriate training sample data can be automatically selected from the data stored in the storage system, while avoiding the introduction of additional new training sample data as much as possible. Specifically, the data acquisition method is installed in a data acquisition device in the form of a computer program, and the data acquisition device is provided in a computer device in the form of a processor. The data acquisition device in the computer device executes the data acquisition method provided in the embodiment of the present application by running a computer program corresponding to the data acquisition method. Figure 1 As shown, Figure 1A schematic diagram of a data acquisition method provided in an embodiment of the present application, specifically, comprising steps S110 to S140:

[0052] S110, obtaining data demand information; the data demand information carries data feature information.

[0053] In the embodiments of the present application, data requirement information generally refers to the information required by users, that is, technical developers of enterprises, to complete model training when they need to develop new AI projects, for example, using training sample data to train a neural network model for a specific purpose. For example, more commonly, data requirement information generally includes data feature information used to describe the characteristics of the required training sample data. Specifically, the data feature information may be sample data that needs to be newly stored, sample data with low CPU (Central Processing Unit) consumption, or sample data suitable for PMML (Predictive Model Markup Language), etc. Of course, in addition to the above-mentioned data feature information, the data requirement information may also include other information, such as domain scenario information, which describes the domain restrictions on the training samples that need to be used, such as logistics scenarios, value-added service scenarios, etc., as well as information on the required amount of training sample data, etc.

[0054] S120, determining a number of target training sample data from the initial training sample data according to the data feature information and the preset label information corresponding to the initial training sample data.

[0055] In the embodiment of the present application, the label information corresponding to the training sample data is generally used to describe the data characteristics of the training sample data. For example, similar to the aforementioned data characteristic information, the characteristics of the training sample data can be described from the dimensions of storage time, CPU consumption, and model applicability. Specifically, the label information corresponding to the training sample data can be set based on the evaluation of the neural network model after the neural network model is trained. For specific implementation schemes, please refer to the subsequent Figure 2 and its explanatory contents.

[0056] Further, based on the above, the data acquisition device will use the data feature information in the data demand information to compare with the label information corresponding to each initial training sample data stored in the storage system, and filter out the target training sample data whose label information is similar to the data feature information, that is, the target training sample data that meets the data demand information. For specific implementation schemes for determining target training sample data from initial training sample data, please refer to the following Figure 4 and its explanatory contents.

[0057] In addition, the initial training sample data usually refers to all the training sample data stored in the AI ​​storage system. However, considering that the training sample data used in different domain scenarios are often different, as an optional embodiment of the present application, the data acquisition device can first determine the associated storage associated with this domain scenario information from the storage based on the domain scenario information carried in the data demand information, and then determine the training sample data stored in these storages as the initial training sample data. For specific implementation schemes, please refer to the subsequent Figure 6 and its explanatory contents.

[0058] S130, querying a preset database to obtain target storage corresponding to each of the target training sample data.

[0059] In an embodiment of the present application, in combination with the relevant description provided above, when determining the target training sample data that meets the data requirement information, since different target training sample data are pre-stored in different storages, the data acquisition device can determine the target storage (medium) storing these target training sample data by querying the database.

[0060] S140: Send a data acquisition request to each of the target storages to acquire the target training sample data.

[0061] In an embodiment of the present application, based on the aforementioned determination of the target storage, the corresponding target training sample data can be read from the storage for subsequent AI model training by further sending a data acquisition request to the target storage.

[0062] Further, as a feasible embodiment of the present application, a solution for implementing training sample data storage based on a storage gateway is proposed. The specific solution can be found in the following Figure 7 and its explanatory contents.

[0063] The data acquisition method provided by the present application, after obtaining the user's data demand information, uses the data feature information carried in the data demand information and the label information corresponding to the initial training sample data to determine the target training sample data that meets the data demand information from the initial training sample data, and then after querying the storage corresponding to the target training sample data, sends a data acquisition request to the target storage to obtain the target training sample data. The data acquisition method provided by the embodiment of the present application can integrate different training data, and automatically request to obtain appropriate training sample data based on the user's demand for training data, thereby ensuring the effect of subsequent model training.

[0064] like Figure 2 As shown, Figure 2A schematic flow chart of a step of updating the label of training sample data provided in an embodiment of the present application, specifically, comprising steps S210 to S230:

[0065] S210, obtaining a target network model through training based on the target training sample data, and determining effect evaluation information corresponding to the target network model.

[0066] In an embodiment of the present application, after acquiring the target training sample data that can be used for model training, the data acquisition device will further train the target network model based on the target training sample data. Specifically, under normal circumstances, the training method for the target network model can be based on conventional machine learning, that is, based on a supervised method, and the specific training process of the embodiment of the present application will not be repeated here. However, considering the diversity of the target training sample data, that is, the target training sample data is often the sample data that has been used to train other models in the past, and there may be differences in their corresponding manual annotations. In the absence of corresponding manual annotations, the target training sample data often cannot be directly trained to obtain the target network model based on a supervised method. Therefore, as a feasible embodiment of the present application, the model effect of the trained target network model can be improved based on a semi-supervised method. The specific implementation scheme can be referred to in the subsequent Figure 3 and its explanatory contents.

[0067] On the basis of the target network model obtained through the aforementioned training, the target network model can be further evaluated based on the current model evaluation system. Specifically, the effect evaluation information can usually be evaluated from dimensions such as the model's accuracy, training cycle, etc. Of course, the effect evaluation information usually needs to correspond to the label information corresponding to the training sample data. For example, the training cycle can reflect the CPU consumption of the training sample data to a certain extent, and the accuracy of the model can reflect the adaptability of the training sample data to the model to a certain extent, and so on.

[0068] S220, updating the label information corresponding to the target training sample data based on the effect evaluation information to obtain updated label information.

[0069] In an embodiment of the present application, based on the effect evaluation information corresponding to the target network model obtained above, the label information corresponding to the target training sample data can be updated based on this effect evaluation information to obtain updated label information. Compared with the original label information, the updated label information is obtained based on the effect evaluation information of the trained network model and can therefore better reflect the data characteristics of the target training sample data.

[0070] S230: Store the association relationship between the updated label information and the target training sample data in a preset database.

[0071] In an embodiment of the present application, after the association relationship between the updated label information and the target training sample data is stored in a preset database, the subsequent data acquisition device can first determine the label information corresponding to each initial training sample data, and then determine a number of target training sample data from the initial training sample data in combination with the data feature information in the data demand information.

[0072] The data acquisition method provided in the embodiment of the present application, after training different neural network models and continuously updating and adjusting the label information, the label information corresponding to the final training sample data can well describe the characteristics of the target training sample data. Subsequently, after obtaining the user's data demand information, it can more accurately recommend training sample data that meets the user's needs for model training.

[0073] like Figure 3 As shown, Figure 3 A schematic flow chart of a step of obtaining a target network model through semi-supervised training is provided in an embodiment of the present application, specifically, including steps S310 to S330:

[0074] S310: Train and obtain an initial network model based on first training sample data carrying data labels in the target training sample data.

[0075] In an embodiment of the present application, since the target training sample data comes from different storages, there will often be some first training sample data carrying data labels and second training sample data not carrying data labels in the target training sample data. Specifically, the data labels here refer to labels used to implement supervised training in the model training process. Therefore, the initial network model can be first obtained based on the first training sample data carrying data labels in the target training sample data.

[0076] S320: Process the second training sample data in the target training sample data according to the initial network model to obtain a prediction label corresponding to the second training sample data.

[0077] After the initial network model is obtained through training, the second training sample data that does not carry data labels in the target training sample data is further processed according to the initial network model to obtain the predicted label corresponding to the second training sample data. It should be noted that the predicted label here is often a prediction result, which is different from the true label to a certain extent, but can also be used to train the model to a certain extent.

[0078] S330: Train the initial network model according to the second training sample data, the predicted label corresponding to the second training sample data, the first training sample data, and the data label carried by the first training sample data to obtain a target network model.

[0079] In an embodiment of the present application, based on the foregoing, by further utilizing the first training sample data and the data labels it carries, as well as the second training sample data and its corresponding prediction labels to optimize the initial network model, a target network model with better model effect can be further obtained.

[0080] In the embodiment of the present application, a target network model is obtained by training target training sample data in a semi-supervised manner, thereby overcoming the problem of insufficient label information in the target training sample data and ensuring the model effect of the trained target model.

[0081] like Figure 4 As shown, Figure 4 A schematic flow chart of a step of determining target training sample data based on similarity provided in an embodiment of the present application, specifically, comprising steps S410 to S420:

[0082] S410, calculating the relevance of each of the initial training sample data to the data demand information according to the similarity between the label information corresponding to the preset initial training sample data and the data feature information.

[0083] In the embodiment of the present application, in combination with the description provided above, it can be known that the label information corresponding to the initial training sample data refers to the information describing the data characteristics of the training sample data from the dimensions of resource consumption, model fitness, update time, etc. Similarly, the data feature information is also the information describing the user's needs from the dimensions of resource consumption, model fitness, update time, etc., and this information can usually be in the form of a feature vector (value), wherein the value of each dimension in the feature vector corresponds to the characteristics of the training sample data in a certain dimension, or corresponds to the characteristics of the user's needs in a certain dimension. Therefore, as a feasible embodiment of the present application, the cosine vector metric can be used to calculate the similarity between the label information corresponding to the initial training sample data and the data feature information, so as to obtain the correlation degree describing the degree of correlation between the training sample data and the data demand information. Specifically, the higher the degree of correlation between a certain training sample data and the data demand information, that is, the greater the correlation degree of the initial training sample data corresponding to the data demand information, the more the training sample data meets the data demand information, and vice versa, the smaller the correlation degree of the initial training sample data corresponding to the data demand information, the less the training sample data meets the data demand information.

[0084] Furthermore, in the process of calculating the relevance of the initial training sample data to the data demand information based on the similarity, the effect level of the label information corresponding to the initial training sample data can be further considered. For example, for a certain label information, a value of 1 to 5 can be used to describe the effect level of the initial training sample data corresponding to the label information. For example, the higher the value, the higher the effect level. When calculating the similarity, the label information will be considered more, that is, a higher weight will be set for the label information. For the specific implementation scheme, please refer to the subsequent Figure 5 and its explanatory contents.

[0085] S420: Determine a number of target training sample data from the initial training sample data based on the magnitude relationship of the association degrees.

[0086] In the embodiment of the present application, on the basis of the above, by further comparing the magnitude relationship of the correlation degree, a number of target training sample data can be determined from the initial training sample data. For example, as a feasible embodiment of the present application, the training sample data with a correlation degree higher than a preset threshold can be used as the target training sample data. Of course, as another feasible implementation scheme, when the data demand information also carries the training sample data demand, the data acquisition device can further determine the training sample data with a higher correlation degree in the initial training sample data and a number equal to the training sample data demand as the target training sample data.

[0087] like Figure 5 As shown, Figure 5 A schematic diagram of a flow chart of a step of calculating a correlation degree based on weighted calculation is provided in an embodiment of the present application, specifically, comprising steps S510 to S520:

[0088] S510, obtaining the effect level of the label information corresponding to the initial training sample data.

[0089] In the embodiment of the present application, the effect level of the label information corresponding to the initial training sample data can also be set based on the effect evaluation information corresponding to the target network model provided above, and the embodiment of the present application will not be repeated here.

[0090] S520, weighting the similarity between the label information and the data feature information according to the weight corresponding to the effect level, to obtain the correlation between the initial training sample data and the data demand information.

[0091] In an embodiment of the present application, by setting corresponding weights for different effect levels and weighting the similarity between the label information and the data feature information based on the weights, the correlation between the initial training sample data and the corresponding data demand information can be obtained more accurately.

[0092] like Figure 6 As shown, Figure 6 A schematic flow chart of a step of screening initial training sample data provided in an embodiment of the present application, specifically, comprising steps S610 to S620:

[0093] S610: According to the domain scenario information carried in the data demand information, determine the associated storage associated with the domain scenario information from the preset initial storage.

[0094] In the embodiments of the present application, combined with the relevant description provided above, it can be known that the data demand information usually carries domain scene information, for example, the logistics field, the value-added service field, etc., and different storages often store training sample data in different domain scenes. For example, waybill data is generally training sample data in the logistics field. Therefore, based on the domain scene information carried by the data demand information, the associated storage (medium) associated with the domain scene information can be determined from the preset initial storage, for example, the storage medium storing waybill data, etc.

[0095] S620: Determine the training sample data in the associated storage as initial training sample data.

[0096] In the embodiment of the present application, the training sample data in the associated storage is the training sample data related to the domain scenario information carried by the data demand information. Therefore, this part of the training sample data can be determined as the initial training sample data.

[0097] like Figure 7 As shown, Figure 7 A schematic flow chart of a step of obtaining target training sample data based on a reverse proxy service provided in an embodiment of the present application, specifically, comprising steps S710 to S720:

[0098] S710: Send a data request to a target gateway corresponding to each target storage based on a reverse proxy service.

[0099] In an embodiment of the present application, a solution is provided for storing training sample data in a storage gateway to facilitate acquisition of training sample data. Specifically, different target training sample data are stored in different S3 gateways in the form of S3 protocol and associated in the form of reverse proxy services. At this time, the data acquisition device can query the target gateway corresponding to each target storage based on the reverse proxy service and send a data request to the corresponding gateway.

[0100] S720: Receive target training sample data returned by each target gateway in response to the data request.

[0101] In the embodiment of the present application, after receiving the data request sent by the data acquisition device, the target gateway returns the corresponding target training sample data to the data acquisition device.

[0102] To facilitate understanding of the data acquisition method provided in the embodiment of the present application, the following will be combined with the aforementioned Figure 1 to Figure 7 The content provided provides a complete implementation process of a data acquisition method, specifically, including the following steps:

[0103] The user stores the training sample data in the storage of the s3 protocol through the s3 gateway. For example, each training sample data contains a series of label information, including domain scenario information, and other personalized labels, such as new data, low CPU consumption, model-specific, etc. These training sample data will be scattered in multiple s3 gateways, and the reverse proxy service will store the s3 gateway where each training sample data is located at the same time. At first, the weight of the label in each feature set is the initial value. When the user starts machine learning training, he will specify a data requirement information, which includes data feature information. The reverse proxy service will automatically request multiple s3 gateways according to the user's requirements, and extract the training sample data required by the user, and finally merge these training sample data and return them to the user. After the user uses these training sample data for model training and prediction, the model effect is written to the s3 gateway in the specified format. The reverse proxy service will read the model effect regularly and update the weight of the corresponding label. As the number of feature queries written to the s3 gateway and the number of model training times increases, the weights of the labels will become richer and richer, so the s3 gateway can provide richer functions. For example, when users query the s3 gateway, they only need to enter the data requirement information, and the reverse proxy service will sort them according to the weights of the labels, and request the s3 gateways where the training sample data with the highest rankings are located, and return the merged results to the user.

[0104] In order to better implement the data acquisition method provided in the embodiment of the present application, based on the data acquisition method, the embodiment of the present application also provides a data acquisition device. Figure 8 As shown, Figure 8 A schematic diagram of the structure of a data acquisition device provided in an embodiment of the present application. Specifically, the data acquisition device includes:

[0105] The acquisition module 801 is used to acquire data demand information; the data demand information carries data feature information;

[0106] A determination module 802 is used to determine a number of target training sample data from the initial training sample data according to the data feature information and the label information corresponding to the preset initial training sample data;

[0107] A query module 803 is used to query a preset database to obtain target storage corresponding to each target training sample data;

[0108] The request module 804 is used to send a data acquisition request to each of the target storages to acquire the target training sample data.

[0109] As a feasible embodiment of the present application, the determination module 802 is also used to obtain a target network model based on the target training sample data training, and determine the effect evaluation information corresponding to the target network model; based on the effect evaluation information, update the label information corresponding to the target training sample data to obtain updated label information; store the association relationship between the updated label information and the target training sample data in a preset database.

[0110] As a feasible embodiment of the present application, the determination module 802 is also used to train an initial network model based on the first training sample data carrying data labels in the target training sample data; process the second training sample data in the target training sample data according to the initial network model to obtain the predicted label corresponding to the second training sample data; train the initial network model according to the second training sample data, the predicted label corresponding to the second training sample data, the first training sample data and the data label carried by the first training sample data to obtain the target network model.

[0111] As a feasible embodiment of the present application, the determination module 802 is also used to calculate the correlation between each initial training sample data and the data demand information based on the similarity between the label information corresponding to the preset initial training sample data and the data feature information; based on the size relationship of the correlation, determine a number of target training sample data from the initial training sample data.

[0112] As a feasible embodiment of the present application, the determination module 802 is also used to obtain the effect level of the label information corresponding to the initial training sample data; weight the similarity between the label information and the data feature information according to the weight corresponding to the effect level, and obtain the correlation between the initial training sample data and the data demand information.

[0113] As a feasible embodiment of the present application, the determination module 802 is also used to determine the associated storage associated with the domain scene information from the preset initial storage according to the domain scene information carried by the data demand information; and determine the training sample data in the associated storage as the initial training sample data.

[0114] As a feasible embodiment of the present application, the request module 804 is also used to send data requests to the target gateways corresponding to each target storage based on the reverse proxy service; and receive target training sample data returned by each target gateway in response to the data request.

[0115] The specific definition of the data acquisition device can be found in the definition of the data acquisition method above, which will not be repeated here. Each module in the above data acquisition device can be implemented in whole or in part by software, hardware, or a combination thereof. Each of the above modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory of the computer device in the form of software, so that the processor can call and execute the operations corresponding to each of the above modules. Specifically, Fig. 9 As shown, Fig. 9 A schematic diagram of the structure of a computer device provided in an embodiment of the present application.

[0116] Specifically, the computer device may include one or more processing cores of the processor 901, one or more storage media of the memory 902, a power supply 903, and an input unit 904. Those skilled in the art will understand that Fig. 9 The computer device structure shown in the figure does not constitute a limitation on the computer device, and may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently. Among them:

[0117] The processor 901 is the control center of the computer device. It uses various interfaces and lines to connect various parts of the entire computer device. By running or executing software programs and / or modules stored in the memory 902 and calling data stored in the memory 902, it executes various functions of the computer device and processes data, thereby monitoring the computer device as a whole. Optionally, the processor 901 may include one or more processing cores; preferably, the processor 901 may integrate a data acquisition device and a modem processor, wherein the data acquisition device mainly processes the operating system, user interface, and application programs, and the modem processor mainly processes wireless communications. It is understandable that the above-mentioned modem processor may not be integrated into the processor 901.

[0118] The memory 902 can be used to store software programs and modules. The processor 901 executes various functional applications and data acquisition by running the software programs and modules stored in the memory 902. The memory 902 may mainly include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function (such as a sound playback function, an image playback function, etc.), etc.; the data storage area may store data created according to the use of the computer device, etc. In addition, the memory 902 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other volatile solid-state storage devices. Accordingly, the memory 902 may also include a memory controller to provide the processor 901 with access to the memory 902.

[0119] The computer device also includes a power supply 903 for supplying power to various components. Preferably, the power supply 903 can be logically connected to the processor 901 through a power management system, so as to manage charging, discharging, and power consumption through the power management system. The power supply 903 can also include any components such as one or more DC or AC power supplies, recharging systems, power failure detection circuits, power converters or inverters, and power status indicators.

[0120] The computer device may further include an input unit 904, which may be used to receive input digital or character information and generate keyboard, mouse, joystick, optical or trackball signal input related to user settings and function control.

[0121] Although not shown, the computer device may further include a display unit, etc., which will not be described in detail herein. Specifically in this embodiment, the processor 901 in the computer device will load the executable files corresponding to the processes of one or more application programs into the memory 902 according to the following instructions, and the processor 901 will run the application programs stored in the memory 902, thereby executing the following steps:

[0122] Acquire data demand information; the data demand information carries data feature information;

[0123] According to the data feature information and the label information corresponding to the preset initial training sample data, a number of target training sample data are determined from the initial training sample data;

[0124] Querying a preset database to obtain target storage corresponding to each of the target training sample data;

[0125] A data acquisition request is sent to each of the target storages to acquire the target training sample data.

[0126] As a feasible embodiment of the present application, the processor runs an application stored in the memory and can also perform the following steps: obtain a target network model based on the target training sample data, and determine effect evaluation information corresponding to the target network model; update the label information corresponding to the target training sample data based on the effect evaluation information to obtain updated label information; store the association relationship between the updated label information and the target training sample data in a preset database.

[0127] As a feasible embodiment of the present application, the processor runs an application stored in the memory and can also perform the following steps: training an initial network model based on first training sample data carrying data labels in the target training sample data; processing second training sample data in the target training sample data according to the initial network model to obtain a predicted label corresponding to the second training sample data; training the initial network model according to the second training sample data, the predicted label corresponding to the second training sample data, the first training sample data, and the data label carried by the first training sample data to obtain a target network model.

[0128] As a feasible embodiment of the present application, the processor runs an application stored in the memory and can also perform the following steps: based on the similarity between the label information corresponding to the preset initial training sample data and the data feature information, calculate the association degree between each initial training sample data and the data demand information; based on the size relationship of the association degree, determine a number of target training sample data from the initial training sample data.

[0129] As a feasible embodiment of the present application, the processor runs an application stored in the memory and can also execute the following steps: obtaining the effect level of the label information corresponding to the initial training sample data; weighting the similarity between the label information and the data feature information according to the weight corresponding to the effect level, and obtaining the correlation between the initial training sample data and the data demand information.

[0130] As a feasible embodiment of the present application, the processor runs an application stored in the memory and can also perform the following steps: based on the domain scene information carried by the data demand information, determine the associated storage associated with the domain scene information from the preset initial storage; determine the training sample data in the associated storage as the initial training sample data.

[0131] As a feasible embodiment of the present application, the processor runs the application stored in the memory and can also execute the following steps: sending a data request to the target gateway corresponding to each of the target storages based on the reverse proxy service; receiving the target training sample data returned by each of the target gateways in response to the data request.

[0132] To this end, the embodiment of the present application further provides a computer-readable storage medium, which may include: a read-only memory (ROM), a random access memory (RAM), a disk or an optical disk, etc. The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the following steps are performed:

[0133] Acquire data demand information; the data demand information carries data feature information;

[0134] According to the data feature information and the label information corresponding to the preset initial training sample data, a number of target training sample data are determined from the initial training sample data;

[0135] Querying a preset database to obtain target storage corresponding to each of the target training sample data;

[0136] A data acquisition request is sent to each of the target storages to acquire the target training sample data.

[0137] As a feasible embodiment of the present application, when the computer program is executed by a processor, the following steps may also be performed: obtaining a target network model based on the target training sample data training, and determining effect evaluation information corresponding to the target network model; updating the label information corresponding to the target training sample data based on the effect evaluation information to obtain updated label information; and storing the association relationship between the updated label information and the target training sample data in a preset database.

[0138] As a feasible embodiment of the present application, when the computer program is executed by a processor, the following steps may also be performed: an initial network model is obtained by training based on first training sample data carrying data labels in the target training sample data; second training sample data in the target training sample data is processed according to the initial network model to obtain a predicted label corresponding to the second training sample data; the initial network model is trained according to the second training sample data, the predicted label corresponding to the second training sample data, the first training sample data, and the data label carried by the first training sample data to obtain a target network model.

[0139] As a feasible embodiment of the present application, when the computer program is executed by a processor, the following steps may also be performed: based on the similarity between the label information corresponding to the preset initial training sample data and the data feature information, calculating the correlation between each of the initial training sample data and the data demand information; based on the size relationship of the correlation, determining a number of target training sample data from the initial training sample data.

[0140] As a feasible embodiment of the present application, when the computer program is executed by a processor, the following steps can also be performed: obtaining the effect level of the label information corresponding to the initial training sample data; weighting the similarity between the label information and the data feature information according to the weight corresponding to the effect level, and obtaining the correlation between the initial training sample data and the data demand information.

[0141] As a feasible embodiment of the present application, when the computer program is executed by the processor, the following steps may also be performed: according to the domain scene information carried by the data demand information, determining the associated storage associated with the domain scene information from the preset initial storage; and determining the training sample data in the associated storage as the initial training sample data.

[0142] As a feasible embodiment of the present application, when the computer program is executed by the processor, the following steps may also be performed: sending a data request to the target gateway corresponding to each of the target storages based on the reverse proxy service; receiving the target training sample data returned by each of the target gateways in response to the data request.

[0143] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, please refer to the detailed description of other embodiments above, and will not be repeated here.

[0144] In specific implementation, the above units or structures can be implemented as independent entities, or can be arbitrarily combined to be implemented as the same or several entities. The specific implementation of the above units or structures can refer to the previous method embodiments, which will not be repeated here.

[0145] The specific implementation of the above operations can be found in the previous embodiments, which will not be described in detail here.

[0146] The above is a detailed introduction to a data acquisition method provided in an embodiment of the present application. Specific examples are used herein to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core idea. At the same time, for those skilled in the art, according to the idea of ​​the present invention, there will be changes in the specific implementation method and application scope. In summary, the content of this specification should not be understood as a limitation on the present invention.

Claims

1. A data acquisition method, characterized in that: include: Obtain data requirement information; The data requirement information carries data feature information; According to the data feature information and the label information corresponding to the preset initial training sample data, a number of target training sample data are determined from the initial training sample data; Querying a preset database to obtain target storage corresponding to each of the target training sample data; A data acquisition request is sent to each of the target storages to acquire the target training sample data.

2. The data acquisition method according to claim 1, characterized in that: After the step of sending a data acquisition request to each of the target storage gateways to acquire the target training sample data, the method further includes: Training a target network model based on the target training sample data, and determining effect evaluation information corresponding to the target network model; Based on the effect evaluation information, the label information corresponding to the target training sample data is updated to obtain updated label information; The association relationship between the updated label information and the target training sample data is stored in a preset database.

3. The data acquisition method according to claim 2, characterized in that: The step of training a target network model based on the target training sample data includes: An initial network model is obtained by training first training sample data carrying data labels in the target training sample data; Processing the second training sample data in the target training sample data according to the initial network model to obtain a prediction label corresponding to the second training sample data; The initial network model is trained according to the second training sample data, the predicted label corresponding to the second training sample data, the first training sample data and the data label carried by the first training sample data to obtain a target network model.

4. The data acquisition method according to claim 1, characterized in that: The determining of a number of target training sample data from the initial training sample data according to the data feature information and the preset label information corresponding to the initial training sample data includes: Calculate the relevance of each of the initial training sample data to the data demand information according to the similarity between the label information corresponding to the preset initial training sample data and the data feature information; Based on the magnitude relationship of the association degrees, a number of target training sample data are determined from the initial training sample data.

5. The data acquisition method according to claim 3, characterized in that: The step of calculating the degree of association between each of the initial training sample data and the data demand information according to the similarity between the label information corresponding to the preset initial training sample data and the data feature information comprises: Obtain the effect level of the label information corresponding to the initial training sample data; The similarity between the label information and the data feature information is weighted according to the weight corresponding to the effect level to obtain the correlation between the initial training sample data and the data demand information.

6. The data acquisition method according to claim 1, characterized in that: The step of determining a plurality of target training sample data from the initial training sample data according to the data feature information and the preset label information corresponding to the initial training sample data comprises: According to the domain scene information carried by the data demand information, determining the associated storage associated with the domain scene information from the preset initial storage; The training sample data in the associated storage is determined as initial training sample data.

7. The data acquisition method according to any one of claims 1 to 6, characterized in that: The sending a data acquisition request to each of the target storages to acquire the target training sample data includes: Sending a data request to a target gateway corresponding to each of the target storages based on a reverse proxy service; Receive target training sample data returned by each target gateway in response to the data request.

8. A data acquisition device, characterized in that: include: An acquisition module is used to obtain data requirement information; The data requirement information carries data feature information; A determination module, configured to determine a number of target training sample data from the initial training sample data according to the data feature information and the label information corresponding to the preset initial training sample data; A query module is used to query a preset database to obtain target storage corresponding to each target training sample data; The request module is used to send a data acquisition request to each of the target storages to acquire the target training sample data.

9. A computer device, characterized in that: The computer device includes a processor, a memory, and a computer program stored in the memory and executable on the processor, and the processor executes the computer program to implement the steps in the data acquisition method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and the computer program is executed by a processor to implement the steps in the data acquisition method according to any one of claims 1 to 7.

Images