Active learning method and device

Abstract

The present invention discloses an active learning method and device, the method comprising: obtaining labeled sample data; training the labeled sample data using a machine learning model to obtain an intermediate model; inputting test data into the intermediate model for prediction to obtain a prediction result corresponding to the test data; outputting the test data corresponding to the prediction result with fuzzy boundaries as sample data to be labeled according to a preset evaluation index; wherein the machine learning model is trained and the intermediate model is used for prediction in a multi-threaded parallel manner. The present invention provides users with a highly versatile machine learning platform. The present invention can reduce the amount of labeling of sample data for input model training, reduce the labor cost of manual labeling, and effectively improve processing efficiency. The effect of the model can be quickly iterated and optimized at the minimum cost of labeled data. Minimized consumption of processing resources is achieved.

Description

Technical Field

[0001] The present invention relates to the field of Internet communication technology, and in particular to an active learning method and device. Background Art

[0002] In recent years, with the rapid development of a series of Internet technologies such as big data, machine learning (ML), artificial intelligence (AI), and high-performance computing, all walks of life have built services based on the Internet and provided personalized services to hundreds of millions of users through mobile terminals. One of the key technologies for Internet-based services is to use artificial intelligence and big data capabilities to provide personalized services to users. For example, shopping websites recommend related products based on users' historical shopping behaviors, advertising systems recommend appropriate advertisements based on users' historical click behaviors, and financial systems identify fraudulent transactions based on historical transaction behaviors.

[0003] In order to provide users with better services, it is often necessary to accumulate a certain amount of valuable user data. In existing machine learning, a large amount of labeled data is used for training to achieve the desired effect. However, when the amount of data is huge, the data needs to be manually labeled, which consumes a lot of manpower costs and has low processing efficiency. Therefore, a more efficient solution is needed. Summary of the invention

[0004] In order to solve the problem that the existing technology is inefficient in labeling sample data in machine learning, the present invention provides an active learning method and device:

[0005] In one aspect, the present invention provides an active learning method, the method comprising:

[0006] Obtain labeled sample data;

[0007] Using a machine learning model to train the labeled sample data to obtain an intermediate model;

[0008] Input the test data into the intermediate model for prediction, and obtain the prediction result corresponding to the test data;

[0009] According to the preset evaluation index, the test data corresponding to the prediction result with fuzzy boundaries is output as the sample data to be labeled;

[0010] The machine learning model is trained in a multi-threaded parallel manner and the intermediate model is used for prediction.

[0011] Another aspect provides an active learning device, the device comprising:

[0012] Labeled data acquisition module: used to obtain labeled sample data;

[0013] Model management module: used to train the labeled sample data using a machine learning model to obtain an intermediate model; input test data into the intermediate model for prediction to obtain a prediction result corresponding to the test data; and output the test data corresponding to the prediction result with fuzzy boundaries as sample data to be labeled according to preset evaluation indicators;

[0014] The machine learning model is trained in a multi-threaded parallel manner and the intermediate model is used for prediction.

[0015] On the other hand, a server is provided, which includes a processor and a memory, wherein the memory stores at least one instruction, at least one program, a code set or an instruction set, and the at least one instruction, the at least one program, the code set or the instruction set is loaded and executed by the processor to implement the active learning method as described above.

[0016] On the other hand, a computer-readable storage medium is provided, wherein at least one instruction, at least one program, a code set or an instruction set is stored in the storage medium, and the at least one instruction, the at least one program, the code set or the instruction set is loaded and executed by a processor to implement the active learning method as described above.

[0017] The present invention provides an active learning method and device, which have the following technical effects:

[0018] The present invention provides a highly versatile machine learning platform for users. The present invention can reduce the amount of annotation of sample data for input model training, reduce the labor cost of manual annotation, and effectively improve processing efficiency. With the minimum cost of annotated data, the effect of model optimization can be quickly iterated, and the consumption of processing resources can be minimized. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] In order to more clearly illustrate the technical solutions and advantages in the embodiments of the present invention or the prior art, the drawings required for use in the embodiments or the prior art descriptions are briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0020] Figure 1 is a schematic diagram of an active learning task running environment provided by an embodiment of the present invention;

[0021] Figure 2 This is a schematic diagram of model training and model prediction provided by an embodiment of the present invention;

[0022] Figure 3 is a flow chart of an active learning method provided by an embodiment of the present invention;

[0023] Figure 4 It is a flow chart of using a machine learning model to train the labeled sample data to obtain an intermediate model provided by an embodiment of the present invention;

[0024] Figure 5 A schematic diagram of a process for inputting test data into the intermediate model for prediction to obtain a prediction result corresponding to the test data provided by an embodiment of the present invention;

[0025] Figure 6 is a block diagram of an active learning device provided by an embodiment of the present invention;

[0026] Figure 7 It is also a block diagram of a composition of an active learning device provided by an embodiment of the present invention;

[0027] Figure 8 This is a framework diagram for implementing the active learning device provided by an embodiment of the present invention;

[0028] Fig. 9 is a schematic diagram of a front-end interactive page provided by an embodiment of the present invention;

[0029] Fig.10 is a schematic diagram of long-tail corpus data provided by an embodiment of the present invention;

[0030] Fig.11 This is a comparison diagram of the training model effects of long-tail corpus data provided by an embodiment of the present invention;

[0031] Fig.12 It is a structural diagram of a server provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0032] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0033] It should be noted that the terms "including" and "having" and any variations thereof in the specification and claims of the present invention and the above-mentioned drawings are intended to cover non-exclusive inclusions. For example, a process, method, system, product or server that includes a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0034] See also Figure 1 , Figure 1 is a schematic diagram of an active learning task running environment provided by an embodiment of the present invention, such as Figure 1 As shown, the operating environment may include a client 01 and a server 02 .

[0035] Specifically, the client 01 may include physical devices such as smart phones, desktop computers, tablet computers, laptop computers, digital assistants, smart wearable devices, car-mounted devices, speakers, televisions, robots, etc., and may also include software running in physical devices, such as virtual machines, etc. In the embodiment of the present invention, the client 01 may support users to interact with devices through human language.

[0036] Specifically, in the embodiment of this specification, the server 02 may include an independently operated server, or a distributed server, or a server cluster composed of multiple servers. The server 02 may include a network communication unit, a processor, and a memory, etc. Specifically, the server 02 may provide background services for the above-mentioned client.

[0037] In actual applications, when providing services to users, when the client receives test data with fuzzy predictions from the current model, manual annotation can be used to continuously enhance the understanding of the predictions of the user's acquisition of services. In the embodiment of this specification, the server outputs each test value with fuzzy boundaries after prediction as sample data to be annotated to the client, so that the user can manually annotate these data.

[0038] In actual applications, a variety of services can be provided to users. For example, speech recognition services can be provided to achieve the effect of mapping speech data to complete text; information retrieval services can be provided to achieve the effect of marking key entities in documents (such as names, places, events, etc.); classification and filtering services can be provided to achieve the effect of analyzing data content and marking it as relevant or irrelevant. The service implementation can support simultaneous operations of any number of users based on the client.

[0039] like Figure 2 As shown, Figure 2 This is a schematic diagram of model training and model prediction provided by an embodiment of the present invention. As can be seen from the figure, after machine learning training is performed based on a certain amount of labeled sample data to obtain an intermediate model, the test data is input into the intermediate model for prediction. The intermediate model can output the test data corresponding to the prediction results with fuzzy boundaries as sample data to be labeled based on preset evaluation indicators. After the test data corresponding to the prediction results with fuzzy boundaries are manually labeled, they can be input into the model for training.

[0040] The following describes an active learning method of the present invention. Figure 3 It is a flowchart of an active learning method provided by an embodiment of the present invention. This specification provides method operation steps as described in the embodiment or flowchart, but may include more or fewer operation steps based on conventional or non-creative labor. The order of steps listed in the embodiment is only one way of executing the steps among many orders, and does not represent the only execution order. When the actual device or server product is executed, it can be executed in sequence or in parallel according to the method shown in the embodiment or the accompanying drawings (for example, in a parallel processor or multi-threaded processing environment). Specifically, Figure 3 As shown, the method may include:

[0041] S301: Obtain labeled sample data;

[0042] In the embodiments of this specification, a large amount of sample data may be collected first, and a part or even a small amount of the sample data may be annotated to generate the annotated sample data. For example, when the amount of sample data is 10,000, 10% of it may be annotated. Specifically, the annotation work may be completed offline or online. The annotated data may be stored in a database and / or a disk file, etc.

[0043] S302: Use a machine learning model to train the labeled sample data to obtain an intermediate model;

[0044] In the embodiments of this specification, the machine learning model is not limited to the neural network machine learning model. In practical applications, it may also include other machine learning models, such as a decision tree machine learning model, etc. The embodiments of this specification are not limited to the above.

[0045] In a specific embodiment, taking the training of the intermediate model by the logistic regression model with labeled sample data as an example, it may include: inputting the labeled sample data (positive sample data and negative sample data) into the logistic regression model, the output of the logistic regression model is the probability p (p is a number between 0-1) that the labeled sample data is the positive sample data, and the labels y of the positive sample data and the negative sample data are 1 and 0 respectively, the loss of the sample label y and the probability p is defined as (yp)^2, and accordingly, the error error can be obtained according to (yp)^2 during the training process; using the gradient descent method, each threshold is updated, and the logistic regression model is trained again. The modified threshold will make the error between the probability p and the sample label y output by the next model smaller, and when the error is less than a certain value, the current logistic regression model can be used as an intermediate model.

[0046] In the embodiments of this specification, the machine learning model can be trained in a multi-threaded parallel manner.

[0047] Specifically, Figure 4 As shown, the step of using a machine learning model to train the labeled sample data to obtain an intermediate model includes:

[0048] S401: Create a data import thread (Dumper thread) and a training thread (Trainer thread);

[0049] A process can have many threads, each of which executes different tasks in parallel. A thread is the smallest unit that the operating system can schedule operations on. A thread is the basic unit of independent scheduling and dispatching. A thread can be a kernel thread scheduled by the operating system kernel, such as a Win32 (32-bit environment of the Microsoft Windows operating system) thread; a user thread scheduled by a user process, such as a POSIX (Portable Operating System Interface of UNIX) thread on the Linux platform; or a mixed scheduling of kernel and user processes, such as Windows 7 (Windows 7, an operating system developed by Microsoft). Multiple threads in the same process will share all system resources in the process, such as virtual address space, file descriptors, and signal processing, etc. However, multiple threads in the same process have their own call stack, their own register context, and their own thread-local storage. The benefits of using multi-threaded programming on multi-core or multi-CPU (Central Processing Unit), or CPUs that support Hyper-threading (Hyper-Threading Technology) are obvious, that is, improving the execution throughput of the program. On a computer with a single CPU and a single core, by using multi-threading technology, the part of the process that is responsible for I / O (Input / Output) processing and human-computer interaction and is often blocked can be separated from the part that is intensively calculated and executed separately, and a special workhorse thread can be written to perform intensive calculations, thereby improving the execution efficiency of the program.

[0050] S402: Obtain current status of the plurality of machine learning models;

[0051] The corresponding thread to be assigned in the next step is determined by obtaining the current state of the machine learning model. The states of the machine learning model include the first initial state, the data import completion state, and the training completion state. A machine learning model will be in the first initial state, the data import completion state, and the training completion state in the process of training the labeled sample data to generate the intermediate model.

[0052] S403: Allocating the data import thread to the machine learning model in the first initial state, so as to import the labeled sample data into the machine learning model in the first initial state in the data import thread;

[0053] For example, the current state of machine learning model A, the current state of machine learning model B, the current state of machine learning model C, and the current state of machine learning model D are obtained. Among them, the current state of machine learning model A is the first initial state, then the data import thread is assigned to machine learning model A to import the labeled sample data into the machine learning model A in the data import thread. After importing the labeled sample data into machine learning model A, machine learning model A is in a data import completion state. Of course, the data imported into machine learning model A here may also include parameters required for model training, etc. In the embodiments of this specification, the database and / or disk files may also store machine learning models and parameters required for model training, etc. The data import thread can then pull relevant data from the database and / or disk files and import them into the machine learning model in the first initial state.

[0054] In actual applications, the data import thread includes multiple data import sub-threads, and the data import sub-threads are assigned to the machine learning model in the first initial state according to a first preset rule. For example, the current state of machine learning model A, the current state of machine learning model B, the current state of machine learning model C, and the current state of machine learning model D are obtained. Among them, the current state of machine learning model A is the first initial state, and the current state of machine learning model B is also the first initial state. According to the waiting time of machine learning model A and machine learning model B in the first initial state, the idle data import sub-thread can be preferentially assigned to the machine learning model B with a longer waiting time.

[0055] S404: Allocate the training thread to the machine learning model in the data import completion state, so as to train the machine learning model in the data import completion state in the training thread.

[0056] For example, the current state of machine learning model A, the current state of machine learning model B, the current state of machine learning model C, and the current state of machine learning model D are obtained. Among them, the current state of machine learning model C is the data import completion state, then the training thread is allocated to machine learning model C to train the machine learning model C in the training thread. After training machine learning model C, machine learning model C is in the training completion state, and the corresponding intermediate model is obtained.

[0057] In actual applications, the training thread includes multiple training sub-threads, and the training sub-threads are assigned to the machine learning model in the data import completion state according to the second preset rule. For example, the current state of machine learning model A, the current state of machine learning model B, the current state of machine learning model C, and the current state of machine learning model D are obtained. Among them, the current state of machine learning model C is the data import completion state, and the current state of machine learning model D is also the data import completion state. According to the waiting time of machine learning model C and machine learning model D in the data import completion state, the idle training sub-thread can be preferentially assigned to the machine learning model C with a longer waiting time.

[0058] In a specific embodiment, after step S402 obtains the current status of the multiple machine learning models, the data import thread is assigned to the machine learning model A to import the labeled sample data to the machine learning model A in the data import thread (corresponding to step S403) and the training thread is assigned to the machine learning model C to train the machine learning model C in the training thread (step S404) can be performed simultaneously.

[0059] Through the data import thread (data import sub-thread) and the training thread (training sub-thread), the machine learning model is processed accordingly according to the current state, so as to achieve more efficient and fine-grained management of multiple machine learning models in the process of training and generating corresponding intermediate models, thereby ensuring the rational use of processing resources.

[0060] S303: Input the test data into the intermediate model for prediction, and obtain the prediction result corresponding to the test data;

[0061] In the embodiments of this specification, the test data may come from the unlabeled part of the original sample data, may come from the newly collected sample data, etc. The test data may be stored in a database and / or a disk file.

[0062] like Figure 5 As shown, the step of inputting the test data into the intermediate model for prediction to obtain the prediction result corresponding to the test data includes:

[0063] S501: Create a prediction thread (Predictor thread);

[0064] The prediction thread here can refer to the above description of the data import thread and the training thread.

[0065] S502: Acquire the current status of the plurality of intermediate models;

[0066] The corresponding thread to be allocated in the next step is determined by obtaining the current state of the intermediate model. The states of the intermediate model include the second initial state and the prediction completion state. An intermediate model will be in the second initial state and the prediction completion state in the process of predicting the test data. Specifically, after a machine learning model is trained, the corresponding intermediate model is obtained, and the intermediate model is in the second initial state at this time.

[0067] S503: Allocate the prediction thread to the intermediate model in the second initial state, so as to use the intermediate model in the second initial state in the prediction thread to perform prediction evaluation on the test data.

[0068] For example, the current state of the intermediate model E and the current state of the intermediate model F are obtained. Among them, the current state of the intermediate model E is the second initial state, then the prediction thread is assigned to the intermediate model E, so that the intermediate model E is used in the prediction thread to predict and evaluate the test data. After the intermediate model E is used to predict the test data, the intermediate model E is in a prediction completion state. Specifically, the prediction thread can pull the test data from the database and / or disk file and input it into the intermediate model in the second initial state.

[0069] In practical applications, the prediction thread includes a plurality of prediction sub-threads, and the training sub-thread is assigned to the machine learning model in the second initial state according to a third preset rule. For example, the current state of the intermediate model E and the current state of the intermediate model F are obtained. Among them, the current state of the intermediate model E is the second initial state, and the current state of the intermediate model F is also the second initial state. According to the waiting time of the intermediate model E and the intermediate model F in the second initial state, the idle prediction sub-thread can be preferentially assigned to the intermediate model E with a longer waiting time.

[0070] In a specific embodiment, after step S402 obtains the current status of multiple machine learning models + step S502 obtains the current status of multiple intermediate models, the following threads can be performed simultaneously: 1) assigning the data import thread to the machine learning model A to import the labeled sample data to the machine learning model A in the data import thread (corresponding to step S403); 2) assigning the training thread to the machine learning model C to train the machine learning model C in the training thread (step S404); and 3) assigning the prediction thread to the intermediate model E to use the intermediate model E in the prediction thread to perform predictive evaluation on the test data (corresponding to step S503).

[0071] The machine learning model is trained in a multi-threaded parallel manner and the intermediate model is used for prediction. The machine learning model is processed accordingly according to the current state through the data import thread (data import sub-thread) and the training thread (training sub-thread), and the prediction thread (prediction sub-thread) processes the intermediate model accordingly according to the current state. More efficient and fine-grained management of multiple machine learning models in the process of training and generating corresponding intermediate models and in the process of using intermediate models to predict test data is achieved, ensuring the rational use of processing resources.

[0072] S304: Outputting the test data corresponding to the prediction result with blurred boundaries as sample data to be labeled according to the preset evaluation index;

[0073] In the embodiments of this specification, the evaluation index includes classification evaluation index and regression evaluation index, such as F1 score, accuracy, area under the receiver operating characteristic curve (AUC), logarithmic loss (Logloss) and mean square error (MSE).

[0074] In a specific embodiment, a target threshold is obtained from the evaluation index, for example, the target threshold is 50%. After the test data is input into the intermediate model, the corresponding prediction result is obtained. Then, for a certain test data a, the prediction result is calculated according to the evaluation index to obtain 60% (>50%), and the test data a can be regarded as a positive example; the prediction result of a certain test data b is calculated according to the evaluation index to obtain 30% (<50%), and the test data b can be regarded as a negative example. For the prediction result of the test data c, 52%-48% is calculated according to the evaluation index. 52%-48% is within the preset range of the target threshold. The test data c can be regarded as the test data corresponding to the prediction result with blurred boundaries, and the test data c is output as the sample data to be labeled. Of course, the target threshold can be set according to specific needs, and the target threshold includes but is not limited to 50%. Similarly, the difference range between the prediction result with blurred boundaries calculated according to the evaluation index and the target threshold can also be set according to specific needs.

[0075] In practical applications, when applied to semantic recognition, such as in smart speakers and other products, because long-tail corpus data is small, many long-tail corpus data cannot be fully trained and cannot provide sufficient intent information. Fig.10 As shown in , it can be used as the labeled sample data for input into the machine learning model, or as the test data corresponding to the prediction result of the fuzzy boundary of the sample data to be labeled. Fig.11As shown, through the active learning method of the embodiment of this specification, for long-tail corpus data with fewer pointing intents, only a small number of samples with blurred boundaries need to be annotated, which can effectively improve the model's intent recognition ability and solve the problem of insufficient corpus data. The model trained on the active learning platform can greatly improve the accuracy of long-tail intent recognition (from 1% to 96%), and it takes less time (from 10 milliseconds to 1 millisecond).

[0076] It can be seen from the technical solutions provided in the above embodiments of this specification that the embodiments of this specification provide users with a highly versatile machine learning platform, perform machine learning training on labeled sample data to obtain an intermediate model, and then input test data into the intermediate model. The test data corresponding to the prediction results with fuzzy boundaries are output as sample data to be labeled through evaluation indicators, which can reduce the amount of labeling of sample data used for training the input model, reduce the labor cost of manual labeling, and effectively improve processing efficiency. The evaluation indicators are used to select test data with fuzzy model predictions for manual labeling, and the effect of the model is quickly iterated and optimized at the minimum cost of labeled data. The machine learning model is trained and the intermediate model is used for prediction in a multi-threaded parallel manner to ensure the minimum consumption of processing resources.

[0077] The embodiment of the present invention also provides an active learning device, such as Figure 6 As shown, the device comprises:

[0078] The labeled data acquisition module 61 is used to acquire labeled sample data;

[0079] Model management module 62: used to train the labeled sample data using a machine learning model to obtain an intermediate model; input test data into the intermediate model for prediction to obtain a prediction result corresponding to the test data; and output the test data corresponding to the prediction result with fuzzy boundaries as sample data to be labeled according to a preset evaluation index;

[0080] Among them, the active learning device uses a multi-threaded parallel approach to train the machine learning model and uses the intermediate model for prediction.

[0081] In the embodiments of this specification, Figure 7As shown, the model management module 62 includes: a state recording unit 621: used to record the current states of the multiple machine learning models and the current states of the multiple intermediate models. A data import thread unit 622: used to create a data import thread; obtain the current states of the multiple machine learning models from the state recording unit; and assign the data import thread to the machine learning model in the first initial state, so as to import the labeled sample data to the machine learning model in the first initial state in the data import thread. A training thread unit 623: used to create a training thread; obtain the current states of the multiple machine learning models from the state recording unit; and assign the training thread to the machine learning model in the data import completion state, so as to train the machine learning model in the data import completion state in the training thread. A prediction thread unit 624: used to create a prediction thread; obtain the current states of the multiple intermediate models from the state recording unit; and assign the prediction thread to the intermediate model in the second initial state, so as to use the intermediate model in the second initial state in the prediction thread to predict and evaluate the test data.

[0082] Specifically, the model management module also includes: a state writing unit: used to scan multiple machine learning models and multiple intermediate models, and write the current states of multiple machine learning models and multiple intermediate models to the state recording unit. When the state writing unit modifies the current state of the machine learning model in the state recording unit, the data import thread unit and the training thread unit are blocked from obtaining the current state of the corresponding machine learning model from the state writing unit; when the state writing unit modifies the current state of the intermediate model in the state recording unit, the prediction thread unit is blocked from obtaining the current state of the corresponding intermediate model from the state writing unit. After completing the processing of the corresponding model, the data import thread unit 622, the training thread unit 623 and the prediction thread unit 624 generate the corresponding tag file, and the state writing unit updates the current state of the model in the state recording unit by reading the tag file.

[0083] In a specific embodiment, the labeled data acquisition module 61 includes an interface interaction unit, which can be a front-end interactive page (such as Figure 8 , 9 The interface interaction unit is used to receive the labeled sample data uploaded by the user and / or respond to the user's annotation selection on the interactive interface for the sample data to be labeled (such as Fig. 9As shown), the interface interaction unit is used to receive the corresponding algorithm of the machine learning model (which may include the parameters required for model training) uploaded by the user and / or respond to the user's selection operation on the machine learning model (which may include the parameters required for model training) on ​​the interactive interface. The interface interaction unit is used to display the model training and model prediction process to the user based on the data stored in the storage module (intermediate database). Fig. 9 As shown in the figure, the web interface interacts with the user. The front-end interactive page displays the labeling status of the sample data and provides an interface for the user to label the sample data as positive sample data or negative sample data. The front-end interactive page provides a start button for "start training" and a button for viewing the "training status". The front-end interactive page also enables the user to select evaluation indicators and view the prediction results.

[0084] like Figure 8 As shown, the active learning device also includes a storage module (intermediate database), which is used to synchronously store the data acquired by the labeled data acquisition module and the data related to the model management module. The storage module enables the user to see the model training and prediction process in real time through the front-end interactive interface, etc., to achieve timely feedback to the user. The storage module saves the labeled sample data (positive sample data and negative sample data), the storage module saves the data related to the model training and model prediction process (such as the state of the machine learning model, the state of the intermediate model, the parameters and intermediate data related to the machine learning model, the parameters and intermediate data related to the intermediate model, etc.), and the storage module saves the prediction results. The saved data includes core data and intermediate data, which can be used for subsequent viewing and inspection. Of course, some intermediate data can also be cleaned up at preset times.

[0085] like Figure 8As shown, the static global model manager (GloablModelManager) is used to manage the status of all current models (including machine learning models and intermediate models). The status of the model is converted according to the preset order according to the processing of the thread (first initial state -> data import completion state -> training completion state (second initial state) -> prediction completion state). The static global model manager (GloablModelManager) is equivalent to the state recording unit 621. The scanning (Scanner) thread is equivalent to the state writing unit. The evaluation indicators and the test data corresponding to the prediction results with blurred boundaries are written to the storage module through the prediction thread unit 624 for viewing and manual annotation on the front-end interactive page. The user can view the evaluation indicators and prediction results of the current model on the front-end interactive interface. If it meets expectations, the further processing of the model can be terminated and the current intermediate model can be put online; if it does not meet expectations, the sample data to be labeled output by the prediction thread unit is manually labeled, and then the training and prediction of the model are restarted.

[0086] In actual applications, the active learning platform executes multiple tasks (different processing of multiple models) in parallel through multiple threads. For example, when multiple models include: Model A, Model B, Model C, Model D and Model E, the Scanner thread scans and finds out that: the current Model A is in the training completion state, Model B is in the data import completion state, and Models C, D and E are in the prediction completion state. Then the Predictor thread (for example, it has just finished processing Model D and is idle at this time) reads the model status through GloablModelManager, and the Predictor thread goes to process Model A for the next step; the Trainer thread (for example, it has just finished processing Model A and is idle at this time) reads the model status through GloablModelManager, and the Trainer thread goes to process Model B for the next step. When Model B passes through the Trainer thread, its status at this time is the training completion state. The Scanner thread changes the Model B from the data import completion state to the training completion state in GloablModelManager. During the modification process, other threads cannot read the status of Model B.

[0087] It should be noted that the device in the device embodiment and the method embodiment are based on the same inventive concept.

[0088] An embodiment of the present invention provides a server, which includes a processor and a memory, in which at least one instruction, at least one program, a code set or an instruction set is stored, and the at least one instruction, the at least one program, the code set or the instruction set is loaded and executed by the processor to implement the active learning method provided in the above method embodiment.

[0089] The memory can be used to store software programs and modules. The processor executes various functional applications and data processing by running the software programs and modules stored in the memory. The memory can mainly include a program storage area and a data storage area, wherein the program storage area can store an operating system, application programs required for functions, etc.; the data storage area can store data created according to the use of the device, etc. In addition, the memory can include a high-speed random access memory and can also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other volatile solid-state storage devices. Accordingly, the memory can also include a memory controller to provide the processor with access to the memory.

[0090] The embodiment of the present invention also provides a schematic diagram of the structure of a server, see Fig.12 , the server 1200 is used to implement the active learning method provided in the above embodiment. Specifically, the server structure may include the above active learning device. The server 1200 may have relatively large differences due to different configurations or performances, and may include one or more central processing units (CPU) 1210 (for example, one or more processors) and a memory 1230, and one or more storage media 1220 (for example, one or more mass storage devices) storing application programs 1223 or data 1222. Among them, the memory 1230 and the storage medium 1220 may be short-term storage or permanent storage. The program stored in the storage medium 1220 may include one or more modules, each of which may include a series of instruction operations in the server. Furthermore, the central processing unit 1210 may be configured to communicate with the storage medium 1220 to execute a series of instruction operations in the storage medium 1220 on the server 1200. The server 1200 may also include one or more power supplies 1260, one or more wired or wireless network interfaces 1250, one or more input and output interfaces 1240, and / or one or more operating systems 1221, such as Windows ServerTM, Mac OS XTM, UnixTM, LinuxTM, FreeBSDTM, etc.

[0091] An embodiment of the present invention also provides a storage medium, which can be set in a server to store at least one instruction, at least one program, code set or instruction set related to an active learning method in a method embodiment. The at least one instruction, the at least one program, the code set or instruction set is loaded and executed by the processor to implement the active learning method provided by the above method embodiment.

[0092] Optionally, in this embodiment, the storage medium may be located in at least one of the multiple network servers of the computer network. Optionally, in this embodiment, the storage medium may include, but is not limited to, various media that can store program codes, such as a USB flash drive, a read-only memory (ROM), a random access memory (RAM), a mobile hard disk, a magnetic disk, or an optical disk.

[0093] It should be noted that the sequence of the embodiments of the present invention described above is for description only and does not represent the advantages and disadvantages of the embodiments. The above describes specific embodiments of this specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in an order different from that in the embodiments and still achieve the desired results. In addition, the processes depicted in the drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0094] Each embodiment in this specification is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the device and server embodiments, since they are basically similar to the method embodiments, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiments.

[0095] A person skilled in the art will understand that all or part of the steps to implement the above embodiments may be accomplished by hardware or by instructing related hardware through a program, and the program may be stored in a computer-readable storage medium, and the above-mentioned storage medium may be a read-only memory, a disk or an optical disk, etc.

[0096] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. An active learning method, characterized in that: Applied to an active learning platform, the active learning platform trains multiple machine learning models in a multi-threaded parallel manner, and uses intermediate models corresponding to each of the multiple machine learning models to perform predictions, the method comprising: Obtaining a current state of each of the plurality of machine learning models; Allocating a data import thread to the machine learning model in the first initial state, so as to import a plurality of labeled sample data into the machine learning model in the first initial state in the data import thread; Allocating a training thread to the machine learning model in the data import completion state, so as to train the machine learning model in the data import completion state in the training thread, and obtaining an intermediate model corresponding to the machine learning model in the data import completion state; Allocating a prediction thread to the intermediate model corresponding to the machine learning model in the training completed state, so as to use the intermediate model corresponding to the machine learning model in the training completed state in the prediction thread to predict the test data, obtain the prediction result corresponding to the test data, and output the test data corresponding to the prediction result with fuzzy boundaries as the sample data to be labeled according to the preset evaluation index; The data import thread includes a plurality of data import sub-threads, and the step of assigning a data import thread to the machine learning model in the first initial state includes: assigning the corresponding data import sub-thread to the machine learning model in the first initial state according to a first preset rule; The training thread includes a plurality of training sub-threads, and the allocating a training thread to the machine learning model in a data import completion state includes: allocating the corresponding training sub-thread to the machine learning model in a data import completion state according to a second preset rule; The prediction thread includes multiple prediction sub-threads, and the prediction thread is assigned to the intermediate model corresponding to the machine learning model in the training completion state, including: according to a third preset rule, the corresponding prediction sub-thread is assigned to the intermediate model corresponding to the machine learning model in the training completion state.

2. The method according to claim 1, characterized in that The method further comprises: The data import thread, the training thread, and the prediction thread are created.

3. The method according to claim 1, characterized in that The preset evaluation indicators include F1 score, accuracy, mean square error, area under the receiver operating characteristic curve and logarithmic loss.

4. An active learning device, characterized in that: The device is configured on an active learning platform, the active learning platform uses a multi-threaded parallel method to train multiple machine learning models, and uses the intermediate models corresponding to the multiple machine learning models to perform predictions, and the device includes: A state acquisition module: used to obtain the current state of each of the multiple machine learning models; A data import module: used to assign a data import thread to the machine learning model in the first initial state, so as to import a plurality of labeled sample data into the machine learning model in the first initial state in the data import thread; Training module: used to allocate a training thread to the machine learning model in the data import completion state, so as to train the machine learning model in the data import completion state in the training thread to obtain an intermediate model corresponding to the machine learning model in the data import completion state; Prediction module: used to allocate a prediction thread to the intermediate model corresponding to the machine learning model in the training completed state, so as to use the intermediate model corresponding to the machine learning model in the training completed state in the prediction thread to predict the test data, obtain the prediction result corresponding to the test data, and output the test data corresponding to the prediction result with blurred boundaries as the sample data to be labeled according to the preset evaluation index; The data import thread includes a plurality of data import sub-threads, and the step of assigning a data import thread to the machine learning model in the first initial state includes: assigning the corresponding data import sub-thread to the machine learning model in the first initial state according to a first preset rule; The training thread includes a plurality of training sub-threads, and the allocating a training thread to the machine learning model in a data import completion state includes: allocating the corresponding training sub-thread to the machine learning model in a data import completion state according to a second preset rule; The prediction thread includes multiple prediction sub-threads, and the prediction thread is assigned to the intermediate model corresponding to the machine learning model in the training completion state, including: according to a third preset rule, the corresponding prediction sub-thread is assigned to the intermediate model corresponding to the machine learning model in the training completion state.

5. The device according to claim 4, characterized in that The device also includes: Thread creation module: used to create the data import thread, the training thread and the prediction thread.

6. The device according to claim 4, characterized in that The preset evaluation indicators include F1 score, accuracy, mean square error, area under the receiver operating characteristic curve and logarithmic loss.

7. The device according to claim 4, characterized in that The device also includes a state recording module; the state recording module is used to record the current state of each of the multiple machine learning models.

8. The device according to claim 7, characterized in that The device also includes: A state writing module: used for scanning the multiple machine learning models and writing the current state of the machine learning model to the state recording module; Among them, when the state writing module modifies the current state of the machine learning model in the state recording module, the state acquisition module is blocked from acquiring the current state of the machine learning model from the state recording module.

9. The device according to claim 4, characterized in that The device further comprises a storage module, which is used to synchronously store the data acquired by the labeled data acquisition module and the data related to the training module and the prediction module; The labeled data acquisition module includes an interface interaction unit, which is used to receive the labeled sample data uploaded by the user and / or respond to the user's selection of the label of the sample data to be labeled on the interactive interface, and the interface interaction unit is used to receive the corresponding algorithm of the machine learning model uploaded by the user and / or respond to the user's selection operation on the machine learning model on the interactive interface; The interface interaction unit is used to display the model training and model prediction process to the user based on the data stored in the storage module.

10. A server, characterized in that: The server includes a processor and a memory, wherein the memory stores at least one instruction, at least one program, a code set or an instruction set, and the at least one instruction, at least one program, a code set or an instruction set is loaded and executed by the processor to implement the active learning method as described in any one of claims 1 to 3.

11. A computer-readable storage medium, characterized in that: The storage medium stores at least one instruction, at least one program, a code set or an instruction set, and the at least one instruction, at least one program, a code set or an instruction set is loaded and executed by a processor to implement the active learning method as described in any one of claims 1 to 3.

Images