Model automatic evolution method, device and equipment based on reinforcement learning

Abstract

The invention relates to the technical field of computers, and discloses an automatic model evolution method, device and equipment based on reinforcement learning, and the method comprises the steps: employing a random sampling mode, extracting data from an original data set, and generating a new data set; reasoning the new data set based on a first local model to form a reasoning result set; mutating the reasoning result set to form an amplified sample set; retraining the first local model by using the amplified sample set and the training sample set to generate a second local model; if the accuracy rate of the second local model is smaller than or equal to that of the first local model, evolution is stopped, and the first local model serves as an online model.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to a method, device and equipment for automatic model evolution based on reinforcement learning. Background Art

[0002] During the model training / retraining process, the quality of the training samples is directly related to the model's reasoning effect. The training samples used for model training / retraining mainly rely on manual labeling, which requires a large amount of manual work and has different labeling levels, making it difficult to meet the model training / retraining scenarios under large-scale samples. Therefore, how to achieve model evolution and make the model "better with use" is an important issue that needs to be solved.

[0003] In current technology, reinforcement learning is used for model construction. Reinforcement learning is an online learning method that uses a "trial and error" approach. Unlike supervised learning and unsupervised learning, it does not require any data to be given in advance. Instead, it obtains corresponding learning information by receiving feedback from the environment on the action, and adjusts and optimizes the relevant parameters of the model accordingly to achieve continuous improvement of the model. This method only involves the construction process before the model goes online, and does not involve the retraining process after the model goes online. In addition, this method completely relies on external input reward and punishment information to complete the model's own training, and cannot achieve automatic evolution of the model.

[0004] Based on this, a new reinforcement learning-based automatic model evolution method is needed. Summary of the invention

[0005] The embodiments of this specification provide a method, device and equipment for automatic model evolution based on reinforcement learning, which are used to solve the following technical problems: In current technology, reinforcement learning is used to build models. Reinforcement learning is an online learning method that adopts the idea of ​​"trial and error". Unlike supervised learning and unsupervised learning, it does not require any data to be given in advance, but obtains corresponding learning information by receiving feedback from the environment on the action, and adjusts and optimizes the relevant parameters of the model accordingly to achieve continuous improvement of the model. This method only involves the construction process before the model goes online, and does not involve the retraining process after the model goes online. Moreover, this method completely relies on external input reward and punishment information to complete the training of the model itself, and cannot achieve automatic evolution of the model.

[0006] To solve the above technical problems, the embodiments of this specification are implemented as follows: The embodiment of this specification provides a model automatic evolution method based on reinforcement learning, including: Use random sampling to extract data from the original data set to generate a new data set; Performing reasoning on the new data set based on the first local model to form a reasoning result set; mutating the inference result set to form an amplified sample set; Retraining the first local model with the augmented sample set and the training sample set to generate a second local model; If the accuracy of the second local model is ≤ the accuracy of the first local model, the evolution is stopped and the first local model is used as the online model.

[0007] The embodiment of this specification also provides a model automatic evolution device based on reinforcement learning, including: The random generation module uses random sampling to extract data from the original data set to generate a new data set; An inference module, for inferring the new data set based on the first local model to form an inference result set; A mutation module, mutating the inference result set to form an amplified sample set; A retraining module, retraining the first local model with the augmented sample set and the training sample set to generate a second local model; The online module stops evolving if the accuracy of the second local model is less than or equal to the accuracy of the first local model, and the first local model is used as the online model.

[0008] The embodiment of the present specification also provides an electronic device, including: at least one processor; and, a memory communicatively connected to the at least one processor; wherein, The memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to: Use random sampling to extract data from the original data set to generate a new data set; Performing reasoning on the new data set based on the first local model to form a reasoning result set; mutating the inference result set to form an amplified sample set; Retraining the first local model with the augmented sample set and the training sample set to generate a second local model; If the accuracy of the second local model is ≤ the accuracy of the first local model, the evolution is stopped and the first local model is used as the online model.

[0009] The reinforcement learning-based model automatic evolution method provided in the embodiments of this specification adopts a random sampling method to extract data from the original data set to generate a new data set; the new data set is inferred based on the first local model to form an inference result set; the inference result set is mutated to form an amplified sample set; the first local model is retrained with the amplified sample set and the training sample set to generate a second local model; if the accuracy of the second local model is ≤ the accuracy of the first local model, the evolution is stopped. The first local model, as the online model, can form high-quality amplified samples through automatic extraction, automatic mutation and automatic evaluation of the model inference results, thereby realizing the automatic evolution capability of the model. The entire process does not require human participation and other external inputs, which significantly reduces the workload of model developers and can be effectively applied to various data analysis business fields. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] In order to more clearly illustrate the embodiments of this specification or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in this specification. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0011] Figure 1 A schematic diagram of the system architecture of a model automatic evolution method based on reinforcement learning provided in an embodiment of this specification; Figure 2 A flowchart of a model automatic evolution method based on reinforcement learning provided in an embodiment of this specification; Figure 3 A schematic diagram of the generation process of the amplification sample set provided in the embodiments of this specification; Figure 4 A framework diagram of a model automatic evolution method based on reinforcement learning provided in an embodiment of this specification; Figure 5 A schematic diagram of a model automatic evolution device based on reinforcement learning provided in an embodiment of this specification. DETAILED DESCRIPTION

[0012] In order to enable those skilled in the art to better understand the technical solutions in this specification, the technical solutions in the embodiments of this specification will be clearly and completely described below in conjunction with the drawings in the embodiments of this specification. Obviously, the described embodiments are only part of the embodiments of this application, not all of them. Based on the embodiments of this specification, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of this application.

[0013] Reinforcement learning is an online learning method that uses a "trial and error" approach. Unlike supervised learning and unsupervised learning, it does not require any data to be given in advance. Instead, it obtains corresponding learning information by receiving feedback from the environment on the action, and adjusts and optimizes the relevant parameters of the model accordingly to achieve continuous improvement of the model. According to the given conditions, reinforcement learning can be divided into model-based reinforcement learning and model-free reinforcement learning.

[0014] Therefore, the embodiments of this specification provide a new method for automatic model evolution based on reinforcement learning, which utilizes the methods of "self-extraction of reasoning results, self-mutation of reasoning results, and self-evaluation of reasoning results" to achieve the ability to discover amplified samples for model reasoning results, thereby supporting subsequent model retraining, relying only on existing internal evaluation samples and reasoning results, without the need for other external input, to achieve continuous optimization of model capabilities.

[0015] Figure 1 The following is a schematic diagram of the system architecture of a reinforcement learning-based model automatic evolution method provided in the embodiments of this specification. Figure 1 As shown, the system architecture 100 may include terminal devices 101, 102, 103, a network 104 and a server 105. The network 104 is used to provide a medium for communication links between the terminal devices 101, 102, 103 and the server 105. The network 104 may include various connection types, such as wired, wireless communication links or optical fiber cables, etc.

[0016] The terminal devices 101, 102, 103 interact with the server 105 via the network 104 to receive or send messages, etc. Various client applications may be installed on the terminal devices 101, 102, 103, such as dedicated programs such as the automatic model evolution method based on reinforcement learning.

[0017] The terminal devices 101, 102, 103 may be hardware or software. When the terminal devices 101, 102, 103 are hardware, they may be various dedicated or general electronic devices, including but not limited to smart phones, tablet computers, laptop computers, desktop computers, etc. When the terminal devices 101, 102, 103 are software, they may be installed in the electronic devices listed above. They may be implemented as multiple software or software modules (e.g., multiple software or software modules for providing distributed services), or they may be implemented as a single software or software module.

[0018] The server 105 may be a server that provides various services, such as a backend server that provides services for client applications installed on the terminal devices 101, 102, and 103. For example, the server may perform automatic model evolution based on reinforcement learning so as to display the automatic model evolution result on the terminal device servers 101, 102, and 103, and the server may also perform automatic model evolution based on reinforcement learning so as to display the automatic model evolution result on the terminal devices 101, 102, and 103.

[0019] The server 105 may be hardware or software. When the server 105 is hardware, it may be implemented as a distributed server cluster consisting of multiple servers, or as a single server. When the server 105 is software, it may be implemented as multiple software or software modules (e.g., multiple software or software modules for providing distributed services), or as a single software or software module.

[0020] Figure 2 The following is a flowchart of a method for automatic model evolution based on reinforcement learning provided in the embodiments of this specification. From the program perspective, the execution subject of the process can be a program installed on an application server or application terminal. It can be understood that the method can be executed by any device, equipment, platform, or device cluster with computing and processing capabilities. Figure 2 As shown, the model automatic evolution method includes: Step S201: extract data from the original data set by random sampling to generate a new data set.

[0021] In the embodiments of this specification, the original data set D is an unprocessed data set, which includes data that can be used for model training and data that is useless for model training. Therefore, it is necessary to go through steps such as cleaning and preprocessing to remove noise, fill missing values, convert formats, or extract new features to be suitable for model training. The original data set D is numerical data and / or data that can be converted into numerical data. Specifically, the original data set D is in key-value format, i.e., a key-value pair, where key represents an index value and value represents a value.

[0022] From the original data set D, a first preset proportion x% of data is randomly extracted to form a new data set D1. The selection of x depends on the specific business scenario. In one embodiment, x defaults to 10, and its value is adjustable, which is not specifically limited here. Similarly, the new data set D1 is numerical data and / or data that can be converted into numerical data. In this illustrative embodiment, the new data set D1 is used for the first local model M1.

[0023] Step S203: The new data set is inferred by the first local model to form an inference result set.

[0024] Continuing with the previous example, the original data set D is subjected to preprocessing and other steps to form a training data set S. The training data set S is a data set used to train the model. Similarly, the training data set S is numerical data and / or data that can be converted into numerical data.

[0025] In this embodiment, the local model M is trained using the training data set S to form a first local model M1. Of course, the first local model M1 may also be a trained model. In other words, the first local model may be a model obtained through training or a model that has been trained.

[0026] In the embodiment of the present specification, the first local model can process numerical data and / or data that can be converted into numerical data. It should be noted that in the embodiment of the present specification, the data involved are all numerical data and / or data that can be converted into numerical data, and will not be described in detail later.

[0027] It can be understood that the method provided in the embodiments of this specification is used to process numerical data and / or data that can be converted into numerical data, thereby achieving retraining of the model corresponding to the data processed numerical data and / or data that can be converted into numerical data.

[0028] The method provided in the embodiments of this specification, specifically in terms of data types, may be image data, and / or text data, and / or video data, and / or audio data, etc.

[0029] The first local model is a model that has been trained or a model obtained through training, that is, the first local model is an online model. Since the trained model may have certain differences between the data used as training samples and the real data of the model user during the training process, in order to ensure the accuracy of the data recognition results, the first local model needs to be adjusted after the model is deployed online, that is, in the use phase after the model is deployed, that is, the model needs to be automatically evolved. It can be seen that the model automatic evolution method provided in this embodiment is not applied before the model is deployed online, but is applied to the use phase after the model is deployed.

[0030] Continuing with the previous example, the first local model M1 is used to infer the new data set D1 to form an inference result set D1' with inference results.

[0031] Step S205: Inferring the new data set based on the first local model to form an inference result set.

[0032] Since the accuracy of the data quality of the inference result set D1' is not high, the model evolution cannot be performed well. Based on this, the inference result set D1' is mutated to form the inference result set D1''.

[0033] In the embodiment of this specification, the mutating of the inference result set to form an augmented sample set specifically includes: Using the inference result set as an initialization amplification sample set; Randomly selecting data from the initialization amplification sample set as a new inference result set; mutating the new reasoning result set to form a mutated reasoning result set; If the accuracy of the mutated inference result set is greater than the accuracy of the new inference result set, updating the amplified sample set with the mutated inference result set to obtain an updated amplified sample set; The updated amplification sample set is used as the initialization amplification sample set, and the operation is repeated to perform mutation to form the amplification sample set.

[0034] In the embodiment of this specification, mutating the new reasoning result set to form a mutated reasoning result set specifically includes: Determine a target record in the new inference result set from the initialization amplification sample set, use the weight of the target record and the highest value as a first candidate value, and use the weight of the target record and the second highest value as a second candidate value; According to a first preset probability, each record in the new inference result set is replaced with the first candidate value, and according to a second preset probability, each record in the new inference result set is replaced with the second candidate value to form the mutated inference result set, and the sum of the first preset probability and the second preset probability is 1.

[0035] In the embodiment of this specification, determining the target record in the new reasoning result set from the initialization expansion sample set specifically includes: From the initialization amplification sample set, several records closest to each record in the new reasoning result set are selected as target records in the new reasoning result set.

[0036] In the embodiment of the present specification, the updated amplification sample set is used as the initialization amplification sample set, and the operation is performed in a loop to mutate and form the amplification sample set, wherein the default value of the number of loop operations n is 10, and the number of loop operations n can be adjusted according to the specific business scenario and is not limited here.

[0037] The initialization amplification sample set refers to the initial state of the amplification sample set. Continuing with the previous example, the inference result set D1' is used as the initialization amplification sample set D01", and the second preset proportion y% of data is randomly selected from the initialization amplification sample set as the new inference result set E. The value range of y is 1~10, and the selection of y depends on the specific business scenario. In one embodiment, y defaults to 5, and its value is adjustable and is not specifically limited here.

[0038] For each record d in the new inference result set E, the z records with the closest distribution distance to record d are obtained from the initialized amplified sample set D01”= as the target records in the new inference result set. The value corresponding to each record is weighted according to 1 / (z+the distribution distance between the record and record d ranked from small to large), and the sum of the weights of each record is used as the sum of the weights of the target record. The value with the highest sum of weights among the z records is selected as the first candidate value v1, and the value with the second highest sum of weights is selected as the candidate value v2. The value of record d is replaced with the first candidate value v1 according to the probability of the first preset probability w, and the value of record d is replaced with the second candidate value v2 according to the second preset probability (1-w), forming the mutated inference result set E'.

[0039] In this embodiment, the value range of z is 1 to 5, and the default value is 5. The value of z can be determined according to the business scenario and its value can be adjusted. Each record d in the inference result set E is <key-value>The value range of the first preset threshold w is 0.8~1.0, and the default value is 0.9.

[0040] Continuing with the previous example, the evaluation sample set C is formed by selecting part of the data from the original data set D and annotating it. Therefore, the original data set D covers the evaluation sample set C.

[0041] The accuracy q1 of the mutated inference result set E' is obtained by evaluating the evaluation sample set C. Specifically, the accuracy q1 of the mutated inference result set E'=the number of samples with the same value from the mutated inference result set E' and the evaluation sample set C / count (the key duplicate set of the mutated inference result set E' and the evaluation sample set C).

[0042] Similarly, the accuracy q2 of the new inference result set E is obtained based on the evaluation sample set C. Specifically, the accuracy q2 of the new inference result set E = the number of samples with the same value from the new inference result set E and the evaluation sample set C / count (the key of the new inference result set E and the evaluation sample set C are repeated sets).

[0043] That is, if the accuracy q1 of the mutated inference result set E'> the accuracy q2 of the new inference result set E, the mutated inference result set is used to update the augmented sample set.

[0044] In the embodiment of the present specification, if the accuracy of the mutated inference result set is greater than the accuracy of the new inference result set, then updating the amplified sample set with the mutated inference result set to obtain an updated amplified sample set further includes: If the accuracy of the updated amplification sample set is ≤ the accuracy of the inference result set, then stop mutation, and replace the updated amplification sample set with the inference result set as the amplification sample set; If the accuracy of the updated amplification sample set is greater than the accuracy of the inference result set, and the difference between the accuracy of the updated amplification sample set and the accuracy of the inference result set is less than a second preset threshold, then stop mutation and use the updated amplification sample set as the amplification sample set; If the accuracy of the updated amplification sample set is greater than the accuracy of the inference result set, and the difference between the accuracy of the updated amplification sample set and the accuracy of the inference result set is greater than or equal to a second preset threshold, the updated amplification sample set is used as the initialization amplification sample set.

[0045] Continuing with the previous example, the accuracy q3 of the updated amplified sample set D1" is obtained based on the evaluation sample set C. Specifically, the accuracy q3 of the updated amplified sample set D1" = the number of samples with the same value from the updated amplified sample set D1" and the evaluation sample set C / count (the key duplicate set of the updated amplified sample set D1" and the evaluation sample set C).

[0046] Similarly, the accuracy q4 of the inference result set D1' is obtained based on the evaluation sample set C. Specifically, the accuracy q4 of the inference result set D1' = the number of samples with the same value from the inference result set D1' and the evaluation sample set C / count (the key duplicate set of the inference result set D1' and the evaluation sample set C).

[0047] That is, if the accuracy q3 of the updated amplification sample set D1″ is ≤ the accuracy q4 of the inference result set D1′, the mutation is stopped, and the updated amplification sample set is replaced by the inference result set as the amplification sample set.

[0048] If the accuracy q3 of the updated amplified sample set D1″ is greater than the accuracy q4 of the inference result set D1′, and q3-q4 is less than the second preset threshold p2, then the mutation is stopped and the updated amplified sample set is used as the amplified sample set; If the accuracy q3 of the updated amplified sample set D1″>the accuracy q4 of the inference result set D1′, and q3-q4≥the second preset threshold p2, stop mutation and repeat the following steps: randomly select data from the initialized amplified sample set as a new inference result set.

[0049] The selection of the second preset threshold value p2 should not be greater than the second preset ratio, that is, the second preset threshold value ≤ the second preset ratio. In this embodiment, the default value of the second preset threshold value p2 is 0.05%. The second preset threshold value p2 can be adjusted according to the specific business scenario and is not specifically limited here.

[0050] In order to further understand the generation of the amplification sample set, the generation process of the amplification sample set is described below. Figure 3 This is a schematic diagram of the generation process of the amplification sample set provided in the embodiments of this specification. Figure 3 As shown, the inference result set is used as the initialization amplification sample set; data is extracted from the initialization amplification sample set by random sampling to generate a new inference result set; based on the new inference result set, mutation is performed to form a mutated inference result set; if the accuracy q1 of the mutated inference result set is greater than the accuracy q2 of the new inference result set, the amplification sample set is updated using the mutated inference result set; the operation is repeated to form an updated amplification sample set; if the accuracy q3 of the updated amplification sample set is less than or equal to the accuracy q4 of the inference result set, the mutation is stopped, and the updated amplification sample set is replaced by the inference result set as the amplification sample set; if the accuracy q3 of the updated amplification sample set D1″ is greater than the accuracy q4 of the inference result set D1′, and q3-q4≥the second preset threshold value p2, the mutation is stopped, and the following steps are repeated: data is randomly selected from the initialization amplification sample set as a new inference result set, thereby executing the subsequent operation of "randomly selecting data from the initialization amplification sample set as a new inference result set". Step S207: retraining the first local model with the augmented sample set and the training sample set to generate a second local model.

[0051] When retraining the first local model, if the key of the augmented sample set is the same as that of the training sample set but the value is different, that is, there is a value conflict under the same key, the value of the training sample set is used as the result value.

[0052] Step S209: If the accuracy of the second local model is ≤ the accuracy of the first local model, the evolution is stopped and the first local model is used as the online model.

[0053] The accuracy of the first local model and the accuracy of the second local model are both determined based on the evaluation sample set C. The accuracy of the first local model q5 = (the number of results obtained by predicting the samples in the evaluation sample set using the first local model and the same as the evaluation result set) / the number of evaluation sample sets; the accuracy of the second local model q6 = (the number of results obtained by predicting the samples in the evaluation sample set using the second local model and the same as the evaluation result set) / the number of evaluation sample sets. If q6≤q5, the evolution is stopped and the first local model is used as the online model.

[0054] In the embodiment of this specification, the method further comprises: If the accuracy of the second local model is greater than the accuracy of the first local model, and (the accuracy of the second local model - the accuracy of the first local model) is less than the first preset threshold, the evolution is stopped and the second local model is used as the online model; If the accuracy of the second local model is greater than the accuracy of the first local model, and (the accuracy of the second local model - the accuracy of the first local model) is greater than or equal to the first preset threshold, the augmented sample set is added to the training sample set as an updated training sample set, the second local model replaces the first local model, and the following steps are repeated: data is extracted from the original data set using random sampling to generate a new data set.

[0055] Continuing with the previous example, if q6>q5, and q6-q5<the first preset threshold p1, the evolution is stopped and the second local model M2 is online; if q6>q5, and q6-q5≥the first preset threshold p1, the amplified sample set is added to the training sample set as an updated training sample set, and the second local model replaces the first local model, and the following steps are repeated: data is extracted from the original data set by random sampling to generate a new data set, thereby performing the subsequent operation of "extracting data from the original data set by random sampling to generate a new data set".

[0056] The first preset threshold value is determined according to a specific business scenario, and a default value of the first preset threshold value p1 is 0.1%.

[0057] The reinforcement learning-based model automatic evolution method provided in the embodiments of this specification is used to automatically evolve the model after the model is deployed online.

[0058] In order to further understand the automatic model evolution method based on reinforcement learning provided in the embodiments of this specification, the embodiments of this specification further provide a framework diagram of the automatic model evolution method based on reinforcement learning. Figure 4 As shown, the local model M is trained with the training sample set S to obtain the first local model M1; data D is extracted from the original data set by random sampling to generate a new data set D1; the new data set D1 is inferred by the first local model M1 to form an inference result set D1' with inference results; the inference result set D1' is mutated to form an augmented sample set D1'; the first local model M1 is retrained with the training sample set S + the augmented sample set D1' to obtain the second local model M2, and the evaluation sample set C is used to evaluate the accuracy q5 of the first local model M1 and the accuracy q6 of the second local model M2. rate q6; if q6≤q5, the evolution is stopped, the first local model M1 is online, and the first local model M1 is used as the online model; if q6>q5, and q6-q5<first preset threshold, the evolution is stopped, the second local model M2 is online, and the second local model M2 replaces the first local model M1 as the online model; if q6>q5, and q6-q5≥first preset threshold, the augmented sample set is added to the training sample set as an updated training sample set, and the second local model replaces the first local model, and the following steps are repeated: data is extracted from the original data set by random sampling to generate a new data set.

[0059] The reinforcement learning-based model automatic evolution method provided in the embodiments of this specification adopts a random sampling method to extract data from the original data set to generate a new data set; the new data set is inferred based on the first local model to form an inference result set; the inference result set is mutated to form an amplified sample set; the first local model is retrained with the amplified sample set and the training sample set to generate a second local model; if the accuracy of the second local model is ≤ the accuracy of the first local model, the evolution is stopped. The first local model, as the online model, can form high-quality amplified samples by automatically extracting, mutating and evaluating the model inference results, thereby realizing the automatic evolution capability of the model. The entire process does not require human participation and other external inputs, which significantly reduces the workload of model developers and can be effectively applied to various data analysis business fields.

[0060] The above content describes in detail a method for automatic model evolution based on reinforcement learning. Correspondingly, this specification also provides a device for automatic model evolution based on reinforcement learning, such as Figure 5 shown. Figure 5 A schematic diagram of a model automatic evolution device based on reinforcement learning provided in an embodiment of this specification, the model automatic evolution device includes: The random generation module 501 extracts data from the original data set by random sampling to generate a new data set; The reasoning module 503 performs reasoning on the new data set based on the first local model to form a reasoning result set; A mutation module 505 mutates the inference result set to form an amplified sample set; A retraining module 507, retraining the first local model with the augmented sample set and the training sample set to generate a second local model; The online module 509 stops evolving if the accuracy of the second local model is less than or equal to the accuracy of the first local model, and the first local model is used as the online model.

[0061] In the embodiment of this specification, the model automatic evolution device further includes: Update module 511, if the accuracy of the second local model is greater than the accuracy of the first local model, and (accuracy of the second local model - accuracy of the first local model) is less than a first preset threshold, then the evolution is stopped and the second local model is used as the online model; If the accuracy of the second local model is greater than the accuracy of the first local model, and (the accuracy of the second local model - the accuracy of the first local model) is greater than or equal to a first preset threshold, the inference result set is added to the training sample set as an updated training sample set, the second local model replaces the first local model, and the following steps are repeated: data is extracted from the original data set using random sampling to generate a new data set.

[0062] The embodiment of the present specification also provides an electronic device, including: at least one processor; and, a memory communicatively connected to the at least one processor; wherein, The memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to: Use random sampling to extract data from the original data set to generate a new data set; Performing reasoning on the new data set based on the first local model to form a reasoning result set; mutating the inference result set to form an amplified sample set; Retraining the first local model with the augmented sample set and the training sample set to generate a second local model; If the accuracy of the second local model is ≤ the accuracy of the first local model, the evolution is stopped and the first local model is used as the online model.

[0063] The above is a description of a specific embodiment of the specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in an order different from that in the embodiments and still achieve the desired results. In addition, the processes depicted in the drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0064] Each embodiment in this specification is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the device, electronic device, and non-volatile computer storage medium embodiments, since they are basically similar to the method embodiments, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiments.

[0065] The apparatus, electronic device, non-volatile computer storage medium and method provided in the embodiments of this specification correspond to each other, and therefore, the apparatus, electronic device and non-volatile computer storage medium also have similar beneficial technical effects as the corresponding method. Since the beneficial technical effects of the method have been described in detail above, the beneficial technical effects of the corresponding apparatus, electronic device and non-volatile computer storage medium will not be repeated here.

[0066] In the 1990s, it was very clear whether the improvement of a technology was a hardware improvement (for example, improvements to the circuit structure of diodes, transistors, switches, etc.) or a software improvement (improvement of the method flow). However, with the development of technology, many improvements in the method flow today can be regarded as direct improvements in the hardware circuit structure. Designers almost always obtain the corresponding hardware circuit structure by programming the improved method flow into the hardware circuit. Therefore, it cannot be said that the improvement of a method flow cannot be implemented with a hardware entity module. For example, a programmable logic device (PLD) (such as a field programmable gate array (FPGA)) is such an integrated circuit whose logical function is determined by the user's programming of the device. Designers can "integrate" a digital system on a PLD by programming themselves, without having to ask chip manufacturers to design and produce dedicated integrated circuit chips. Moreover, nowadays, instead of manually making integrated circuit chips, this kind of programming is mostly implemented by "logic compiler" software, which is similar to the software compiler used when developing and writing programs, and the original code before compilation must also be written in a specific programming language, which is called hardware description language (HDL). There is not only one HDL, but many kinds, such as ABEL (Advanced Boolean Expression Language), AHDL (Altera Hardware Description Language), Confluence, CUPL (Cornell University Programming Language), HDCal, JHDL (Java Hardware Description Language), Lava, Lola, MyHDL, PALASM, RHDL (Ruby Hardware Description Language), etc. The most commonly used ones are VHDL (Very-High-Speed ​​Integrated Circuit Hardware Description Language) and Verilog. Those skilled in the art should also know that it is only necessary to program the method flow slightly in the above-mentioned hardware description languages ​​and program it into the integrated circuit, and then it is easy to obtain the hardware circuit that implements the logic method flow.

[0067] The controller may be implemented in any suitable manner, for example, the controller may take the form of a microprocessor or processor and a computer-readable medium storing a computer-readable program code (e.g., software or firmware) executable by the (micro)processor, a logic gate, a switch, an application-specific integrated circuit (ASIC), a programmable logic controller, and an embedded microcontroller, examples of which include but are not limited to the following microcontrollers: ARC 625D, Atmel AT91SAM, Microchip PIC18F26K20, and Silicone Labs C8051F320, and the memory controller may also be implemented as part of the control logic of the memory. It is also known to those skilled in the art that, in addition to implementing the controller in a purely computer-readable program code manner, the controller may be implemented in the form of a logic gate, a switch, an application-specific integrated circuit, a programmable logic controller, and an embedded microcontroller by logically programming the method steps. Therefore, such a controller may be considered as a hardware component, and the devices for implementing various functions included therein may also be considered as structures within the hardware component. Or even, the devices for implementing various functions may be considered as both software modules for implementing the method and structures within the hardware component.

[0068] The systems, devices, modules or units described in the above embodiments may be implemented by computer chips or entities, or by products with certain functions. A typical implementation device is a computer. Specifically, the computer may be, for example, a personal computer, a laptop computer, a cellular phone, a camera phone, a smart phone, a personal digital assistant, a media player, a navigation device, an email device, a game console, a tablet computer, a wearable device, or a combination of any of these devices.

[0069] For the convenience of description, the above devices are described in terms of functions and are divided into various units. Of course, when implementing one or more embodiments of this specification, the functions of each unit can be implemented in the same or multiple software and / or hardware.

[0070] Those skilled in the art will appreciate that the embodiments of this specification may be provided as methods, systems, or computer program products. Therefore, the embodiments of this specification may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the embodiments of this specification may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program codes.

[0071] This specification is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of this specification. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0072] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0073] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0074] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0075] The memory may include non-permanent storage in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. The memory is an example of a computer-readable medium.

[0076] Computer readable media include permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. Information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer readable media does not include temporary computer readable media (transitory media), such as modulated data signals and carrier waves.

[0077] It should also be noted that the term "includes", "comprising" or any other variation thereof is intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of further restrictions, an element defined by the sentence "comprising a ..." does not exclude the existence of other identical elements in the process, method, commodity or device including the element.

[0078] The specification may be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. The specification may also be practiced in distributed computing environments where tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules may be located in local and remote computer storage media including storage devices.

[0079] Each embodiment in this specification is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the system embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0080] The above is only an embodiment of this specification and is not intended to limit the present application. For those skilled in the art, the present application may have various changes and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application should be included in the scope of the claims of the present application.

Claims

1. A model automatic evolution method based on reinforcement learning, characterized in that: The model automatic evolution method comprises: Use random sampling to extract data from the original data set to generate a new data set; Performing reasoning on the new data set based on the first local model to form a reasoning result set; mutating the inference result set to form an amplified sample set; Retraining the first local model with the augmented sample set and the training sample set to generate a second local model; If the accuracy of the second local model is ≤ the accuracy of the first local model, the evolution is stopped and the first local model is used as the online model.

2. The automatic model evolution method according to claim 1, characterized in that: The method further comprises: If the accuracy of the second local model is greater than the accuracy of the first local model, and (the accuracy of the second local model - the accuracy of the first local model) is less than the first preset threshold, the evolution is stopped and the second local model is used as the online model; If the accuracy of the second local model is greater than the accuracy of the first local model, and (the accuracy of the second local model - the accuracy of the first local model) ≥ the first preset threshold, the inference result set is added to the training sample set as an updated training sample set, the second local model replaces the first local model, and the steps of claim 1 are repeated: data is extracted from the original data set by random sampling to generate a new data set.

3. The automatic model evolution method according to claim 1, characterized in that: The mutating the inference result set to form an augmented sample set specifically includes: Using the inference result set as an initialization amplification sample set; Randomly selecting data from the initialization amplification sample set as a new inference result set; mutating the new reasoning result set to form a mutated reasoning result set; If the accuracy of the mutated inference result set is greater than the accuracy of the new inference result set, updating the amplified sample set with the mutated inference result set to obtain an updated amplified sample set; The updated amplification sample set is used as the initialization amplification sample set, and the operation is repeated to perform mutation to form the amplification sample set.

4. The automatic model evolution method according to claim 3, characterized in that: The mutating the new reasoning result set to form a mutated reasoning result set specifically includes: Determine a target record in the new inference result set from the initialization amplification sample set, use the weight of the target record and the highest value as a first candidate value, and use the weight of the target record and the second highest value as a second candidate value; According to a first preset probability, each record in the new inference result set is replaced with the first candidate value, and according to a second preset probability, each record in the new inference result set is replaced with the second candidate value to form the mutated inference result set, and the sum of the first preset probability and the second preset probability is 1.

5. The automatic model evolution method according to claim 4, characterized in that: The step of determining the target record in the new reasoning result set from the initialization amplification sample set specifically includes: From the initialization amplification sample set, several records closest to each record in the new reasoning result set are selected as target records in the new reasoning result set.

6. The automatic model evolution method according to claim 3, characterized in that: If the accuracy of the mutated inference result set is greater than the accuracy of the new inference result set, then the mutated inference result set is used to update the amplified sample set to obtain an updated amplified sample set, further comprising: If the accuracy of the updated amplification sample set is ≤ the accuracy of the inference result set, then stop mutation, and replace the updated amplification sample set with the inference result set as the amplification sample set; If the accuracy of the updated amplification sample set is greater than the accuracy of the inference result set, and the difference between the accuracy of the updated amplification sample set and the accuracy of the inference result set is less than a second preset threshold, then stop mutation and use the updated amplification sample set as the amplification sample set; If the accuracy of the updated amplified sample set is greater than the accuracy of the inference result set, and the difference between the accuracy of the updated amplified sample set and the accuracy of the inference result set is greater than or equal to a second preset threshold, repeat the steps of claim 3: randomly select data from the initialized amplified sample set as a new inference result set.

7. The automatic model evolution method according to claim 1, characterized in that: The first local model is used to process numerical data or data that can be converted into numerical data.

8. The automatic model evolution method according to claim 1, characterized in that: The automatic model evolution method is used to automatically evolve the model after the model is deployed online.

9. A model automatic evolution device based on reinforcement learning, characterized in that: The model automatic evolution device comprises: The random generation module uses random sampling to extract data from the original data set to generate a new data set; An inference module, for inferring the new data set based on the first local model to form an inference result set; A mutation module, mutating the inference result set to form an amplified sample set; A retraining module, retraining the first local model with the augmented sample set and the training sample set to generate a second local model; The online module stops evolving if the accuracy of the second local model is less than or equal to the accuracy of the first local model, and the first local model is used as the online model.

10. An electronic device, comprising: at least one processor; as well as, a memory communicatively connected to the at least one processor; wherein, The memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to: Use random sampling to extract data from the original data set to generate a new data set; Performing reasoning on the new data set based on the first local model to form a reasoning result set; mutating the inference result set to form an amplified sample set; Retraining the first local model with the augmented sample set and the training sample set to generate a second local model; If the accuracy of the second local model is ≤ the accuracy of the first local model, the evolution is stopped and the first local model is used as the online model.

Images