An autonomous continual learning method, apparatus, storage medium and electronic device

Abstract

The specification discloses an autonomous continuous learning method, device, storage medium and electronic equipment. In the autonomous continuous learning method provided in the specification, a target image is acquired, and an image category of the target image is determined; the target image is input into a continuous learning inference model to obtain a confidence of an inference result; the target image with a confidence not greater than a preset threshold is determined as a pending image; a similarity between the pending image and a preset standard image is determined according to the image category of the pending image; a first probability and a second probability of the pending image are determined according to the confidence, the similarity and a sample quantity of sample images in a current sample pool; when the first probability is not less than a first specified threshold, the pending image is determined as a sample image; and when the second probability is not less than a second specified threshold, the sample image is used to train the continuous learning inference model.

Description

Technical Field

[0001] This specification relates to the field of computer technology, and in particular to an autonomous continuous learning method, device, storage medium, and electronic device. Background Art

[0002] Image processing techniques combined with neural network models are now widely used in various fields. For example, in remote sensing, trained neural network models can be used to perform downstream tasks such as object detection, recognition, and semantic segmentation in remote sensing images. Currently, neural network models are mostly trained using offline learning, that is, using pre-collected remote sensing image data or publicly available datasets.

[0003] However, faced with complex and changeable actual application scenarios, unstable actual imaging quality of payload devices, and changing detection targets and tasks, the neural network model obtained by this offline learning method often has indicators such as detection accuracy that are significantly lower than the offline training results in actual applications, and the gap with actual application needs is very obvious.

[0004] Therefore, how to effectively and in real time resolve the large discrepancy between model training results and actual needs is an urgent problem to be solved. Summary of the Invention

[0005] This specification provides an autonomous continuous learning method, device, storage medium, and electronic device to at least partially solve the above-mentioned problems existing in the prior art.

[0006] This manual adopts the following technical solutions:

[0007] This manual provides an autonomous and continuous learning method, including:

[0008] Acquire a target image and determine the image category of the target image;

[0009] Inputting the target image into a pre-trained continuous learning reasoning model to obtain the confidence level of the reasoning result output by the continuous learning reasoning model;

[0010] Determine the target image whose confidence level is not greater than a preset threshold as a pending image;

[0011] Determining the similarity between the pending image and a preset standard image according to the image category of the pending image;

[0012] Determining a first probability and a second probability of the pending image based on the confidence level, the similarity level, and the number of sample images in the current sample pool, wherein the first probability represents a probability that the pending image is used as a sample image, and the second probability represents a probability that continuous learning is initiated for the pending image;

[0013] When the first probability is not less than a first specified threshold, determining the pending image as a sample image and adding it to the sample pool;

[0014] When the second probability is not less than a second specified threshold, the continuous learning inference model is trained using sample images in the sample pool.

[0015] Optionally, a standard image is pre-set, specifically including:

[0016] For each image category, determine a standard image under the image category;

[0017] Determining the similarity between the pending image and a preset standard image according to the image category of the pending image specifically includes:

[0018] Determine the similarity between a standard image under the image category of the pending image and the pending image.

[0019] Optionally, determining the first probability and the second probability of the pending image according to the confidence, the similarity, and the number of sample images in the current sample pool specifically includes:

[0020] The confidence, the similarity, and the number of sample images in the current sample pool are input into a pre-trained autonomous learning model to obtain a first probability and a second probability of the pending image output by the autonomous learning model.

[0021] Optionally, the autonomous learning model includes at least a first strategy subnet and a second strategy subnet;

[0022] Inputting the confidence, the similarity, and the number of sample images in the current sample pool into a pre-trained autonomous learning model to obtain a first probability and a second probability of the pending image output by the autonomous learning model, specifically including:

[0023] Determine the confidence level, the similarity level, and the number of sample images in the current sample pool as the sample state of the pending image, and input them into a pre-trained autonomous learning model;

[0024] Inputting the sample state into the first strategy subnet to obtain a first probability of the pending image;

[0025] The sample state is input into the second strategy subnet to obtain a second probability of the pending image.

[0026] Optionally, inputting the sample state into the second strategy subnet to obtain a second probability of the pending image specifically includes:

[0027] The sample state and the expectation of the first probability distribution are input into the second strategy subnet to obtain a second probability distribution output by the second strategy subnet.

[0028] Optionally, pre-train the autonomous learning model, specifically including:

[0029] Acquire a training image and determine the image category of the training image;

[0030] Inputting the training image into the continuous learning reasoning model to obtain the confidence level of the pre-learning reasoning result output by the continuous learning reasoning model;

[0031] Determining the training image whose confidence level is not greater than a preset threshold as a training image to be determined;

[0032] Determining the similarity between the training image and a preset standard image according to the image category of the training image;

[0033] Inputting the confidence, the similarity, and the number of training samples of the training sample image in the current training sample pool into the autonomous learning model to be trained, and obtaining a first probability to be optimized and a second probability to be optimized of the training image to be determined output by the autonomous learning model;

[0034] When the first probability to be optimized is not less than a first specified threshold, determining the undetermined training image as a training sample image and adding it to the training sample pool;

[0035] When the second probability to be optimized is not less than a second specified threshold, training the continuous learning inference model using the training sample images in the training sample pool;

[0036] Inputting the training image into the trained continuous learning reasoning model to obtain a post-learning reasoning result output by the continuous learning reasoning model;

[0037] determining a training reward according to a difference between the pre-learning reasoning result and the post-learning reasoning result;

[0038] The autonomous learning model is trained according to the training reward.

[0039] Optionally, the autonomous learning model further includes a value subnet;

[0040] Before training the autonomous learning model, the method further includes:

[0041] Inputting the confidence, the similarity, and the number of training samples of the training sample images in the current training sample pool into the value subnet to obtain the policy value output by the value subnet;

[0042] Training the autonomous learning model according to the training reward specifically includes:

[0043] The autonomous learning model is trained according to the training reward and the strategy value.

[0044] This specification provides an autonomous continuous learning device, comprising:

[0045] An acquisition module, configured to acquire a target image and determine an image category of the target image;

[0046] An inference module, configured to input the target image into a pre-trained continuous learning inference model and obtain a confidence level of an inference result output by the continuous learning inference model;

[0047] A screening module, configured to determine the target image whose confidence level is not greater than a preset threshold as a pending image;

[0048] A matching module, configured to determine the similarity between the image to be determined and a preset standard image according to the image category of the image to be determined;

[0049] a determination module, configured to determine a first probability and a second probability of the pending image based on the confidence level, the similarity level, and the number of sample images in the current sample pool, wherein the first probability represents a probability that the pending image is a sample image, and the second probability represents a probability that continuous learning is initiated for the pending image;

[0050] a sample module, configured to determine the pending image as a sample image and add the image to the sample pool when the first probability is not less than a first specified threshold;

[0051] A training module is used to train the continuous learning inference model using sample images in the sample pool when the second probability is not less than a second specified threshold.

[0052] This specification provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, it implements the above-mentioned autonomous continuous learning method.

[0053] This specification provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the above-mentioned autonomous continuous learning method when executing the program.

[0054] At least one of the above technical solutions adopted in this specification can achieve the following beneficial effects:

[0055] In the autonomous continuous learning method provided in this specification, a target image is obtained and the image category of the target image is determined; the target image is input into a continuous learning inference model to obtain the confidence of the inference result; the target image whose confidence is not greater than a preset threshold is determined as a pending image; based on the image category of the pending image, the similarity between the pending image and a preset standard image is determined; based on the confidence, similarity, and the number of sample images in the current sample pool, a first probability and a second probability of the pending image are determined, wherein the first probability represents the probability that the pending image is a sample image, and the second probability represents the probability that the pending image starts continuous learning; when the first probability is not less than a first specified threshold, the pending image is determined as a sample image and added to the sample pool; when the second probability is not less than a second specified threshold, the sample images in the sample pool are used to train the continuous learning inference model.

[0056] When the autonomous continuous learning method provided in this specification is used to train the continuous learning reasoning model, the confidence of each target image processed by the continuous learning reasoning model can be determined during the application of the continuous learning reasoning model, and the pending image can be screened out based on the confidence and the similarity between the pending image and the standard image can be determined; the first probability and the second probability can be obtained based on the confidence, the similarity, and the number of sample images in the sample pool, and it can be determined whether to use the pending image as a sample image based on the first probability, and whether to start continuous learning based on the second probability. By using this method, during the application of the continuous learning reasoning model, some target images processed by the continuous learning reasoning model can be autonomously collected and stored in the sample pool as sample images, and the timing of starting continuous training can be autonomously determined. The sample images in the sample pool can be used to conduct targeted intensive training on the continuous learning reasoning model, effectively solving the problem that the training effect obtained by pre-training is far from the needs of actual application, and enabling the continuous learning reasoning model to be continuously and autonomously optimized. BRIEF DESCRIPTION OF THE DRAWINGS

[0057] The drawings described herein are used to provide a further understanding of this specification and constitute a part of this specification. The exemplary embodiments and descriptions of this specification are used to explain this specification and do not constitute an improper limitation of this specification. In the drawings:

[0058] Figure 1 This is a flowchart of an autonomous continuous learning method in this manual;

[0059] Figure 2 This is a schematic diagram of the structure of an autonomous learning model in this manual;

[0060] Figure 3 This is a schematic diagram of an application structure of a method for realizing autonomous continuous learning by combining a continuous learning reasoning model with an autonomous learning model in this specification;

[0061] Figure 4 This is a schematic diagram of the steps of a method for implementing autonomous continuous learning by combining a continuous learning reasoning model with an autonomous learning model in this specification;

[0062] Figure 5 A schematic diagram of an autonomous continuous learning device provided in this manual;

[0063] Figure 6 The corresponding Figure 1 Schematic diagram of electronic equipment. DETAILED DESCRIPTION

[0064] To make the purpose, technical solutions, and advantages of this specification more clear, the technical solutions of this specification will be clearly and completely described below in conjunction with the specific embodiments of this specification and the corresponding drawings. Obviously, the embodiments described are only part of the embodiments of this specification, not all of the embodiments. Based on the embodiments in this specification, all other embodiments obtained by ordinary technicians in this field without making any creative efforts are within the scope of protection of this application.

[0065] The technical solutions provided by the embodiments of this specification are described in detail below with reference to the accompanying drawings.

[0066] Figure 1 The following is a flowchart of an autonomous continuous learning method in this manual, which specifically includes the following steps:

[0067] S100: Acquire a target image and determine the image category of the target image.

[0068] All steps in the autonomous continuous learning method provided in this specification can be implemented by any electronic device with computing capabilities, such as a terminal, a server, and the like.

[0069] This specification provides an autonomous continuous learning method for a neural network model. In this method, autonomous continuous learning means that after the neural network model is put into practical application, it continues to autonomously collect training samples, autonomously determines the timing of starting training, performs intensive training on the neural network model, and continuously enhances the performance of the neural network model to adapt to more complex application environments and scenarios.

[0070] Theoretically, the autonomous continuous learning method presented in this specification can be applied to any neural network model used to perform reasoning tasks. However, this specification primarily uses a continuous learning reasoning model for image processing as an example. Therefore, in this step, the target image can be first acquired and its category determined for use in subsequent steps.

[0071] The target image can be any image used to perform the inference task; the image category of the target image represents the category of the primary content contained in the target image. For example, image categories may include, but are not limited to, vehicle categories, animal categories, plant categories, architectural categories, landscape categories, or more fine-grained categories such as specific models and sub-categories. The image category can be set based on specific needs, and any number of image categories can be set, and this specification does not impose any specific restrictions on this.

[0072] S102: Inputting the target image into a pre-trained continuous learning reasoning model to obtain the confidence level of the reasoning result output by the continuous learning reasoning model.

[0073] In this step, the target image acquired in step S100 can be input into a pre-trained continuous learning inference model. This generates an inference result output by the continuous learning inference model, along with the confidence level of the inference result. As described in step S100, the purpose of this method is to autonomously and intensively train the continuous learning inference model during its application. In effect, this step represents the normal use of the continuous learning inference model.

[0074] As you can imagine, before a continuous learning reasoning model is put into formal application, it needs to be pre-trained. This pre-training of the continuous learning reasoning model can be achieved in any manner, and this specification does not impose any specific restrictions on this. The only requirement is to ensure that the pre-trained continuous learning reasoning model can properly perform the corresponding reasoning tasks.

[0075] Typically, a common continuous learning inference model can be a binary or multi-classification model. The output logic of a continuous learning inference model generally provides confidence levels for multiple alternative outcomes, and then selects the outcome with the highest confidence as the final inference result. Based on this, in this step, you can obtain the inference result and confidence level of the continuous learning inference model output.

[0076] S104: Determine the target image whose confidence level is not greater than a preset threshold as a pending image.

[0077] After obtaining the confidence of the inference result output by the continuous learning inference model of the target image in step S102, the target image can be screened in this step, and the target image with a confidence not greater than a preset threshold is determined as a pending image.

[0078] During the application of the continuous learning reasoning model, the confidence level of the inference result it outputs must be the highest confidence level among all possible results. Based on this, the higher the confidence level of the inference result, the more confident the continuous learning reasoning model is about the inference, and the stronger its ability to process the input target image. Conversely, the lower the confidence level of the inference result, the less reliable the inference result itself, and the weaker the continuous learning reasoning model's ability to process the current input target image. Based on this idea, it can be concluded that the lower the confidence level of the inference result given by the continuous learning reasoning model when processing a target image, the less adept the continuous learning reasoning model is at processing this type of target image, and the more targeted these target images are needed to strengthen the continuous learning reasoning model's training.

[0079] Therefore, when processing a target image, any target image whose confidence level in the inference result output by the continuous learning inference model is no greater than a preset threshold can be identified as a candidate image. This allows for further determination in subsequent steps regarding whether the candidate image can be used as a sample image for enhanced training of the continuous learning inference model. The preset threshold can be determined based on specific needs and is not specifically limited in this specification.

[0080] S106: Determine the similarity between the image to be determined and a preset standard image according to the image category of the image to be determined.

[0081] To better select candidate images suitable for training samples in subsequent steps, this step can determine the similarity between the candidate image and a preset standard image. Depending on the image categories previously set, there may be multiple preset standard images, one for each image category. Specifically, when determining the standard image, a standard image for each image category can be determined.

[0082] The standard image for an image category is the image that most prominently displays the characteristics of that category among all images in the image database. Alternatively, a standard image is constructed using a generative adversarial network based on the semantics of the category. In other words, the standard image is the most representative image of that category. There are many ways to determine the standard image, and this specification does not impose any specific restrictions.

[0083] When determining the similarity between a candidate image and a standard image, the similarity between the standard image and the candidate image within the same image category as the candidate image needs to be determined. Specifically, the similarity between the standard image within the same image category as the candidate image can be determined. The similarity between the standard image within the same image category as the candidate image and the candidate image can be used to indicate the degree of match between the candidate image and its own image category.

[0084] In the autonomous continuous learning method provided in this specification, similarity can be determined in a variety of ways. This specification provides a specific embodiment for reference. Specifically, similarity can be calculated using a similarity extraction network. This network is based on the backbone network of the continuous learning inference model, namely the image feature extraction network. On this basis, by fusing and calculating the similarity of different dimensions of the feature vectors of the candidate image and the standard image at different levels of shallow space and deep semantics, a similarity extraction network is constructed, which can more accurately assess the distribution difference between the candidate image and the standard image.

[0085] S108: Determine a first probability and a second probability of the pending image based on the confidence, the similarity, and the number of sample images in the current sample pool, wherein the first probability represents the probability that the pending image is a sample image, and the second probability represents the probability that the pending image starts continuous learning.

[0086] In this step, the first probability and second probability of the pending image can be determined based on the confidence level and similarity level determined in the previous steps and the number of sample images in the current sample pool. The sample pool is used to store sample images, which are used to train the continuous learning inference model during the application process of this method.

[0087] In the autonomous continuous learning method provided herein, the first probability of a pending image represents the probability that the pending image can be used as a sample image, and the second probability represents the probability that the pending image can initiate continuous learning. For any pending image, the first and second probabilities can be determined by comprehensively considering its confidence, similarity, and the number of existing samples. Initiating continuous learning for a pending image means that the continuous learning inference model autonomously begins a round of training after completing the inference task for the pending image.

[0088] There are various methods for determining the first probability and the second probability. This specification provides a specific embodiment for reference. Specifically, the confidence level, the similarity level, and the number of sample images in the current sample pool can be input into a pre-trained autonomous learning model to obtain the first probability and the second probability of the undetermined image output by the autonomous learning model.

[0089] In the autonomous continuous learning method provided in the present specification, a pre-trained autonomous learning model can be used to determine the first probability and the second probability of a pending image according to the confidence, the similarity, and the number of sample images in the current sample pool. Figure 2 A structural diagram of an autonomous learning model provided in the present specification is shown in FIG. 1. Figure 2 As shown, the autonomous learning model can at least include a first strategy subnetwork and a second strategy subnetwork. When determining the first probability and the second probability by using the autonomous learning model, the confidence, the similarity, and the number of sample images in the current sample pool can be determined as the sample state of the pending image, and the pre-trained autonomous learning model can be input. The sample state can be input into the first strategy subnetwork to obtain the first probability of the pending image. The sample state can be input into the second strategy subnetwork to obtain the second probability of the pending image.

[0090] The confidence, the similarity, and the number of sample images in the current sample pool of a pending image can be regarded as the sample state of the pending image. In the autonomous learning model, the first strategy subnetwork can be used to determine the first probability, and the second strategy subnetwork can be used to determine the second probability. Since the methods for determining the first probability and the second probability are similar, both are obtained by obtaining a distribution probability and sampling the distribution probability, the structures of the first strategy subnetwork and the second strategy subnetwork can also be similar, both including a processing layer and a sampling layer, and of course, the parameters inside the two are completely different.

[0091] It should be noted that the first strategy subnetwork can obtain the first probability distribution, that is, the probability distribution of the pending image as a sample image, only according to the sample state of the pending image, that is, the confidence, the similarity, and the number of sample images contained in the current sample pool. The second strategy subnetwork needs to obtain the probability distribution of the pending image starting continuous learning, that is, the second probability distribution, according to the sample state of the pending image and the first probability distribution. After the first probability distribution is input into the second strategy subnetwork, the second strategy subnetwork can determine the second probability distribution of the pending image according to the expected data of the first probability distribution and in combination with the sample state. The first probability distribution and the second probability distribution can be any kind of probability distribution, such as Gaussian distribution, and the present specification does not make specific limitations thereon.

[0092] Sampling the probability distribution can be understood as selecting a value from all values in the probability distribution according to the distribution of the probability. For example, in a Gaussian distribution, the value closer to the center of the distribution has a higher probability of being selected. It should be noted that in the probability distribution, the value range of all values should be between [0, 1], that is, 0% to 100%. If the value distribution in the obtained probability distribution is not in this interval, the value in the probability distribution should be first clamped to the interval [0, 1] and then sampled.

[0093] S110: When the first probability is not less than a first specified threshold, the pending image is determined as a sample image and added to the sample pool.

[0094] On the one hand, when the first probability of the pending image, that is, the probability of being a sample image, is not less than a first specified threshold, the pending image can be determined as a sample image and added to the sample pool. The first specified threshold can be set according to specific needs, such as 70%, 80%, etc., and this specification does not impose any specific restrictions on this.

[0095] S112: When the second probability is not less than a second specified threshold, the continuous learning inference model is trained using sample images in the sample pool.

[0096] On the other hand, when the second probability of the pending image, that is, the probability of starting continuous learning, is not less than the second specified threshold, a new round of training can be started, and the sample images accumulated in the sample pool can be used as training samples to train the continuous learning inference model.

[0097] When the autonomous continuous learning method provided in this specification is used to train the continuous learning reasoning model, the confidence of each target image processed by the continuous learning reasoning model can be determined during the application of the continuous learning reasoning model, and the pending image can be screened out based on the confidence and the similarity between the pending image and the standard image can be determined; the first probability and the second probability can be obtained based on the confidence, the similarity, and the number of sample images in the sample pool, and it can be determined whether to use the pending image as a sample image based on the first probability, and whether to start continuous learning based on the second probability. By using this method, during the application of the continuous learning reasoning model, some target images processed by the continuous learning reasoning model can be autonomously collected and stored in the sample pool as sample images, and the timing of starting continuous training can be autonomously determined. The sample images in the sample pool can be used to conduct targeted intensive training on the continuous learning reasoning model, effectively solving the problem that the training effect obtained by pre-training is far from the needs of actual application, and enabling the continuous learning reasoning model to be continuously and autonomously optimized.

[0098] In addition, the present description states that the autonomous learning model used in the autonomous continuous learning method provided can be trained in advance. Specifically, a training image can be obtained, and the image category of the training image can be determined; the training image can be input into the continuous learning reasoning model to obtain the confidence of the pre-learning reasoning result output by the continuous learning reasoning model; the training image whose confidence is not greater than a preset threshold is determined as a training image to be determined; according to the image category of the training image to be determined, the similarity between the training image to be determined and a preset standard image is determined; the confidence, the similarity, and the number of training samples of the training sample image in the current training sample pool are input into the autonomous learning model to be trained to obtain the autonomous The learning model outputs a first probability to be optimized and a second probability to be optimized of a training image to be determined; when the first probability to be optimized is not less than a first specified threshold, the training image to be determined is determined as a training sample image and added to the training sample pool; when the second probability to be optimized is not less than a second specified threshold, the training sample images in the training sample pool are used to train the continuous learning reasoning model; the training image is input into the trained continuous learning reasoning model to obtain a post-learning reasoning result output by the continuous learning reasoning model; a training reward is determined based on the difference between the pre-learning reasoning result and the post-learning reasoning result; and the autonomous learning model is trained based on the training reward.

[0099] Since the purpose of the autonomous learning model is to select suitable sample images so that the continuous learning inference model can achieve good training results after training with these sample images, the rewards during training of the autonomous learning model are positively correlated with the training results of the continuous learning inference model. The better the training results of the continuous learning inference model, the greater the rewards for the autonomous learning model. The training results of the continuous learning inference model can be determined by the difference between the inference results obtained by the continuous learning inference model on the same image before and after training.

[0100] Training a continuous learning reasoning model can be performed in a variety of different ways. This specification provides a specific implementation for reference. Because this method trains the continuous learning reasoning model during its application, two identical continuous learning reasoning models can be obtained by duplicating the model. While one model remains in use, the other is trained. After training is complete, the new model is used to replace the original model.

[0101] Additionally, it can be seen from the above-mentioned method of training the autonomous learning model in this specification that, in this method, the training of the autonomous learning model and the continuous learning reasoning model are carried out alternately. The autonomous learning model can screen out sample images for training the continuous learning reasoning model. In the process of training the continuous learning reasoning model using the screened sample images, the reasoning results of the continuous learning reasoning model before training are retained, and the reasoning results of the continuous learning reasoning model after training are additionally obtained, so as to train the autonomous learning model using the reasoning results of the continuous learning reasoning model before and after training. By repeating this process continuously, autonomous continuous learning of the autonomous learning model and the continuous learning reasoning model can be formed. This also shows that for the autonomous learning model and the continuous learning reasoning model used in this specification, the application and training of the two can be carried out simultaneously, that is, training can be carried out using the data generated during application.

[0102] For example, the existing image can be divided into several sub-datasets It contains a number of new and old data collected in time series. The processing tasks in the same data set can be the same or different. Among them, the new data can include image data that has not been processed or is being processed by the continuous learning reasoning model, and the old data can include all image data that has been processed by the continuous learning reasoning model. Completing the training of each sub-dataset in the autonomous continuous learning training dataset is called completing an epoch training. A complete round of autonomous learning and continuous learning training is called an episode training. In an episode training, the autonomous learning model is trained on the sub-datasets. The continuous learning model is trained by screening training sample images for continuous learning and determining when to start continuous learning. After starting continuous learning, the continuous learning model is trained using the screened training sample images. During a training episode, the continuous learning model is trained based on the decisions made by the policy subnet in the autonomous learning model. After the continuous learning model is trained, the model is used to evaluate the inference performance of the training sample images, calculate rewards, and update the parameters of the autonomous learning model based on the rewards.

[0103] According to the convergence of autonomous learning and continuous learning in the current new and old tasks, several episodes of training are performed. Each episode of training includes T time steps, and each time step corresponds to a processing sub-dataset. one of the image data in the current episode. When the autonomous learning model judges to start a continuous learning training, the inference of the autonomous learning model in the current episode is ended, and the continuous learning training is started. After the continuous learning inference model training converges, the autonomous learning model is trained. Until a number of episode training is performed, the autonomous learning network converges.

[0104] In constructing the reward function, specifically, when The u-th episode training on the sub-data set is completed, the continuous learning inference model is updated once, and after the update, the immediate reward value of the t-th time step in the u-th episode training is calculated And the cumulative reward value R (u) The average cross-entropy of the inference result of the continuous learning inference model obtained in the u-th episode training And the average cross-entropy of the inference result of the continuous learning inference model obtained in the u-1-th episode training The average cross-entropy Whether the average cross-entropy is reduced, if it is reduced, a positive reward is given, and if it is increased, a negative penalty is given. That is:

[0105]

[0106] The way of determining the cross-entropy is different for different inference tasks. Taking the task of identifying the target in the image as an example, the cross-entropy can be determined according to the following formula:

[0107]

[0108] The average cross-entropy between the inference result and the ground truth distribution of the training image input at the t-th time step by using the continuous learning inference model obtained in the u-th episode training, K is the total number of identified targets in each training image, C is the total number of categories of the current task, y x The ground truth of the identified target belonging to the category x is 0 or 1, The prediction probability of the n-th identified target belonging to the x-th category. In addition, the determination result obtained based on other sensor data or multi-sensor fusion data can be used for comparison instead of the comparison between the inference result and the ground truth distribution, as the evaluation of the inference effect of the current continuous learning inference model.

[0109] In one episode training, the cumulative reward function is:

[0110]

[0111] wherein g is a discount factor, which can be set according to specific requirements, and the value range is (0, 1).

[0112] Further, as shown in Figure 2 , the autonomous learning model can further include a value subnetwork for assisting training. Specifically, when training the autonomous learning model, the confidence, the similarity, and the number of training samples of the training sample image in the current training sample pool are input into the value subnetwork to obtain a policy value output by the value subnetwork; and the autonomous learning model is trained according to the training reward and the policy value.

[0113] Meanwhile, in the process of training, the policy subnetwork in the autonomous learning model can first copy itself to obtain a new policy subnetwork ActorNew network and an old policy subnetwork ActorOld network with the same initial state. In the process of training, the parameters of the ActorNew network are updated with the ActorOld network as a control, and the ActorNew network obtained by training is finally used as a new policy subnetwork.

[0114] Continuing the above example, in the u-th episode training, a training image I t is input into the continuous learning inference model, and the similarity of the training image and the number of training sample images in the current training sample pool are determined as the sample state S t of the training image I t . S t is input into the ActorNew network in the policy subnetwork to obtain the probability normal distribution parameters of the action, that is, the second probability distribution. According to the probability normal distribution of the action, the action A t is sampled, that is, the second probability, whether to start continuous learning training. If continuous learning training is not started, the next training image I t+1 is input, the state S t+1 corresponding to the training image data is obtained, and input into the ActorNew network to obtain the next action A t+1 . The above process is repeated until the ActorNew network outputs an action that starts continuous learning training. After the continuous learning inference model training is completed, the u-th continuous learning inference model is obtained. The training images I t , t = 1…T collected in the u-th episode training are input into the continuous learning inference model to obtain inference results, and the corresponding reward value R is calculated. u The data pair sequence [(S1, A1, r1), (S2, A2, r2), …, (S T , A T , r T)], as a batch data Train the self-learning model.

[0115] Will The state information S of each time step t Input them into the value subnet CriticNet network in sequence to get the value corresponding to each state And calculate the corresponding discount reward and gain value

[0116]

[0117]

[0118] Assign the ActorNew network parameters in the strategy subnet to the ActorOld network in the strategy network and calculate the loss function L of the ActorNew network a (θ), using the gradient descent method, back-propagating the error, updating the ActorNew network parameters θ, and repeating the process several times until the ActorNew network converges.

[0119]

[0120]

[0121] In the above formula, π new (a t |S t ) is the probability distribution of the ActorNew network output action strategy, π old (a t |S t ) is the probability distribution of the ActorOld network output action strategy, and the penalty term KL(π old |π θ ) is π old (a t |S t ) and π new (a t |S t ), other forms of penalty terms such as Clip can also be used to replace KL(π old |π θ ), λ and η θ To adjust the coefficient, after several iterations, the ActorNew network parameters are used to update the ActorOld network parameters.

[0122] On the other hand, the loss function of the value network CriticNet network is calculated Use gradient descent method to back propagate error and update CriticNet network parameters Repeat the same process several times until the CriticNet network converges:

[0123]

[0124]

[0125] For the training of episodes 1 to N, we can sequentially use the training data pairs collected in each episode to form batch data, and train ActorNew and CriticNet in sequence; or we can divide the training of episodes 1 to N into multiple threads and perform them in parallel, sampling from the training data pairs of each episode to form batch data, and train ActorNew and CriticNet. After training on the in Training can be performed on the same dataset, or batch data can be sampled from different sub-datasets to train ActorNew and CriticNet in parallel. The autonomous learning model can also be trained using other deep reinforcement learning methods such as DQN, DDPG, and A3C, and this manual does not impose specific restrictions on this.

[0126] Figure 3 This is a schematic diagram of an application structure of a method for realizing autonomous continuous learning by combining a continuous learning reasoning model with an autonomous learning model in this specification. Figure 4 This is a schematic diagram of a method for realizing autonomous continuous learning by combining a continuous learning reasoning model with an autonomous learning model in this specification. Figure 3 and Figure 4 , you can fully implement the autonomous continuous learning method provided in this manual.

[0127] The above is the autonomous continuous learning method provided in this manual. Based on the same idea, this manual also provides a corresponding autonomous continuous learning device, such as Figure 5 shown.

[0128] Figure 5 This is a schematic diagram of an autonomous continuous learning device provided in this manual, specifically including:

[0129] An acquisition module 200 is configured to acquire a target image and determine an image category of the target image;

[0130] The reasoning module 202 is configured to input the target image into a pre-trained continuous learning reasoning model to obtain the confidence level of the reasoning result output by the continuous learning reasoning model;

[0131] A screening module 204 is configured to determine the target image whose confidence level is not greater than a preset threshold as a pending image;

[0132] A matching module 206 is configured to determine the similarity between the image to be determined and a preset standard image according to the image category of the image to be determined;

[0133] a determination module 208 configured to determine a first probability and a second probability of the pending image based on the confidence level, the similarity level, and the number of sample images in the current sample pool, wherein the first probability represents a probability that the pending image is a sample image, and the second probability represents a probability that continuous learning is initiated for the pending image;

[0134] A sample module 210 is configured to determine the pending image as a sample image and add the image to the sample pool when the first probability is not less than a first specified threshold;

[0135] The training module 212 is configured to train the continuous learning inference model using sample images in the sample pool when the second probability is not less than a second specified threshold.

[0136] Specifically, the apparatus further includes a preset module 214, specifically configured to determine, for each image category, a standard image under the image category;

[0137] The matching module 206 is specifically configured to determine the similarity between a standard image in the image category of the pending image and the pending image.

[0138] Specifically, the determination module 208 is specifically used to input the confidence, the similarity, and the number of sample images in the current sample pool into a pre-trained autonomous learning model to obtain the first probability and the second probability of the pending image output by the autonomous learning model.

[0139] Specifically, the autonomous learning model includes at least a first strategy subnet and a second strategy subnet;

[0140] The determination module 208 is specifically used to determine the confidence, the similarity, and the number of sample images in the current sample pool as the sample state of the pending image, and input them into a pre-trained autonomous learning model; input the sample state into the first strategy subnet to obtain a first probability of the pending image; and input the sample state into the second strategy subnet to obtain a second probability of the pending image.

[0141] Specifically, the determining module 208 is configured to input the sample state and the expectation of the first probability distribution into the second strategy subnet to obtain a second probability distribution output by the second strategy subnet.

[0142] Specifically, the device also includes a pre-training module 216, which is specifically used to obtain training images and determine the image category of the training images; input the training images into the continuous learning reasoning model to obtain the confidence of the pre-learning reasoning results output by the continuous learning reasoning model; determine the training images whose confidence is not greater than a preset threshold as training pending images; determine the similarity between the training pending images and a preset standard image based on the image category of the training pending images; input the confidence, the similarity, and the number of training samples of the training sample images in the current training sample pool into the autonomous learning model to be trained, and obtain the training output by the autonomous learning model. A first probability to be optimized and a second probability to be optimized of a pending image; when the first probability to be optimized is not less than a first specified threshold, determining the training pending image as a training sample image and adding it to the training sample pool; when the second probability to be optimized is not less than a second specified threshold, using the training sample images in the training sample pool to train the continuous learning reasoning model; inputting the training image into the trained continuous learning reasoning model to obtain a post-learning reasoning result output by the continuous learning reasoning model; determining a training reward based on the difference between the pre-learning reasoning result and the post-learning reasoning result; and training the autonomous learning model based on the training reward.

[0143] Specifically, the autonomous learning model also includes a value subnet;

[0144] The pre-training module 216 is specifically used to input the confidence, the similarity, and the number of training samples of the training sample images in the current training sample pool into the value subnet to obtain the strategy value output by the value subnet; and train the autonomous learning model according to the training reward and the strategy value.

[0145] This specification also provides a computer-readable storage medium, which stores a computer program that can be used to execute the above Figure 1 Provides an independent and continuous learning method.

[0146] This manual also provides Figure 6 The schematic structure diagram of the electronic device shown in FIG. Figure 6As mentioned above, at the hardware level, the electronic device includes a processor, an internal bus, a network interface, a memory, an inference accelerator card, a training accelerator card, and a non-volatile memory. Of course, it may also include hardware required for other services. The processor reads the corresponding computer program from the non-volatile memory into the memory and then runs it to achieve the above Figure 1 Of course, in addition to software implementation, this specification does not exclude other implementation methods, such as logic devices or a combination of software and hardware, etc., that is, the execution subject of the following processing flow is not limited to each logic unit, but can also be hardware or logic devices.

[0147] Improvements to a technology can be clearly distinguished as either hardware improvements (for example, improvements to circuit structures such as diodes, transistors, and switches) or software improvements (improvements to process flows). However, with technological advancements, many process flow improvements today can now be considered direct improvements to hardware circuit structures. Designers almost always program the improved process flow into the hardware circuit to obtain the corresponding hardware circuit structure. Therefore, it cannot be said that a process flow improvement cannot be implemented using a hardware module. For example, a programmable logic device (PLD), such as a field programmable gate array (FPGA), is an integrated circuit whose logical function is determined by user programming. Designers can "integrate" a digital system on a PLD by programming it themselves, without having to hire a chip manufacturer to design and manufacture a dedicated integrated circuit chip. Moreover, nowadays, instead of manually manufacturing integrated circuit chips, this programming is mostly done using "logic compiler" software. This is similar to the software compiler used when developing programs. Before compilation, the original code must also be written in a specific programming language, called a hardware description language (HDL). There is not just one HDL, but many, such as ABEL (Advanced Boolean Expression Language), AHDL (Altera Hardware Description Language), Confluence, CUPL (Cornell University Programming Language), HDCal, JHDL (Java Hardware Description Language), Lava, Lola, MyHDL, PALASM, RHDL (Ruby Hardware Description Language), etc. The most commonly used ones are VHDL (Very-High-Speed ​​Integrated Circuit Hardware Description Language) and Verilog. Those skilled in the art will also understand that by simply programming the method flow in one of these hardware description languages ​​and then programming it into an integrated circuit, a hardware circuit that implements the logic method flow can be easily obtained.

[0148] The controller can be implemented in any suitable manner. For example, the controller can take the form of a microprocessor or processor and a computer-readable medium storing computer-readable program code (e.g., software or firmware) executable by the (micro)processor, logic gates, switches, application-specific integrated circuits (ASICs), programmable logic controllers, and embedded microcontrollers. Examples of controllers include, but are not limited to, the following microcontrollers: ARC 625D, Atmel AT91SAM, Microchip PIC18F26K20, and Silicone Labs C8051F320. The memory controller can also be implemented as part of the control logic of the memory. Those skilled in the art will also know that in addition to implementing the controller in a purely computer-readable program code format, the controller can be implemented in the form of logic gates, switches, application-specific integrated circuits, programmable logic controllers, and embedded microcontrollers by logically programming the method steps. Therefore, such a controller can be considered a hardware component, and the devices included therein for implementing various functions can also be considered as structures within the hardware component. Or even, the devices for implementing various functions can be considered as both software modules that implement the method and structures within the hardware component.

[0149] The systems, devices, modules, or units described in the above embodiments may be implemented by computer chips or entities, or by products having certain functions. A typical implementation device is a computer. Specifically, the computer may be, for example, a personal computer, a laptop computer, a cellular phone, a camera phone, a smartphone, a personal digital assistant, a media player, a navigation device, an email device, a game console, a tablet computer, a wearable device, or a combination of any of these devices.

[0150] For the convenience of description, the above devices are described as being divided into various units according to their functions. Of course, when implementing this specification, the functions of each unit can be implemented in the same or multiple software and / or hardware.

[0151] Those skilled in the art will appreciate that the embodiments of the present application can be provided as methods, systems, or computer program products. Therefore, the present application can adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application can adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) that contain computer-usable program code.

[0152] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the steps in the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0153] These computer program instructions may also be stored in a computer readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 a process or multiple processes and / or boxes Figure 1 The function specified in one or more boxes.

[0154] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operational steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing the instructions executed on the computer or other programmable device for implementing the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0155] In a typical configuration, a computing device includes one or more processors (CPUs), input / output interfaces, network interfaces, memory, inference accelerator cards, and training accelerator cards.

[0156] Memory may include non-permanent storage in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. Memory is an example of a computer-readable medium.

[0157] Computer-readable media includes permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. The information can be computer-readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer-readable media does not include transitory computer-readable media (transitory media), such as modulated data signals and carrier waves.

[0158] It should also be noted that the terms "comprises," "includes," or any other variations thereof are intended to encompass non-exclusive inclusion, such that a process, method, commodity, or apparatus that includes a series of elements includes not only those elements but also other elements not explicitly listed, or includes elements inherent to such process, method, commodity, or apparatus. In the absence of further limitations, an element defined by the phrase "comprises a ..." does not exclude the presence of other identical elements in the process, method, commodity, or apparatus that includes the element.

[0159] Those skilled in the art will appreciate that the embodiments of this specification may be provided as methods, systems, or computer program products. Thus, this specification may take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware. Furthermore, this specification may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0160] This specification may be described in the general context of computer-executable instructions, such as program modules, executed by a computer. Generally, program modules include routines, programs, objects, components, data structures, and the like that perform specific tasks or implement specific abstract data types. This specification may also be practiced in distributed computing environments where tasks are performed by remote processing devices connected through a communications network. In a distributed computing environment, program modules may be located in both local and remote computer storage media, including storage devices.

[0161] The various embodiments in this specification are described in a progressive manner. Similar parts between the various embodiments can be referred to in conjunction with each other. Each embodiment focuses on the differences between the other embodiments. In particular, the system embodiments are generally similar to the method embodiments, so the description is relatively simple. For relevant parts, refer to the description of the method embodiments.

[0162] The foregoing is merely an embodiment of the present invention and is not intended to limit the present invention. Various modifications and variations are possible for those skilled in the art. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention are intended to be included within the scope of the claims of this application.

Claims

1. An autonomous continuous learning method, characterized in that: include: Acquire a target image and determine the image category of the target image; Inputting the target image into a pre-trained continuous learning reasoning model to obtain the confidence level of the reasoning result output by the continuous learning reasoning model; Determine the target image whose confidence level is not greater than a preset threshold as a pending image; Determining the similarity between the pending image and a preset standard image according to the image category of the pending image; Determining a first probability and a second probability of the pending image based on the confidence level, the similarity level, and the number of sample images in the current sample pool, wherein the first probability represents a probability that the pending image is used as a sample image, and the second probability represents a probability that continuous learning is initiated for the pending image; When the first probability is not less than a first specified threshold, determining the pending image as a sample image and adding it to the sample pool; When the second probability is not less than a second specified threshold, training the continuous learning inference model using sample images in the sample pool; The method of determining the first probability and the second probability of the pending image according to the confidence level, the similarity level, and the number of sample images in the current sample pool specifically includes: Inputting the confidence level, the similarity level, and the number of sample images in the current sample pool into a pre-trained autonomous learning model to obtain a first probability and a second probability of the pending image output by the autonomous learning model; The autonomous learning model includes at least a first strategy subnet and a second strategy subnet; Inputting the confidence, the similarity, and the number of sample images in the current sample pool into a pre-trained autonomous learning model to obtain a first probability and a second probability of the pending image output by the autonomous learning model, specifically including: Determine the confidence level, the similarity level, and the number of sample images in the current sample pool as the sample state of the pending image, and input them into a pre-trained autonomous learning model; Inputting the sample state into the first strategy subnet to obtain a first probability of the pending image; The sample state is input into the second strategy subnet to obtain a second probability of the pending image.

2. The method according to claim 1, wherein Pre-set standard images, including: For each image category, determine a standard image under the image category; Determining the similarity between the pending image and a preset standard image according to the image category of the pending image specifically includes: Determine the similarity between a standard image under the image category of the pending image and the pending image.

3. The method according to claim 1, wherein Inputting the sample state into the second strategy subnet to obtain a second probability of the pending image specifically includes: The sample state and the expectation of the first probability distribution are input into the second strategy subnet to obtain a second probability distribution output by the second strategy subnet.

4. The method according to claim 1, wherein Pre-trained autonomous learning models, including: Acquire a training image and determine the image category of the training image; Inputting the training image into the continuous learning reasoning model to obtain the confidence level of the pre-learning reasoning result output by the continuous learning reasoning model; Determining the training image whose confidence level is not greater than a preset threshold as a training image to be determined; Determining the similarity between the training image and a preset standard image according to the image category of the training image; Inputting the confidence, the similarity, and the number of training samples of the training sample image in the current training sample pool into the autonomous learning model to be trained, and obtaining a first probability to be optimized and a second probability to be optimized of the training image to be determined output by the autonomous learning model; When the first probability to be optimized is not less than a first specified threshold, determining the undetermined training image as a training sample image and adding it to the training sample pool; When the second probability to be optimized is not less than a second specified threshold, training the continuous learning inference model using the training sample images in the training sample pool; Inputting the training image into the trained continuous learning reasoning model to obtain a post-learning reasoning result output by the continuous learning reasoning model; determining a training reward according to a difference between the pre-learning reasoning result and the post-learning reasoning result; The autonomous learning model is trained according to the training reward.

5. The method according to claim 4, wherein The autonomous learning model also includes a value subnet; Before training the autonomous learning model, the method further includes: Inputting the confidence, the similarity, and the number of training samples of the training sample images in the current training sample pool into the value subnet to obtain the policy value output by the value subnet; Training the autonomous learning model according to the training reward specifically includes: The autonomous learning model is trained according to the training reward and the strategy value.

6. An autonomous continuous learning device, characterized in that: include: An acquisition module, configured to acquire a target image and determine an image category of the target image; An inference module, configured to input the target image into a pre-trained continuous learning inference model and obtain a confidence level of an inference result output by the continuous learning inference model; A screening module, configured to determine the target image whose confidence level is not greater than a preset threshold as a pending image; A matching module, configured to determine the similarity between the image to be determined and a preset standard image according to the image category of the image to be determined; a determination module, configured to determine a first probability and a second probability of the pending image based on the confidence level, the similarity level, and the number of sample images in the current sample pool, wherein the first probability represents a probability that the pending image is a sample image, and the second probability represents a probability that continuous learning is initiated for the pending image; a sample module, configured to determine the pending image as a sample image and add the image to the sample pool when the first probability is not less than a first specified threshold; a training module, configured to train the continuous learning inference model using sample images in the sample pool when the second probability is not less than a second specified threshold; The determination module is specifically configured to input the confidence level, the similarity level, and the number of sample images in the current sample pool into a pre-trained autonomous learning model to obtain a first probability and a second probability of the image to be determined output by the autonomous learning model; The autonomous learning model includes at least a first strategy subnet and a second strategy subnet; The determination module is specifically used to determine the confidence, the similarity, and the number of sample images in the current sample pool as the sample state of the pending image, and input the pre-trained autonomous learning model; input the sample state into the first strategy subnet to obtain the first probability of the pending image; input the sample state into the second strategy subnet to obtain the second probability of the pending image.

7. A computer-readable storage medium, characterized in that The storage medium stores a computer program, and when the computer program is executed by a processor, the method according to any one of claims 1 to 5 is implemented.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the program, the method according to any one of claims 1 to 5 is implemented.

Images