An artificial intelligence method, system, and storage medium for predicting from a sequence of images

Abstract

Embodiments of the present disclosure provide artificial intelligence methods and systems for prediction from image sequences. The method can include receiving a sequence of images taken at different time points. The method can also include applying a stabilizing model to process the sequence of images for prediction. A working model is trained with the stabilizing model and a plastic model. The training enforces consistency between the working model, the stabilizing model, and the plastic model. The working model is trained using a loss function that includes a cross-entropy loss on a joint of a training batch and memory samples and a consistency loss on the memory samples.

Description

Technical Field

[0001] This disclosure relates to methods and systems for making predictions from image sequences using a trained model. It also relates to methods and systems for training such prediction models. More specifically, this disclosure relates to an artificial intelligence inference model trained using a stable model and a plastic model for making predictions. This model can be applied to computer vision applications. Furthermore, this disclosure relates to continuously acquiring and consolidating knowledge from non-stationary data streams. Background Technology

[0002] Dynamic image processing technology is widely used in applications such as autonomous driving, surveillance, and medical imaging. Dynamic image data essentially includes sequences of images acquired at different points in time when capturing a dynamic environment. Machine learning methods such as deep neural networks (DNNs) have been developed in the field of computer vision to process images, such as still images, and make intelligent predictions based on them. However, most DNN methods do not fully utilize the knowledge gained by processing previous image frames, leading to the use of continuous learning methods.

[0003] Humans excel at continuously learning from and accumulating knowledge in constantly changing environments, which remains a challenge for DNNs. Continuous learning (CL) refers to the ability of a learning agent to continuously interact with dynamic environments and process information flows to acquire new knowledge while consolidating and retaining previously acquired knowledge.

[0004] A major challenge in implementing continuous learning in DNNs is that continuously acquiring incremental available information from non-stationary data distributions often leads to catastrophic forgetting or interference, causing the model's performance on previously learned tasks to plummet as it learns new tasks. Continuous learning methods aim to address the catastrophic forgetting problem in DNNs and achieve effective continuous learning.

[0005] Several methods have been proposed to address the catastrophic forgetting problem in computational calculus (CL). These methods can be broadly categorized into regularization-based methods that penalize changes in network weights, network expansion-based methods that specialize a different set of network parameters for different tasks, and rehearsal-based methods that maintain a memory buffer and replay samples from previous tasks. Among these, rehearsal-based methods have proven to be more effective on challenging CL tasks. In particular, current experience replay methods, Dark Experience Replay (DER), preserve the network response throughout the entire optimization trajectory and add a consistency loss to Experience Replay (ER). However, the optimal method for replaying memory samples and constraining model updates to effectively accumulate knowledge remains an open question. Summary of the Invention

[0006] To address these and other issues with existing DNN methods, this disclosure provides improved methods and systems for training and using artificial intelligence inference models that can leverage the interaction between fast instance-based learning and slow structured learning.

[0007] A novel method and system for complementary learning system-based experience replay (CLS-ER) are disclosed.

[0008] In one aspect, embodiments of this disclosure provide an artificial intelligence method for prediction from image sequences. The method may include receiving image sequences acquired at different time points. The method may also include applying a stable model to process the image sequences for prediction. A working model is trained together with the stable model and the plastic model. This training strengthens the consistency among the working model, the stable model, and the plastic model. The working model is trained using a loss function that includes a joint cross-entropy loss of training batches and memory samples, and a consistency loss on the memory samples.

[0009] In another aspect, embodiments of this disclosure provide an artificial intelligence system for making predictions from image sequences acquired by an image acquisition device at different time points. The system may include a storage device for storing a stable model, and a working model trained together with the stable and plastic models. This training enhances the consistency among the working, stable, and plastic models. The working model is trained using a loss function that includes a joint cross-entropy loss of the training batch and memory samples, and a consistency loss on the memory samples. The system may also include a processor for applying the stable model to process the image sequences for prediction.

[0010] In another aspect, embodiments of this disclosure provide a method for training an artificial intelligence inference model. The method may include receiving training batches from a data stream and receiving memory samples from an episodic memory bank. The method may further include updating the working model based on a loss function that reinforces at least one of the working model and the stable model and plastic model for consistency on the memory samples. This loss function includes a joint cross-entropy loss of the training batches and memory samples and a consistency loss on the memory samples. The method may further include updating the stable model and the plastic model based on the working model. The method may further include determining that the updated working model, the updated stable model, and the updated plastic model satisfy training conditions. The method may further include providing the stable model as the artificial intelligence inference model.

[0011] On the other hand, embodiments of this disclosure provide a non-transitory computer-readable storage medium storing instructions that, when executed by a processor, are used to implement an artificial intelligence method for predicting from image sequences or a method for training an artificial intelligence inference model as described above.

[0012] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and not intended to limit the claimed invention. Attached Figure Description

[0013] The accompanying drawings, which are incorporated herein and form part of the specification, illustrate embodiments of the present disclosure and, together with the specification, further serve to explain the principles of the present disclosure and enable those skilled in the art to complete and use the present disclosure.

[0014] Figure 1 A schematic diagram of an exemplary system for supplementary learning according to some embodiments of the present disclosure is shown.

[0015] Figure 2 Different models for supplementary learning according to some embodiments of this disclosure are shown.

[0016] Figure 3 A schematic diagram of an exemplary image analysis system according to some embodiments of the present disclosure is shown.

[0017] Figure 4 A schematic diagram of an exemplary image processing apparatus according to some embodiments of the present disclosure is shown.

[0018] Figure 5 A flowchart illustrating an exemplary artificial intelligence method for making predictions from image sequences according to some embodiments of the present disclosure is shown.

[0019] Figure 6A A flowchart is shown of an exemplary method for training a CLS-ER-based artificial intelligence inference model according to some embodiments of the present disclosure.

[0020] Figure 6B Pseudocode is shown for an exemplary CLS-ER-based method for training a supplementary inference model according to some embodiments of this disclosure.

[0021] Figure 7 An exemplary data stream for training and evaluating a CLS-ER model according to some embodiments of this disclosure is shown.

[0022] Embodiments of this disclosure will be described with reference to the accompanying drawings. Detailed Implementation

[0023] Reference will now be made in detail to exemplary embodiments, examples of which are shown in the accompanying drawings. Where possible, the same reference numerals will be used throughout the drawings to denote the same or similar parts.

[0024] While specific configurations and arrangements have been discussed, it should be understood that this is for illustrative purposes only. Those skilled in the art will recognize that other configurations and arrangements can be used without departing from the spirit and scope of this disclosure. It will be apparent to those skilled in the art that this disclosure can also be used in a variety of other applications.

[0025] It should be noted that references to "an embodiment," "an example embodiment," "some embodiments," "certain embodiments," etc., in the specification indicate that the described embodiment may include specific features, structures, or characteristics, but not every embodiment must include such specific features, structures, or characteristics. Furthermore, these phrases do not necessarily refer to the same embodiment. Moreover, when a specific feature, structure, or characteristic is described in connection with an embodiment, whether explicitly described or not, implementing such a feature, structure, or characteristic in conjunction with other embodiments will be within the knowledge of those skilled in the art.

[0026] Generally, terms can be understood, at least in part, from their use in context. For example, the term "one or more," as used herein, depends at least in part on the context and can be used to describe any feature, structure, or characteristic in a singular sense, or to describe a combination of features, structures, or characteristics in a plural sense. Similarly, terms such as "a," "one," or "this" can also be understood to express either a singular or a plural usage, at least in part on the context. Furthermore, the term "based on" can be understood to not necessarily be intended to express a set of exclusive factors, but may allow for the presence of additional factors that are not necessarily explicitly described, again depending at least in part on the context.

[0027] As mentioned above, continuous learning presents challenges for DNNs. The goal of this disclosure is to gain insights into how well the human brain excels at continuous learning and to simulate this process to achieve effective continuous learning in DNNs. The embodiments aim to reduce forgetting of previous tasks, acquire new knowledge, and integrate it with previously learned knowledge so that the model performs well on all tasks seen to date, including both recent and past samples.

[0028] Effective lifelong learning in the human brain is achieved through a set of neurophysiological processing principles and multiple memory systems. Notably, the Complementary Learning Systems (CLS) theory explains how the interaction between rapid instance-based learning and slow, structured learning is crucial for accumulating and retaining knowledge. In contrast, existing DNNs lack any such mechanisms to modulate synaptic plasticity and stability.

[0029] To this end, some embodiments of this disclosure provide a novel ER method, CLS-ER, for brain-based supplemental learning systems. This method maintains two exponentially weighted average models. In some embodiments, these models include a plastic model and a stable model, which are updated at different frequencies to simulate fast and slow adaptation of information. The exponentially weighted average is a first-order infinite impulse response filter that applies exponentially decreasing weighting factors. The weight of each older data point decreases exponentially and never reaches zero. Therefore, newer data is favored, but older data always influences the model. The use of the exponentially weighted average will be further described below.

[0030] In some embodiments, the working model receives feedback from both the plasticity and stability models, which strengthens the consistency of the model's predictions for memorized samples. Thus, the working model effectively maintains a balancing act between stability and plasticity. CLS-ER does not utilize task boundaries or make any assumptions about the data distribution, making it generalizable and suitable for "general continuous learning."

[0031] The ability to learn continuously in a constantly changing environment is a hallmark of intelligence. In the human brain, the capacity to acquire, refine, and transfer knowledge over time is mediated by a rich set of neurophysiological processing principles. A central theme in neuroscience is that intelligent behavior relies on multiple memory systems. In particular, the complementary learning systems theory posits that the hippocampus exhibits short-term adaptation and rapid learning of episodic information, followed by gradual consolidation in the neocortex for slower learning of structured information. The interaction between hippocampal and neocortical functions is crucial for simultaneously learning the details of effective generalized representations and instance-based episodic memory.

[0032] The disclosed CLS-ER-based learning method attempts to simulate the slow and fast adaptation of the human brain to information in DNNs and has a mechanism to incorporate them into working memory to achieve better CL performance in DNNs.

[0033] The brain performs two complementary tasks crucial for effective learning: cross-experience generalization and retention of memories of episodic events. The Complementary Learning Systems (CLS) approach provides a well-established theory of how the brain retrieves the general statistical structure of experiences, with the aim of extending this to new situations and details in episodic memory. The interaction between episodic memory (specific experiences) and semantic memory (general structured knowledge) provides key insights into the mechanisms employed by the brain to effectively consolidate knowledge.

[0034] CLS-ER is a continuous learning approach based on the brain's complementary learning system, which can be scaled to current computer vision datasets and achieves improved performance on standard benchmarks and more realistic general continuous learning settings. Therefore, CLS-ER is useful and performs well in current use cases and shows promise for scalability and generality.

[0035] The disclosed CLS-ER-based learning method simulates the fast and slow adaptation of information by maintaining two additional exponentially weighted average models updated at different frequencies. In some embodiments, these two models include a stable model and a plastic model that complement the working model. The stable model and the plastic model are updated more frequently than the stable model and maintain long-term and short-term semantic memories (learned representations) of experiential events (training samples), respectively. Both models interact with a memory buffer (episodic memory), which can efficiently replay not only the memory samples but also the associated neural activity (activation).

[0036] In some embodiments, the goal of incorporating fast and slow learning memories into the working model is achieved by adding a consistency term that encourages the working model to match its logits (model pre-softmax predictions) on the memory samples with the plastic model (recent samples) and the stable model (older samples). This can be viewed as a replay of the information encoded in semantic memory. Here, logits are the outputs of a linear layer without any activation function.

[0037] The softmax function, also known as soft argmax or normalized exponential function, is a generalization of the multidimensional logistic function. It is commonly used as the final activation function in neural networks to normalize the network's output to a probability distribution predicting the output class.

[0038] In some embodiments, the softmax function (denoted as σ(z)) takes a vector z of K real numbers as input and normalizes it to a probability distribution consisting of K probabilities proportional to the exponents of the input numbers. In short, the softmax function applies a standard exponential function to each element of the input vector and normalizes these values ​​by dividing by the sum of all these exponents. This normalization ensures that the sum of the components of the output vector σ(z) is 1.

[0039] In some embodiments, the exponent base for the logits and softmax functions can be e. However, other exponent bases may be used in different embodiments.

[0040] In some embodiments, the "softmax" function can be a smoothed maximum (a smooth approximation of a maximum function), as is commonly known in machine learning, or it can be a smooth approximation of the argmax function (a function whose value is the index in which the maximum value is).

[0041] The interaction between the working model and short-term and long-term semantic memory enables effective knowledge consolidation and maintains a moderating balance between the model's plasticity and stability. This is further described below. Figure 1-2 This highlights the similarities between the CLS theory in the brain and the main components of this method.

[0042] In some embodiments, the disclosed CLS-ER-based method involves training a working model on a data stream D sampled from a non-IDD (independent and identically distributed) distribution. This makes the distribution neither independent nor distinct. The working model maintains two exponentially weighted average models as semantic memory, particularly the plasticity model. and stable model .

[0043] Since CLS-ER aims to be a versatile and general incremental learning method, the disclosed method does not require any strong assumptions about task boundaries or sample distributions. The disclosed CLS-ER-based learning method employs a reservoir sampling approach to maintain small episodic memories M, which attempts to match the distribution of the data stream D and give each sample an equal chance of being added to the episodic memory.

[0044] In some embodiments, at each training step, the working model receives training batches X from the data stream. b And retrieve sample X from episodic memory. m Training then involves retrieving semantic information about the samples from both semantic memories, as described further below. In some embodiments, the parameters of the plastic model and the stable model are designed so that the plastic model performs better on recent tasks, while the stable model preferentially retains information about older tasks.

[0045] In some embodiments, to avoid utilizing any task information and employ a simple and flexible approach, the implementation selects replay logits Z from the plastic and stable model logits based on which model has the highest softmax score in the ground truth class, such as... Figure 6B Lines 5-6 of Algorithm 1 are shown and further described below.

[0046] In some embodiments, replay logits from semantic memory are then used to reinforce the execution consistency term on the working model so that the working model does not deviate from the learned experience. Therefore, the joint cross-entropy loss (denoted as X) of the data stream and episodic memory samples and the sample consistency loss (denoted as X) are used. m The combination of ) is used to update the working model.

[0047] In some embodiments, the training method may use a loss function such as that defined by equation (1) below. (Refer to...) Figure 6B The function will be described in further detail.

[0048] (1)

[0049] In equation (1), σ is the softmax function, λ is the regularization parameter, and L... MSE It is the mean squared error loss. Additionally, L CE It is cross-entropy loss. After updating the working model (e.g., using gradient descent), the published training method at a rate r P and r S The plasticity and stability models are updated randomly. In some embodiments, r P≥r S This means that the plasticity model is updated more frequently. In some embodiments, the plasticity model represents the most recent training samples, while the stable model represents older training samples from a longer period of time.

[0050] In some embodiments, the plasticity model employs a decay parameter α. P The working model is updated by an exponentially weighted average of the parameters, while the stable model is updated by using a decay parameter α. S The working model parameters are updated by an exponentially weighted average. For example, equation (2) shows an exemplary formula for updating the model parameters of the plastic and stable models.

[0051] (2)

[0052] For example, the attenuation parameter is chosen as α. P ≤α S This allows plastic models to rapidly adapt to changing information. In contrast, stable models simulate slower, structured memories. Figure 6B Algorithm 1 provides additional details of an exemplary training method for training the model in pseudocode.

[0053] Figure 1 A schematic diagram of an exemplary system for supplementary learning according to some embodiments of the present disclosure is shown. Figure 1 The data stream D 102 includes, for example, multiple images that can be used as training samples. Episodic memory 104 uses a reservoir sampling method to maintain a reservoir database 114. The reservoir database 114 stores data samples used as an episodic memory library. During training, episodic memory 104 provides data samples X to semantic memory 112 and the working model 106. M The working model 106 consists of a set of model parameters θ. W Definition. Semantic memory 112 includes its model parameters θ P Defined plastic model 108 and its model parameters θ S Defined stable model 110.

[0054] The working model 106 and the semantic memory 112 interact with each other. For example, the semantic memory 112 has a plastic model 108 and a stable model 110 as instances, and can use a rate r. P and r S Update (by updating their respective model parameters), such as Figure 6B As shown in the diagram. Furthermore, the working model 106 is updated based on the plasticity model 108 and / or the stable model 110, and in addition to the cross-entropy loss, it is also based on the consistency loss, as shown in the diagram. Figure 6B As shown.

[0055] Figure 2Different models for supplementary learning according to some embodiments of this disclosure are shown. Figure 2 The working model 106, the plastic model 108, and the stable model 110 are shown. Figure 2 The characteristics of each model are also shown.

[0056] For example, working model 106 memorizes events with similar plots. Working model 106 also learns the statistical structure of perceived events. Plastic model 108 is suitable for rapid learning of recent experiences. In particular, plastic model 108 is suitable for short-term adaptation and efficient representation of recent tasks. Stable model 110 is suitable for slow learning of structural knowledge. In particular, stable model 110 is suitable for long-term retention. Accordingly, stable model 110 can provide efficient representations across tasks. Thus, working model 106, plastic model 108, and stable model 110 are trained to have different properties, as defined by how they are updated.

[0057] Figure 3 A schematic diagram of an exemplary image analysis system 300 according to some embodiments of the present disclosure is shown. Figure 3 As shown, the image analysis system 300 may include components for performing two phases: a training phase and a prediction phase. The prediction phase may also be referred to as a classification phase or a recognition phase. In some embodiments, the prediction phase acquires the raw image and interprets or assigns meaning to it. For example, the prediction phase may perform image analysis tasks such as image segmentation, image classification, object recognition, motion prediction, etc.

[0058] To perform the training phase, the image analysis system 300 may include a training database 301 and a model training device 302. To perform the prediction phase, the image analysis system 300 may include an image processing device 303 and an image database 304. In some embodiments, the image analysis system 300 may include... Figure 3 The image analysis system 300 may include more or fewer components as shown. For example, when a predictive model for providing image-based predictions is pre-trained and provided, the image analysis system 300 may include only the image processing device 303 and the image database 304. As another example, the image analysis system 300 may include only the training database 301 and the model training device 302 for performing model training tasks only.

[0059] Image analysis system 300 may optionally include network 306 to facilitate communication between various components of image analysis system 300, such as databases 301 and 304, and devices 302, 303, and 305. For example, network 306 may be a local area network (LAN), a wireless network, a cloud computing environment (e.g., Software as a Service, Platform as a Service, Infrastructure as a Service), a client-server architecture, a wide area network (WAN), etc. In some embodiments, network 306 may be replaced by a wired data communication system or device.

[0060] In some embodiments, such as Figure 3 As shown, the various components of the image analysis system 300 may be geographically distant from each other or located in different locations, and are connected via network 306. In some alternative embodiments, some components of the image analysis system 300 may be located in the same location or within a single device. For example, the training database 301 may be located in the field together with the model training device 302, or may be part of the model training device 302. As another example, the model training device 302 and the image processing device 303 may be located within the same computer or processing device.

[0061] Model training device 302 can use training data received from training database 301 to train a predictive model for analyzing images received from, for example, image database 304, in order to provide prediction or recognition results. Figure 3 As shown, the model training device 302 can communicate with the training database 301 to receive one or more sets of training data. In some embodiments, each set of training data may include ground truth (e.g., image classification labels, image recognition results, etc.). The trained prediction model may include one or more inference models. For example, the prediction model may take the form of a working model 106, a plastic model 108, and a stable model 110. (The last sentence appears to be incomplete and possibly refers to a different implementation.) Figures 6A-6B Describe an exemplary method for training such a model.

[0062] The training images stored in training database 301 can be obtained from an image database containing previously acquired images (which have been analyzed and correlated with their ground realities). In some embodiments, training database 301 may include a scenario database (e.g., reservoir database 114) that stores data samples for use as scenario memory. The data samples stored in the scenario database include paired images and corresponding logits provided by plasticity model 108 and / or stable model 110.

[0063] In some embodiments, during the training phase, images may be processed by model training device 302 to identify specific types of images and image characteristics or features. The prediction results are compared with an initial probability analysis, and based on the differences, the model training device 302 refines / optimizes the model parameters. For example, the initial classification or prediction may be performed and validated.

[0064] In some embodiments, the training phase can be performed "online" or "offline." "Online" training refers to performing the training phase concurrently with the prediction phase; for example, learning the model in real time before analyzing images. "Online" training may help obtain an up-to-date inference model based on the training data available at that time.

[0065] However, the computational cost of performing "online" training can be high, and it may not always be feasible if the training data is large and / or the model is complex. Offline trained inference models are saved and reused for image analysis. Furthermore, the use of the working model 106, the plastic model 108, and the stable model 110 allows training and prediction to better reflect how the training data from the training database 301 changes over time.

[0066] The model training device 302 can be implemented using hardware specifically programmed by software that performs the training process. For example, the model training device 302 may include a processor and at least one non-transitory computer-readable medium, such as in combination with... Figure 4 This will be discussed in further detail. The processor can perform training by executing instructions of a training process stored in a computer-readable medium. The model training device 302 may also include input and output interfaces for communicating with the training database 301, the network 306, and / or the user interface (not shown).

[0067] The user interface can be used to select the training dataset, adjust one or more parameters of the training process, select or modify the framework of the inference model, and provide prediction results associated with the training images. However, once the data stream is provided, such as Figures 6A-6B The provided training can run automatically and requires no user intervention during successful execution.

[0068] Figure 6A A flowchart illustrating an exemplary method 600 for training a CLS-ER-based artificial intelligence inference model according to some embodiments of the present disclosure is shown. Using, as Figure 6B The pseudocode shown is an example to describe in Figure 6A The steps performed in method 600. Figure 6B An experience replay algorithm for a supplementary learning system is shown for training working model 106, plastic model 108, and stable model 110 to manage the learning of multiple DNNs over time. Figure 6A and 6B They will be described together.

[0069] Figure 6B The CLS-ER algorithm is defined. Specifically, Figure 6B Includes data flow D, learning rate η, consistency weight λ, and update rate r. P and r S and the attenuation parameter α P and α S These input parameters will be further described below in the context of how they are used in the CLS-ER algorithm.

[0070] In step S602, method 600 initializes the working model, the stable model, and the plastic model. This initialization prepares the model for the training process. The initialization is performed on the model parameters θ of the working model, respectively. W Model parameters θ of the plastic model P and the model parameters θ of the stable model S Assign initial values. In some embodiments, such as Figure 6B As shown, this initialization can set the three sets of model parameters to be the same, that is, θ W =θ P =θ S In some alternative embodiments, the model can be initialized to simulate a previously trained model used for the same or similar prediction task and data acquisition setup. For example, if a model is trained for an autonomous vehicle traveling on a road segment, the model parameters of those trained models can be used to initialize a model for another autonomous vehicle traveling on the same or similar road segment. This initialization can accelerate the convergence of the training process, thereby saving computational costs. Method 600 also initializes the context memory M to an empty set because there is no information available as part of the context memory.

[0071] After initialization, method 600 then iterates through steps S604-S614 to update the model parameters θ. W θ P and θ S This continues until the algorithm converges. For example, in Figure 6B In line 1, the pseudocode defines a while loop. Specifically, the main body of the pseudocode provided for the CLS-ER algorithm is a while loop that provides a series of training operations (provided in lines 2-12) until training is complete. For example, training can continue until the results of one or more models meet a predetermined metric or threshold, until the training while loop has been executed a set number of times, or until some other condition indicating that no further training is needed is met, such as the model having stabilized and no longer changing significantly as training progresses.

[0072] In step S604, method 600 receives training batches from a data stream and samples from an episodic memory sample library. Training batches and samples are selected to allow for successful updates to the working model. Training batches include training samples used to update the model to reflect new data, while samples include training samples used to help the model retain its completed learning.

[0073] For example, in Figure 6B In line 2, the algorithm defines the probability distribution of the training batch, specifically (X... b Y b The distribution is based on the distribution of the data stream D. Therefore, the training batch is sampled from the data stream D and has the same distribution, so the training batch can be used as a data source when training models for prediction and classification based on the data stream D.

[0074] exist Figure 6B In line 3, the algorithm samples the samples, specifically from the episodic memory M (X). m , Y m Since the data of episodic memory is stored in the reservoir database 114, such as... Figure 1 As shown, samples can be taken from reservoir database 114. Reservoir sampling assigns an equal probability to each injection to be added to the memory buffer, and the sample distribution in the memory buffer tracks the overall data distribution across all tasks.

[0075] exist Figure 6B In line 4, the algorithm defines (X, Y) as the union of the training batch and the sample, used to update the working memory.

[0076] In step S606, method 600 selects the optimal semantic memory based on a stable model and a plastic model. Specifically, method 600 considers the joint features of the training batch and the samples, and uses such features to select the optimal semantic memory. For example, in Figure 6B In lines 5 and 6, the algorithm selects the optimal semantic memory. This selection involves two component operations. First, in line 5, replay logits Z are extracted for each sample. P and Z S Then, based on the highest softmax score of the class with ground truth, the replay logits Z is selected. P and Z S One of them is Z. The ground truth class represents information that is known to be true. This choice is made in... Figure 6B The sixth line is expressed mathematically.

[0077] In step S608, method 600 calculates the value of the loss function based on the current model output and the logits selected in step S606. This loss function includes cross-entropy loss and consistency loss. The loss function considers the joint cross-entropy loss of the data stream and the contextual memory sample X, as well as the consistency loss based on sample Xm. For example, in Figure 6B In line 7, the algorithm calculates the value of the loss function. This loss function is used to update the working model. The loss function is defined above as equation (1), and is copied here for further discussion:

[0078] (1)

[0079] Here, L represents the overall loss function. CE L represents the joint cross-entropy loss of the data stream and the contextual memory sample X. MSE The term is the mean squared error loss term, managed by the regularization parameter λ. MSE The term is used as a consistency term because the current state of the working model is evaluated using the Z-value derived in lines 5-6 to establish discrepancies, which are then corrected as part of the overall loss function. The loss function is calculated to help determine how the working model deviates from achieving correct predictions and can be used to train the working model with parameters that provide better predictions.

[0080] In step S610, method 600 updates the working model based on the calculated value of the loss function. For example, in Figure 6B In line 8, the algorithm updates the working model using the loss function value calculated in line 8. Specifically, this update uses the gradient descent method, where... This represents the gradient operator. To use this gradient descent method, the working model parameters θ... W The gradient is updated to reflect the product of the working model's gradient, the learning rate η, and the loss function L, as calculated in line 7. Adjusting the learning rate η may be appropriate for different use cases. For example, if the learning rate is too small, training will take a long time, but if the learning rate is too large, training may fail. Although gradient descent is used to update the working model, as... Figure 6B As shown, however, it is conceivable that other types of updates can be used in other embodiments. For example, gradient descent sometimes encounters the problem that local minima and maxima are not global minima and maxima.

[0081] In step S612, method 600 updates the stable and plastic models using an exponentially weighted average based on the model parameters of the working model. Generally, the plastic model adapts to new information quickly, while the stable model adapts more slowly and thus retains information for a longer period.

[0082] For example, in Figure 6B In lines 9-11, the plasticity and stability models are updated. In some embodiments, method 600 uses two variables, a and b, to determine whether the plasticity and stability models should be updated in the current iteration. In some embodiments, this update can be performed randomly. For example, in Figure 6B In line 9, variables a and b are distributed according to a continuous uniform probability distribution U(0,1). Then, variables a and b are respectively compared with the update rate r. P and r S Compare the results to determine whether the model should be updated.

[0083] For example, in Figure 6B In line 10, if a <r P Then, the plasticity model is updated. Specifically, the plasticity model uses a decay parameter α. P The working model parameters are an exponentially weighted average. This update is summarized by the following equation (2), which is reproduced here:

[0084] (2)

[0085] Otherwise, if a>=r P The plastic model is not updated, and its model parameter θ P Remain unchanged. Figure 6B In line 11, if b <r S Then, the update performed on the stable model is similar to the update performed on the plastic model in line 10, except that the decay parameter used is α. S Otherwise, if b>=r S If , then the stable model will not be updated, and its model parameters θ S It remains unchanged. This is because the update rate of the plastic model is higher than that of the stable model, i.e., r... P ≥r S The plastic model updates more frequently than the stable model.

[0086] Note α P ≤α S Therefore, the plastic model simulates rapid adaptation of information, while the stable model simulates slow, structured, and retained memory. For reasoning, the implementation uses the stable model because it retains long-term memory across tasks, consolidates structural knowledge, and features effective learned representations for generalization, such as... Figure 1-2 As shown.

[0087] In step S614, method 600 adds data to a context memory sample library. In some embodiments, the library is designed to retain a random sample of the memory samples so that the library will resemble the overall distribution of the samples. For example, in Figure 6BIn line 12, the algorithm updates the episodic memory. Specifically, the algorithm adds information from the training batches provided by the data stream to the library to be stored as episodic memory M. As discussed, the episodic memory is updated based on the training batches in a way that gives the episodic memory a distribution similar to that of the data stream D.

[0088] In step S616, method 600 checks whether the training stopping criterion is met. If the stopping criterion is not met (step S616: No), method 600 returns to step S604. Otherwise, if the stopping criterion is met (step S616: Yes), method 600 proceeds to step S618, where training ends. For example, if one or more models meet or exceed a performance metric threshold, method 600 may determine that training is complete and provide the parameters of the trained model for subsequent use. Figure 6B As shown, after line 12, if the training while loop continues, then training will continue. Figure 6B The second line is restored. If training is complete, the training method returns θ. W θ P and θ S, These are the parameters of the trained working model 106, plastic model 108, and stable model 110, respectively.

[0089] Consistent with some embodiments, the trained stable model can be used by image processing device 303 to analyze new images for prediction purposes. Image processing device 303 can receive one or more prediction models from model training device 302 and train them as described. Image processing device 303 may include a processor and a non-transitory computer-readable medium (in combination with...) Figure 4 (Detailed discussion follows). The processor can execute instructions from an image analysis program stored on the medium.

[0090] Image processing device 303 may additionally include input and output interfaces (in combination) Figure 4 (Discussed in detail), to communicate with image database 304, network 306, and / or user interface (not shown). The user interface can be used to select images for analysis, prediction, and / or recognition, initiate the analysis process, and display prediction results.

[0091] Image processing device 303 can communicate with image database 304 to receive images. Images can be acquired by image acquisition device 305. In some embodiments, image acquisition device 305 can be a sensor such as a camera, video camera, LiDAR, medical imaging scanner, etc. The acquired images can depict the environment or scene around the sensor. For example, image acquisition device 305 may include one or more sensors, such as cameras and LiDAR, equipped on autonomous or semi-autonomous vehicles to capture images of the environment around the vehicle. As another example, image acquisition device 305 may be a surveillance camera that acquires images of the surroundings to capture objects appearing in the vicinity and their activities.

[0092] Image processing device 303 can perform initial processing on images. For example, various preprocessing techniques can be performed on the images to make them easier to predict. In some embodiments of this disclosure, image processing device 303 can perform analysis to identify the type or attributes of an image. For example, image processing device 303 can generate probability scores for the type or features of an image. Image processing device 303 can also generate and provide prediction results based on the probability scores of underlying objects.

[0093] Figure 4 A schematic diagram of an exemplary image processing apparatus 400 according to some embodiments of the present disclosure is shown. The systems and methods of the present disclosure can be implemented using computer systems, such as... Figure 4 As shown. The image processing device 400 can be combined with Figure 3The image processing device 303 described herein is an embodiment. In some embodiments, the image processing device 400 may be a dedicated device or a general-purpose device. For example, the image processing device 400 may be a computer customized for handling image data acquisition and image data processing tasks, or a server in a cloud environment. The image processing device 400 may include one or more processors 408, one or more storage devices 404, and one or more memory devices 406. The processors 408, storage devices 404, and memory devices 406 may be configured in a centralized or distributed manner. The image processing device 400 may also include an image database (optionally stored in storage device 404 or remote storage), input / output devices (not shown, but may include a touchscreen, keyboard, mouse, speaker / microphone, etc.), a network interface (e.g., communication interface 402), a display (not shown, but may be a cathode ray tube (CRT), liquid crystal display (LCD), light emitting diode (LED), etc.), and other accessories or peripherals. Various components of the image processing device 400 can be connected via a bus 410, which can be a physical and / or logical bus within or between computing devices.

[0094] Processor 408 may be a processing device that includes one or more general-purpose processing devices, such as a microprocessor, a central processing unit (CPU), a graphics processing unit (GPU), etc. More specifically, processor 408 may be a complex instruction set computing (CISC) microprocessor, a reduced instruction set computing (RISC) microprocessor, a very long instruction word (VLIW) microprocessor, a processor running other instruction sets, or a processor running a combination of instruction sets. Processor 408 may also be one or more special-purpose processing devices, such as application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), digital signal processors (DSPs), system-on-chips (SoCs), etc.

[0095] Processor 408 may be communicatively coupled to storage device 404 and / or memory device 406 and configured to execute computer-executable instructions stored therein. For example, such as Figure 4 As shown, bus 410 can be used, although a logical or physical star or ring topology would be an example of other acceptable communication topologies. Storage device 404 and / or memory device 406 may include read-only memory (ROM), flash memory, random access memory (RAM), static memory, volatile or non-volatile, magnetic, semiconductor, magnetic tape, optical, removable, non-removable, or other types of storage devices or tangible (e.g., non-transitory) computer-readable media. In some embodiments, storage device 404 may store computer-executable instructions of one or more processing programs for processing a learning network (e.g., models 106, 108, and 110) and data generated during the execution of the computer program (e.g., images for prediction / classification). Data may be read from storage device 404 and stored in memory device 406 one at a time or simultaneously. Processor 408 may execute processing programs to implement the various steps of the methods described below. The processor 408 can also send data to or receive data from the storage device 404 and / or memory device 406 via the bus 410.

[0096] Image processing device 400 may also include one or more digital and / or analog communication (input / output) devices. Figure 4 Not shown. For example, input / output devices may include a keyboard and mouse (or trackball) that allow the user to provide input. The image processing device 400 may also include a network interface, such as the communication interface 402 shown, for example, a network adapter, cable connector, serial connector, USB connector, parallel connector, high-speed data transmission adapter (e.g., fiber optic, USB 3.0, Lightning), wireless network adapter (e.g., Wi-Fi adapter), or telecommunications (3G, 4G / LTE, 5G NR, etc.) adapter, etc. The image processing device 400 can be connected to a network via the network interface. As described above, the image processing device 400 may also include a display. In some embodiments, the display may be any display device suitable for displaying images and their prediction or classification results. For example, the image display may be an LCD, CRT, or LED display.

[0097] Image processing device 400 can be connected to model training device 302 and image acquisition device 305, as described above. Figure 3 According to other embodiments, other implementations are also possible.

[0098] Figure 5A flowchart illustrating an exemplary artificial intelligence method for making predictions from image sequences according to some embodiments of the present disclosure is shown.

[0099] In step S502, method 500 receives an image sequence. For example, these images may be a sequence of images acquired at different points in time within a time window. In some embodiments, the image sequence may be captured by sensors such as cameras, video cameras, LiDAR, medical imaging scanners, etc. The image sequence depicts the environment or scene surrounding the sensors. In some embodiments, the image sequence may be a sequence of images captured by different sensors located in the environment. For example, an autonomous or semi-autonomous vehicle may be equipped with multiple sensors (e.g., cameras and LiDAR) to capture images of the environment surrounding the vehicle. The images may depict road conditions, road signs, traffic lights, and other stationary or moving objects in the surrounding area (e.g., other vehicles, pedestrians, trees, etc.). The content depicted by the image sequence changes over time due to the movement of the vehicle and other objects. Method 500 may be performed to make certain predictions based on the images, such as performing a classification task on the images. For example, method 500 may be performed to predict whether the vehicle is likely to collide with an obstacle (e.g., a stationary or moving object in the environment) and make autonomous driving decisions accordingly, such as changing lanes, decelerating or accelerating, generating warnings, etc., to avoid a potential collision.

[0100] To promote more accurate and effective forecasting, Figure 5 Method 500 uses a prediction model trained through continuous learning. Continuous learning combines knowledge learned from previous images in the sequence while simultaneously predicting subsequent images. Since image sequences are captured from a general environment, and although they change over time, images often possess important features and are highly correlated, learning from previous images helps enrich the agent's knowledge for subsequent predictions. Therefore, Method 500 can use a prediction model trained to make more recent images perform better in performing prediction tasks. For example, the prediction model includes a working model trained based on the CLS-ER algorithm.

[0101] Training enhances the consistency between the working model and at least one of the stable and plastic models. Training updates the stable model based on the working model at a first training rate and the plastic model at a second training rate, where the first training rate is slower than the second training rate.

[0102] Therefore, by using different models and training them in the manner discussed in this application, it is possible to better reflect learning over time, so that all training data are considered when training the model, although the most recent training data has a greater impact on the model. Furthermore, the magnitude of this impact may vary depending on the parameters of the rate of change set during model training, such as the update rate and decay rate. The above, combined with... Figures 6A-6BThe training of this prediction model is described.

[0103] In step S504, method 500 applies a trained stable model 110 to process the image sequence for prediction. Once the stable model 110 is applied to the images, predictions can be made based on the images, such as classifying the images or recognizing features of the images. For example, based on images captured by sensors on an autonomous vehicle, the stable model can identify and classify objects depicted in the images and make certain predictions based on knowledge learned through the training process. For example, as an autonomous vehicle moves from one country to another, it may need to acquire new knowledge, such as changes in the shape / appearance of new road signs or signs. The autonomous vehicle can apply the trained stable model to acquire this new knowledge based on images acquired from the new environment. However, predictions are not limited to these examples and can include other predictions, such as predictions for recognition tasks, machine vision tasks, or predictions for other learning tasks.

[0104] Figure 7 An exemplary data flow 700 is shown for training and evaluating a model trained on a CLS-ER according to some embodiments of this disclosure. In 702, the model begins by receiving training data. For example, the training data may be a series of images arriving over a period of time for prediction and / or classification. This training data corresponds to... Figure 1 Data flow D in the data stream. In 704, CLS-ER is trained. The training process includes training the working model using consistency loss from both the stable and plastic models. The above combines... Figures 6A-6B The training process is described.

[0105] After training, the stable model 110 is used as the CLS-ER. In 706, the stable model 110 is applied to the CLS-ER.

[0106] There are too many evaluation protocols in the CL literature, each of which biases the evaluation towards a particular method. In some embodiments, a broad and robust evaluation of the model trained on CLS-ER can be performed to measure the versatility of the method.

[0107] Experimental protocols for training the method on long task sequences, where the boundaries between tasks are not obvious, the tasks themselves are not separate, and the method is uncertain about task boundaries during training or testing, can be considered to meet the expected model quality. The examples focus on the above setup and can be viewed as a general incremental learning (GIL) setup. Here, the examples provide a broad classification of these evaluation protocols for testing different aspects of CL.

[0108] In one example, a stable model can be applied to class incremental learning (Class-IL)708. Class-IL refers to a CL scenario where new classes are added to each subsequent task, and the agent must learn to distinguish not only between classes in the current task but also between classes in previous tasks. Class-IL measures the method's ability to learn general representations, accumulate, consolidate, and propagate the acquired knowledge to learn effective representations and decision boundaries for all classes seen so far. Various Class-IL benchmarks, such as CLS-ER, can be used. These representations increase dataset complexity and extend to longer sequences of Class-IL settings.

[0109] While this is an important and challenging benchmark, it assumes that each subsequent task will have the same number of disjoint classes and uniform samples for each class that does not represent a real-world scenario. Such a benchmark does not consider the relevant Task Increment Learning (Task-IL) setting because it assumes that the availability of task labels during training and inference cannot truly be considered a CL task.

[0110] In another example, a stable model can be applied to domain incremental learning (Domain-IL)710. Domain-IL refers to a class-based learning (CL) scenario where the class remains constant across each subsequent task, but the input distribution changes. For example, a use case could be Rotated-MNIST, where each task contains digits rotated at a fixed angle between 0 and 180 degrees. The example does not consider the relevant popular evaluation protocol Permuted-MNIST, which applies a fixed random permutation to the pixels for each task, as it violates the cross-task similarity requirement and deviates from the goal of continuous learning.

[0111] In another example, a stable model is applied to general incremental learning (GIL) 712. The CL scenario described above cannot absorb real-world challenges, including settings with ambiguous task boundaries where the learning agent must learn from continuous streams of data (classes can reappear and have different data distributions). When dealing with GIL tasks, CL methods must address issues of sample efficiency, imbalanced classes, and efficient knowledge transfer, in addition to preventing catastrophic forgetting.

[0112] To test the effectiveness of the method in this challenging setting, two GIL evaluation protocols can be considered. MNIST-360 models a data stream consisting of batches of two consecutive MNIST images, each sample rotated at an increasing angle, with the sequence repeated three times. This causes the model to exhibit a sharp distribution shift with class changes, while a smoother rotational distribution shift results in a more uniform distribution shift. However, the number of classes and samples remains consistent across tasks.

[0113] Generalized class incremental learning (GCIL) uses probabilistic modeling to sample classes and data distributions for each task. Therefore, the number of classes in each task is not fixed, classes can overlap, and the sample size for each class can vary.

[0114] Therefore, when applied to several types of incremental learning, particularly Class-IL 708, Domain-IL 710, and GIL 712, this approach using CLS-ER provides good results. These are examples of CL in the context of different types of prediction tasks, demonstrating that CLS-ER performs well. As a result of these exemplary evaluations, it can be seen that CLS-ER is well-suited for providing good performance for prediction and classification tasks in machine vision, where CL is associated with successful image classification and computer vision tasks.

[0115] Another aspect of this disclosure relates to a non-transitory computer-readable medium storing instructions that, when executed, cause one or more processors to perform the methods described above. The computer-readable medium may include volatile or non-volatile, magnetic, semiconductor, magnetic tape, optical, removable, non-removable, or other types of computer-readable media or computer-readable storage devices. For example, as disclosed, a computer-readable medium may be a storage device or memory module having computer instructions stored thereon. In some embodiments, the computer-readable medium may be a disk or flash drive having computer instructions stored thereon.

[0116] It will be apparent to those skilled in the art that various modifications and variations can be made to the disclosed systems and related methods. Other embodiments will be readily apparent to those skilled in the art, taking into account the description and practice of the disclosed systems and related methods.

[0117] The specification and examples are intended to be illustrative only, and the true scope is indicated by the claims and their equivalents.

Claims

1. An artificial intelligence method for prediction from image sequences, comprising: Receive the image sequence acquired at different time points; as well as The image sequence is processed using a stable model to make the prediction, wherein the working model is trained together with the stable model and the plastic model, wherein the training enhances the consistency among the working model, the stable model and the plastic model, wherein the working model is trained using a loss function that includes a joint cross-entropy loss of the training batch and the memory samples and a consistency loss on the memory samples; Among them, training batches are received from the data stream, and memory samples are received from the context memory bank; The working model is trained based on a loss function that enhances the working model, stabilizes the model, and plasticizes the model to ensure consistency on the memory samples. The loss function includes the joint cross-entropy loss of the training batch and the memory samples, and the consistency loss on the memory samples. The stable model and the plastic model are updated based on the working model; Determine that the stable model satisfies the training conditions; and The stable model is provided as the artificial intelligence inference model; The working model is used to memorize events with similar plots and learn the statistical structure of perceived events; the stable model is used to slowly learn structural knowledge, maintain long-term semantic memory of experiential events, and provide effective representations across tasks; the plastic model is used to quickly learn recent experiences, maintain short-term semantic memory of experiential events, and provide adaptation and effective representations for recent tasks; the stable model and the plastic model are complementary to the working model, and the working model receives feedback from the stable model and the plastic model.

2. The artificial intelligence method according to claim 1, wherein the training is based on the working model, updating the stable model at a first training rate and updating the plastic model at a second training rate, the first training rate being slower than the second training rate.

3. The artificial intelligence method according to claim 1 or 2, wherein the training batch comes from a data stream and the memory samples come from a contextual memory bank.

4. The artificial intelligence method according to claim 1 or 2, wherein the consistency loss is based on the logits generated by the working model on the memory sample and the replay logits selected for the memory sample from the plastic model or the stable model.

5. The artificial intelligence method of claim 4, wherein the loss function is a weighted combination of the cross-entropy loss and the consistency loss, wherein the consistency loss is the mean square error between the logits generated by the working model on the memory samples and the replay logits from the plastic model or the stable model.

6. The artificial intelligence method according to claim 2, wherein the parameters of the stable model are updated using an exponentially weighted average of the parameters of the working model, and the working model has a first decay parameter at the first training rate; The parameters of the plastic model are updated using an exponentially weighted average of the parameters of the working model, which has a second decay parameter at the second training rate.

7. An artificial intelligence system for making predictions from image sequences acquired by an image acquisition device at different time points, comprising: A storage device for storing a stable model, wherein a working model is trained together with the stable model and a plastic model, wherein the training enhances the consistency among the working model, the stable model, and the plastic model, wherein the working model is trained using a loss function comprising a joint cross-entropy loss of training batches and memory samples and a consistency loss on the memory samples; and Processor, used for: The image sequence is processed using the stable model to make the prediction; Among them, training batches are received from the data stream, and memory samples are received from the context memory bank; The working model is trained based on a loss function that enhances the working model, stabilizes the model, and plasticizes the model to ensure consistency on the memory samples. The loss function includes the joint cross-entropy loss of the training batch and the memory samples, and the consistency loss on the memory samples. The stable model and the plastic model are updated based on the working model; Determine that the stable model satisfies the training conditions; and The stable model is provided as the artificial intelligence inference model; The working model is used to memorize events with similar plots and learn the statistical structure of perceived events; the stable model is used to slowly learn structural knowledge, maintain long-term semantic memory of experiential events, and provide effective representations across tasks; the plastic model is used to quickly learn recent experiences, maintain short-term semantic memory of experiential events, and provide adaptation and effective representations for recent tasks; the stable model and the plastic model are complementary to the working model, and the working model receives feedback from the stable model and the plastic model.

8. The artificial intelligence system of claim 7, wherein the stable model is updated at a first training rate and the plastic model is updated at a second training rate, the first training rate being slower than the second training rate.

9. A non-transitory computer-readable storage medium storing instructions that, when executed by a processor, are used to implement the method as described in any one of claims 1-6.

Images