A visual joint feature incremental learning method and device and a storage medium

Abstract

The application discloses a kind of visual joint feature incremental learning method, device and storage medium.The method utilizes visual language model to model the visual joint feature.Model learning is by obtaining current and sampling historical sample image and the text prompt of all categories, respectively it is encoded and handled, obtains the unknown feature of corresponding image and the class feature of all category prompts of task;Visual feature adaptation network is used to extract visual perception feature based on image task unknown feature;Based on task unknown feature and task perception feature, obtain the visual joint feature of current and sampling historical image;Based on the visual joint feature and all class features, the similarity is calculated to determine the prediction score, and the loss value is determined with the real class of corresponding image;Based on the loss value, the parameters of the visual adaptation network to be updated are updated until the preset end condition is met, and the updated visual adaptation network is obtained.The visual joint feature alleviates the catastrophic forgetting in incremental learning and improves the continuous learning ability of the model.

Description

Technical Field

[0001] This application relates to the field of computer technology, and in particular to an incremental learning method, apparatus and storage medium for visual joint features. Background Technology

[0002] As society continues to develop and progress, new data and knowledge are constantly emerging, prompting machine learning to adapt to the dynamic environment of constantly updated and changing data. In this context, continuous learning has emerged. The vision of continuous learning is that when new data arrives in a streaming manner and new categories or tasks appear, artificial intelligence systems can adapt to new fields and learn new knowledge while retaining old knowledge.

[0003] In related technologies, the continuous learning of image recognition models in visual cognition tasks is essentially the updating and training of the visual adaptation network within the image recognition model. However, as the visual adaptation network continuously learns from new data, it is prone to catastrophic forgetting of older data, leading to a decrease in the recognition accuracy of the updated visual adaptation network, which fails to meet the needs of practical applications. Summary of the Invention

[0004] To address the problems of existing technologies, this application provides an incremental learning method, apparatus, and storage medium for visual joint features. The technical solution is as follows:

[0005] On the one hand, an incremental learning method for visual joint features is provided, the method comprising:

[0006] Obtain the current sample image and the category features of each preset category corresponding to the current sample image;

[0007] Encode the current sample image to obtain the unknown features for the current task;

[0008] Based on the visual adaptation network to be updated, feature extraction is performed on the unknown features of the current task and the unknown features of the historical task to obtain the current task perception features and the historical task perception features; the unknown features of the historical task are the unknown features of the task in the historical update.

[0009] Based on the current task perceived features, the current task unknown features, and the category features of each preset category corresponding to the current sample image, determine the similarity of the current sample image to each preset category;

[0010] Based on the unknown features of the historical task, the perceived features of the historical task, and the category features of each preset category corresponding to the current sample image, the similarity of the unknown features of the historical task to each preset category is determined.

[0011] The loss value is determined based on the similarity of the current sample image to each preset category and the similarity of the unknown features of the historical task to each preset category.

[0012] Based on the loss value, the network parameters of the visual adaptation network to be updated are updated until the preset update termination condition is met, and the updated visual adaptation network is obtained.

[0013] On the other hand, an incremental learning device for visual joint features is provided, the device comprising:

[0014] The acquisition module is used to acquire the current sample image and the category features of each preset category corresponding to the current sample image;

[0015] The encoding module is used to encode the current sample image to obtain unknown features for the current task;

[0016] The feature extraction module is used to extract features from the unknown features of the current task and the unknown features of the historical task based on the visual adaptation network to be updated, to obtain the current task perception features and the historical task perception features; the unknown features of the historical task are the unknown features of the task in the historical update.

[0017] The first similarity determination module is used to determine the similarity of the current sample image to each preset category based on the current task perceived features, the current task unknown features, and the category features of each preset category corresponding to the current sample image.

[0018] The second similarity determination module is used to determine the similarity between the unknown features of the historical task and each preset category based on the unknown features of the historical task, the perceived features of the historical task, and the category features of each preset category corresponding to the current sample image.

[0019] The loss value determination module is used to determine the loss value based on the similarity of the current sample image to each preset category and the similarity of the unknown features of the historical task to each preset category.

[0020] The update module is used to update the network parameters of the visual adaptation network to be updated based on the loss value until the preset update termination condition is met, and the updated visual adaptation network is obtained.

[0021] In one exemplary implementation, the first similarity determination module includes:

[0022] The first fusion module is used to fuse the current task's perceived features and the current task's unknown features to obtain the current joint visual features;

[0023] The third similarity determination module is used to determine the similarity between the category features of each preset category corresponding to the current sample image and the current visual joint features, so as to obtain the similarity of the current sample image to each preset category.

[0024] The second similarity determination module includes:

[0025] The second fusion module is used to fuse unknown features of historical tasks and perceived features of historical tasks to obtain joint historical visual features.

[0026] The fourth similarity determination module is used to determine the similarity between the category features of each preset category corresponding to the current sample image and the historical visual joint features, so as to obtain the similarity between the unknown features of the historical task and each preset category.

[0027] In one exemplary implementation, the loss value determination module includes:

[0028] The normalization processing module is used to normalize the similarity of the current sample image to each preset category, and obtain the normalization result corresponding to the current sample image; the normalization result represents the probability that the current sample image belongs to each preset category;

[0029] The first loss value determination module is used to determine the current classification loss value based on the difference between the normalization result corresponding to the current sample image and the classification label corresponding to the current sample image.

[0030] The first acquisition module is used to acquire the historical similarity of unknown features of historical tasks for each preset category; the historical similarity is the similarity of unknown features of historical tasks in historical updates;

[0031] The second loss value determination module is used to determine the historical classification loss value based on the difference between the similarity of each preset category corresponding to the unknown features of the historical task and the historical similarity.

[0032] The third loss value determination module is used to determine the loss value based on the current classification loss value and the historical classification loss values.

[0033] In one exemplary implementation, the second loss value determination module includes:

[0034] The fourth loss value determination module is used to determine the first historical classification loss value based on the classification labels corresponding to the unknown features of the historical task and the historical prediction classification results.

[0035] The fifth loss value determination module is used to determine the second historical classification loss value based on the difference between the similarity of each preset category corresponding to the unknown features of the historical task and the historical similarity.

[0036] The sixth loss value determination module is used to determine the sum of the first historical classification loss value and the second historical classification loss value to obtain the historical classification loss value.

[0037] In one exemplary embodiment, the fifth loss value determination module includes:

[0038] The difference determination module is used to determine the difference between the similarity of the unknown features of the historical task corresponding to each preset category and the historical similarity of the unknown features of the historical task corresponding to each preset category, so as to obtain the similarity difference value of the perceived features of the historical task corresponding to each preset category.

[0039] The statistics module is used to perform statistical processing on the square of the similarity difference value of each preset category corresponding to the unknown features of the historical task, so as to obtain the target similarity difference value of the unknown features of the historical task.

[0040] The seventh loss value determination module is used to determine the second historical classification loss value based on the target similarity difference value of unknown features in the historical task.

[0041] In one exemplary embodiment, the historical task unknown feature is any one of the task unknown features in the set of historical task unknown features; the apparatus further includes:

[0042] The first storage module is used to store the unknown features of the current task into the unknown feature set of the target task;

[0043] The first update module is used to update the target task unknown feature set based on the newly generated current task unknown features when the number of current task unknown features in the target task unknown feature set is equal to the preset number of task unknown features and new current task unknown features are obtained, so as to obtain a sampled feature set; the number of current task unknown features in the sampled feature set is equal to the preset number of task unknown features.

[0044] The second update module is used to update the unknown feature set of the historical task based on the sampled feature set after the visual adaptation network has been updated.

[0045] In one exemplary implementation, the first update module includes:

[0046] The second storage module is used to store the newly generated unknown features of the current task into the unknown feature set of the target task, so as to obtain the quasi-unknown feature set of the target task.

[0047] The sampling module is used to randomly sample a preset number of unknown features of the current task from the unknown feature set of the quasi-target task to obtain the sampled feature set.

[0048] On the other hand, an electronic device is provided, including a processor and a memory, wherein the memory stores at least one instruction or at least one program, the at least one instruction or the at least one program being loaded and executed by the processor to implement an incremental learning method for visual joint features in any of the above aspects.

[0049] On the other hand, a computer-readable storage medium is provided, wherein at least one instruction or at least one program is stored therein, the at least one instruction or the at least one program being loaded and executed by a processor to implement an incremental learning method for visual joint features as described above.

[0050] On the other hand, a computer program product or computer program is provided, comprising computer instructions stored in a computer-readable storage medium. A processor of an electronic device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the electronic device to perform the incremental learning method for visual joint features according to any of the above aspects.

[0051] In this embodiment, the current task involves acquiring two modal data: the current sample image and the category features corresponding to each preset category. The category features for each preset category are obtained by inputting the category hint text corresponding to each preset category into a text encoding network. The current sample image is first encoded using an image encoding network to obtain unknown features for the current task. Subsequently, a visual adaptation network to be updated is used to extract features from both the unknown features for the current task and the unknown features for historical tasks, obtaining perceptual features for the current task and perceptual features for historical tasks, respectively. The similarity between the current sample image and each preset category, as well as the similarity between the unknown features for historical tasks and each preset category, are calculated and determined based on these two similarities. Determine the loss value; based on the loss value, update the network parameters of the visual adaptation network to be updated until the preset update termination condition is met, and obtain the updated visual adaptation network; since the input of the visual adaptation network to be updated includes two modalities of data, namely image and text data of the current task, it takes into account the semantic information contained in the natural language of the category label text, makes better use of the context relevance in the text space, and provides more effective information, and also includes the task-unknown features in the historical tasks, that is, it includes historical data features. This makes the plasticity of the network model improved while maintaining stability during the continuous learning and updating process of the visual adaptation network, reduces the catastrophic forgetting of the visual adaptation network in visual cognitive tasks, and thus improves the accuracy of the updated visual adaptation network. Attached Figure Description

[0052] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0053] Figure 1 This is a schematic diagram of an implementation environment provided in an embodiment of this application;

[0054] Figure 2 This is a flowchart illustrating an incremental learning method for visual joint features provided in an embodiment of this application;

[0055] Figure 3 This is a schematic diagram of the training process of a visual adaptation network provided in an embodiment of this application;

[0056] Figure 4 This is a flowchart illustrating another incremental learning method for visual joint features provided in an embodiment of this application;

[0057] Figure 5 This is a structural block diagram of an incremental learning device for visual joint features provided in an embodiment of this application;

[0058] Figure 6 This is a hardware structure block diagram of an electronic device provided in an embodiment of this application. Detailed Implementation

[0059] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of this application.

[0060] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or server that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or devices.

[0061] It is understood that in the specific embodiments of this application, data such as user information are involved. When the above embodiments of this application are applied to specific products or technologies, user permission or consent is required, and the collection, use and processing of related data must comply with the relevant laws, regulations and standards of the relevant countries and regions.

[0062] The terminology used in the embodiments of this application will be explained below.

[0063] Catastrophic forgetting: In machine learning, this term is often used to describe a situation where a network model experiences a catastrophic decline in performance on older tasks after learning a new task. The main reason for this is that the new data for the new task has a low similarity to the old data. When the network model is trained based on the new data for the new task, it may disrupt the weights of the network model on the old task, resulting in unsatisfactory test results for the trained network model on the old data.

[0064] Continuous learning refers to a learning method in which a network model learns new task knowledge while simultaneously retaining knowledge from older tasks. It can also be called incremental learning. In this learning process, typically the tasks in the previous t-1 tasks are the same as the current task t, for example, both are visual recognition tasks. Assuming the network model has already learned the previous t-1 tasks (where the training data similarity between tasks in the previous t-1 tasks is low), when faced with the current task t and its corresponding current data (where the current data has low similarity to the training data of the previous t-1 tasks), the goal is to utilize the prior knowledge learned from the previous t-1 tasks to aid in learning the new task t, and then update the knowledge learned by the network model (i.e., the network parameters). Existing incremental learning methods can be mainly divided into the following three paradigms:

[0065] (1) Based on regularization

[0066] The main idea behind regularized incremental learning is to protect knowledge from being overwritten by new knowledge by imposing constraints on the loss function of the new task. Essentially, it involves introducing an additional loss function to correct the gradient, protecting the model's learned knowledge and providing a mitigation against catastrophic forgetting under certain conditions. These methods typically do not require the model to review previously learned tasks using old data, making them relatively elegant. However, this approach is highly dependent on the correlation between new and old tasks. When the differences between the new and old tasks are too large, task confusion can occur, and training time increases linearly with the number of learning tasks. Furthermore, the introduced regularization term often fails to effectively constrain the model's generalization on new tasks.

[0067] (2) Parameter-based isolation

[0068] Incremental learning methods based on parameter isolation consider that changing the parameters of an old task will affect its performance. They attempt to incrementally expand the model without altering the old task's parameters, isolating the parameters of the old and new tasks for incremental learning. Each time a new task arrives, a portion of the model space is used incrementally, and redundant model space is preserved through methods such as pruning, leaving room for the next task. For a new task, the training process consists of two steps: First, the model fixes the parameters of the old task and trains the current task using the entire model. After completion, some unimportant parameters are removed using pruning. Second, the model is retrained using the remaining parameter space. Clearly, this type of method allocates a portion of the parameter space for each task, limiting the number of tasks and imposing requirements on the order of tasks.

[0069] (4) Based on playback

[0070] In incremental learning based on playback, when training a new task, a portion of representative old data is retained and used to review previously learned knowledge in the model. Therefore, retaining some data from the old task and how to use the old data to train the model together with the new data are the main issues that need to be considered for this type of method.

[0071] Artificial Intelligence (AI) is the theory, methods, technology, and application systems that use digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use that knowledge to achieve optimal results. In other words, AI is a comprehensive technology within computer science that attempts to understand the essence of intelligence and produce a new kind of intelligent machine that can react in a way similar to human intelligence. AI studies the design principles and implementation methods of various intelligent machines, enabling them to have perception, reasoning, and decision-making capabilities. AI technology is a comprehensive discipline involving a wide range of fields, encompassing both hardware and software technologies. Fundamental AI technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing, operating / interactive systems, and mechatronics. AI software technologies mainly include computer vision, speech processing, natural language processing, and machine learning / deep learning.

[0072] Computer vision (CV) is a science that studies how to enable machines to "see." More specifically, it refers to machine vision, which uses cameras and computers to replace human eyes in recognizing and measuring targets, and then performs image processing to create images more suitable for human observation or transmission to instruments. As a scientific discipline, computer vision studies related theories and technologies, attempting to build artificial intelligence systems capable of extracting information from images or multidimensional data. Computer vision technologies typically include image processing, image recognition, image semantic understanding, image retrieval, OCR, video processing, video semantic understanding, video content / behavior recognition, 3D object reconstruction, 3D technology, virtual reality, augmented reality, simultaneous localization and mapping (SLAM), and common biometric recognition technologies such as facial recognition and fingerprint recognition.

[0073] Machine Learning (ML) is a multidisciplinary field involving probability theory, statistics, approximation theory, convex analysis, and algorithm complexity theory. It specifically studies how computers can simulate or implement human learning behavior to acquire new knowledge or skills and reorganize existing knowledge structures to continuously improve their performance. Machine learning is the core of artificial intelligence and the fundamental way to endow computers with intelligence; its applications span all areas of artificial intelligence. Machine learning and deep learning typically include techniques such as artificial neural networks, belief networks, reinforcement learning, transfer learning, inductive learning, and instructional learning.

[0074] With the continuous development of artificial intelligence technology, the demand for image recognition processing through models or networks is becoming increasingly significant. In a dynamic environment where new data and knowledge are constantly being updated and changed, higher requirements are being placed on the recognition accuracy of image recognition models.

[0075] Most existing continuous learning methods utilize only the image modality to solve visual cognitive tasks. The semantic information of images is applied only through a series of discretely encoded class labels. This single-modality-based continuous learning approach inevitably suffers from catastrophic forgetting problems when learning new tasks. This problem points to a universal issue faced by neural networks: the stability-plasticity dilemma. Plasticity represents the ability of a neural network model to adapt and integrate new knowledge, while stability represents the ability of the neural network model to retain old knowledge during the adaptation process. Therefore, how to improve the plasticity of the network model while maintaining stability, and achieve a balance between the two, is a challenge faced in the continuous learning research of image recognition models in visual cognitive tasks.

[0076] Continuous learning of image recognition models is essentially incremental learning and training of the visual joint features in the image recognition model. To this end, this application discloses an incremental learning method for visual joint features. This method acquires two modal data: the current sample image and the category features corresponding to each preset category of the current sample image. The category features of each preset category are obtained by inputting the category prompt text corresponding to each preset category into a text encoding network. The current sample image is first encoded using an image encoding network to obtain unknown features for the current task. The visual adaptation network to be updated then extracts features from both the unknown features for the current task and the unknown features for the historical task, obtaining the perceived features for the current task and the perceived features for the historical task. A loss value is determined based on the similarity between the current sample image and each preset category, and the similarity between the unknown features for the historical task and each preset category. Based on the loss value, the network parameters of the visual adaptation network to be updated are updated until a preset update termination condition is met, resulting in the updated visual adaptation network. Since the input to the visual adaptation network to be updated includes both image and text modal data for the current task, it takes into account the semantic information contained in the natural language of the category label text, making better use of the contextual relevance in the text space and providing more effective information. It also includes task-unknown features from historical tasks, i.e., it includes historical data features. This allows the visual adaptation network to improve its plasticity while maintaining stability during continuous learning and updating, reducing the catastrophic forgetting of the visual adaptation network in visual cognitive tasks, and thus improving the accuracy of the updated visual adaptation network.

[0077] Please see Figure 1 The diagram shows an implementation environment provided by an embodiment of the present invention. The implementation environment includes a terminal 110 and a server 120, wherein the terminal 110 and the server 120 can communicate through a wired or wireless network connection.

[0078] Terminal 110 includes, but is not limited to, mobile phones, computers, smart voice interaction devices, smart home appliances, vehicle terminals, and aircraft. Terminal 110 is equipped with client software with image processing capabilities, such as an application (App). This application can be a standalone application or a subroutine within an application.

[0079] Server 120 can provide background services for applications in terminal 110, specifically image recognition services. Specifically, server 120 can store pre-trained image recognition models; these models include an image-text encoding network, a visual adaptation network, and a classification network. Based on the image-text encoding network, features are extracted from the image and its corresponding category prompt text to obtain unknown features for the current task and category features for each preset category corresponding to the current image. Based on the visual adaptation network, features are extracted from the unknown features for the current task to obtain current visual features. Based on the current visual features, the unknown features for the current task, and the category features for each preset category corresponding to the current image, the similarity between the current image and each preset category is determined. Based on the classification network, the similarity between the current image and each preset category is normalized to obtain the image recognition result, which is then returned to client 110.

[0080] Among them, server 120 can be an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN (Content Delivery Network), and big data and artificial intelligence platforms.

[0081] In one exemplary embodiment, both terminal 110 and server 120 can be node devices in a blockchain system, capable of sharing acquired and generated information with other node devices in the blockchain system, thus enabling information sharing among multiple node devices. Multiple node devices in the blockchain system can be configured with the same blockchain, which consists of multiple blocks, and adjacent blocks are related, ensuring that any data tampering in any block can be detected by the next block. This prevents data tampering in the blockchain and guarantees the security and reliability of the data in the blockchain.

[0082] The embodiments disclosed herein can be applied to various scenarios, including but not limited to cloud technology, artificial intelligence, smart transportation, and assisted driving.

[0083] Please see Figure 2The diagram illustrates a flowchart of an incremental learning method for visual joint features provided in this application. This method can be applied to electronic devices, which may be terminals or servers. It should be noted that this specification provides the operational steps described in the embodiments or flowcharts, but more or fewer operational steps may be included based on conventional or non-inventive labor. The order of steps listed in the embodiments is merely one possible execution order among many steps and does not represent the only execution order. In actual system or product execution, the methods shown in the embodiments or drawings can be executed sequentially or in parallel (e.g., in a parallel processor or multi-threaded processing environment). Specifically, as shown... Figure 2 As shown, the method may include:

[0084] S201: Obtain the current sample image and the category features of each preset category corresponding to the current sample image.

[0085] The current sample image can be a set of sample images consisting of multiple sample images, and these sample images can be of the same category, such as all being dogs (see [reference]). Figure 3 (); it can also be multiple, such as cars, cats, and dogs.

[0086] In an exemplary implementation, the method for obtaining the category features of the current sample image corresponding to each preset category in step S201 is to directly obtain the category features of the current sample image corresponding to each preset category from the corresponding storage space. The category features of the current sample image corresponding to each preset category can be the category features of sample images corresponding to each preset category stored in any historical task. However, for any task in the historical task, it is necessary to obtain the category hint text of all preset image categories, and encode these category hint texts of preset image categories into the preset category space based on a text encoding network to obtain the category features of the sample image corresponding to each preset category. (See also) Figure 3 The obtained sample images are then stored along with the category features corresponding to each preset category.

[0087] The preset image categories can be set according to the training requirements of the image recognition model in actual applications. For example, taking road scene detection and recognition in intelligent driving as an example, the image categories can be coarse-grained categories, such as cars, people, traffic lights, and dogs. Of course, the image categories can also be finer-grained categories as needed. For example, for vehicles, they can be divided into sedans, trucks, and electric vehicles, etc. There is no limitation here.

[0088] The category hint text for the preset image category can be a pre-set category hint text corresponding to the sample image; the category hint text corresponding to the sample image can be expressed as "A photo of a [class]", where [class] represents the specific category name, such as cat or dog. When the sample image is an image of a dog, its corresponding category text description can be "A photo of a dog". The category hint text is not limited to the English in the example, and can be Chinese or other languages ​​as needed.

[0089] It should be noted that the application scenario of this application can be that both the new task and the old task are visual cognition tasks, specifically image classification or image recognition tasks. When the image category of the current sample image of the new task is not very similar to the image category of the historical sample image of the old task, so that the current image recognition model cannot accurately identify the image category, it is necessary to learn these newly added image categories and update the network parameters of the visual adaptation network so that the entire image recognition model can accurately identify the newly added image categories. On the other hand, when the image category of the current sample image of the new task is very similar to the image category of the historical sample image of the old task, or even the same, learning the image category of the current sample image of the new task is only to further improve the recognition accuracy of the image recognition model. For example, the image category learned by the old task is cat and dog, and the image category of the sample image of the new task is also dog. However, since the dog images in the training samples of the old task do not include wolfhounds, while the sample images of the new task are wolfhound images, training the model based on the new task can improve the accuracy of the image recognition model in recognizing wolfhounds, that is, recognizing wolfhounds as dogs instead of wolves.

[0090] S203: Encode the current sample image to obtain the unknown features of the current task.

[0091] Among them, encoding the current sample image to obtain the unknown features of the current task can also be called the unknown features of the current task.

[0092] In this embodiment, the method can be applied to image recognition tasks, thereby achieving image classification and recognition by inputting images into an image recognition model; specifically, the image recognition model includes an image encoding network, a text encoding network, a visual adaptation network, and a classification network; see reference. Figure 3 Image coding network The unknown features of the current task are obtained by encoding the current sample image I. in, This represents the network parameters of an image coding network, with unknown features for the current task. Represents the unknown features of the current task t; text encoding network This is used to encode the category hint text T corresponding to the current sample image into a preset category space to obtain the category features of each preset category corresponding to the current sample image. in This represents the network parameters of a text encoding network; Let C represent the category features of the i-th category, and C represent the number of preset categories. For example, when C equals 2, the classification task based on this application belongs to a binary classification task. During the update training phase, the visual adaptation network is used to update unknown features of the current task. and unknown features of historical missions (Where d = 1, 2, ..., t-1, corresponding to historical task 1, historical task 2, ..., historical task t-1 respectively) Feature extraction is performed to obtain the current task perception features. and historical task perception features They are fused separately to obtain the current joint visual features. Combined with historical visual features Referring to steps S207-S209 below, the classification network is used to normalize the similarity z of the current sample image corresponding to each preset category to obtain the final prediction result. Classification networks contain classification functions, such as Softmax.

[0093] In one optional embodiment, the image recognition model can consist of a trained visual language model and a visual adaptation network. The trained visual language model can include an image encoding network and a text encoding network, both of which have fixed network parameters. Therefore, during the continuous learning process based on this image recognition model, only the visual adaptation network needs to be updated and trained, reducing the cost of updating and training the image recognition model. Optionally, depending on the computational power and performance requirements of the actual application, the visual language model can be a network based on a residual network architecture, such as ResNet50, ResNet101, ResNet50x4, and ResNet50x16; or it can be a network based on a Transformer architecture, such as ViT-B / 16 and ViT-B / 32.

[0094] S205: Based on the visual adaptation network to be updated, feature extraction is performed on the unknown features of the current task and the unknown features of the historical task to obtain the current task perception features and the historical task perception features; the unknown features of the historical task are the unknown features of the task in the historical update.

[0095] In this process, feature extraction is performed on unknown features of the current task and unknown features of the historical task based on the visual adaptation network to be updated, to obtain current visual features and historical visual features, which can also be referred to as obtaining current task perception features and historical task perception features.

[0096] To ensure the accuracy of the overall image recognition model while reducing network structure complexity and thus system resource consumption, the visual adaptation network to be updated can be configured as a simple network structure containing a first fully connected layer, an activation function layer, and a second fully connected layer. Therefore, in an exemplary embodiment, step S205 may include: extracting features from unknown features of the current task and unknown features of the historical task based on the first fully connected layer to obtain first unknown features of the current task and first unknown features of the historical task; performing nonlinear mapping processing on the first unknown features of the current task and the first unknown features of the historical task based on the activation layer to obtain second unknown features of the current task and second unknown features of the historical task; and extracting features from the second unknown features of the current task and the second unknown features of the historical task based on the second fully connected layer to obtain current task perceived features and historical task perceived features. Optionally, the activation layer may specifically be a ReLU activation function.

[0097] Among them, visual adaptation networks can also adopt other more complex convolutional network architectures, or a network architecture based on Transformer, but this may increase the number of network parameters of the visual adaptation network, as well as the amount of computation for data processing, and increase the system overhead.

[0098] Several methods can be used to obtain unknown features from historical tasks. One approach is to first acquire the historical sample image set stored in the historical task. By inputting this set into an image encoding network, the corresponding unknown feature set for the historical task can be obtained. Alternatively, during training, all historical sample images from the unknown feature set can be used as training samples input into the image encoding network, or only a subset of these images can be used. However, this method requires further encoding of the sample images, reducing data acquisition efficiency. Therefore, to improve data acquisition efficiency, the historical task's unknown features can also be obtained by encoding the historical images from the historical task itself. If the unknown feature set of the historical task is stored, it can be directly retrieved from the corresponding memory later. Optionally, in order to ensure that the network remembers the knowledge of the old task as much as possible when learning the new task during the network update process, the stored historical sample image set or the data in the historical task unknown feature set contains part of the data in each historical task. For each historical task, the number of stored sample images or task unknown features can be flexibly set as needed; for example, it can be set to 50. Optionally, the data can be stored by sampling in a random and equal probability manner. For details on the specific sampling method of storing data, please refer to steps S215-S219 below.

[0099] S207: Based on the current task-perceived features, the current task-unknown features, and the category features of each preset category corresponding to the current sample image, determine the similarity of the current sample image to each preset category.

[0100] In an exemplary implementation, step S207 may include: fusing current task-perceived features and current task-unknown features to obtain current visual joint features; determining the similarity between the category features of each preset category corresponding to the current sample image and the current visual joint features to obtain the similarity of the current sample image to each preset category. This is achieved by first fusing the current task-perceived features... and unknown features of the current task By fusing the features, we can obtain the current joint visual features. Specifically, it can calculate the category features of each preset category corresponding to the current sample image. Joint features with current vision The cosine distance between them is used to obtain the similarity of the current sample image to each preset category. Among them, z i The similarity score, representing the similarity score for the i-th category, characterizes the degree of matching between the current sample image and the corresponding category hint text. Typically, for a sample image-text pair, the sample image and the category hint text have a high degree of matching, while the sample image has a low degree of matching with the category hint text in other sample image-text pairs. Therefore, the similarity score for each preset category corresponding to the current sample image can effectively indicate the category distribution of the current sample image in the preset category space. Compared with existing technologies that train networks based solely on image modalities and task-unknown features for visual branching, this application incorporates text modalities and integrates multiple task-unknown features, better integrating task-invariant features (such as current task-unknown features). The inclusion of task-aware features (such as current task-aware features) mitigates the catastrophic forgetting of the network model.

[0101] Among them, z i It can be expressed by the following formula (1):

[0102]

[0103] Where sim(·,·) represents the cosine similarity.

[0104] By analyzing the current visual joint features Computation in category space The cosine similarity above can be used to obtain a C-dimensional similarity vector. It represents the class distribution of the current sample image in the preset class space.

[0105] In the above embodiments and This only represents the image representation from the current task t, taking into account the task-related features learned by the visual adaptation network during sequential training, even though task-invariant features are introduced. While this can mitigate catastrophic forgetting to some extent, the nature of task learning still means that catastrophic forgetting can still occur. Therefore, this application trains the visual adaptation network by incorporating features from historical tasks, thereby effectively reducing the network's catastrophic forgetting problem, as detailed in steps S209-S213 below.

[0106] S209: Based on the unknown features of the historical task, the perceived features of the historical task, and the category features of each preset category corresponding to the current sample image, determine the similarity of the unknown features of the historical task to each preset category.

[0107] In an exemplary implementation, step S209 may include: fusing unknown features from the historical task and perceived features from the historical task to obtain historical visual joint features; determining the similarity between the category features of each preset category corresponding to the current sample image and the historical visual joint features to obtain the similarity between the unknown features from the historical task and each preset category. Thus, features from the historical task are introduced into the incremental learning training process of the visual joint features, enabling the network to review old knowledge while learning new knowledge, further mitigating the catastrophic forgetting behavior of visual adaptation networks.

[0108] S211: Determine the loss value based on the similarity of the current sample image to each preset category and the similarity of the unknown features of the historical task to each preset category.

[0109] In one exemplary implementation, see [reference] Figure 4 , Figure 4 This is a flowchart illustrating another incremental learning method for visual joint features provided in an embodiment of this application. Step S211 may include:

[0110] S401: Normalize the similarity of the current sample image to each preset category to obtain the normalization result of the current sample image; the normalization result represents the probability that the current sample image belongs to each preset category.

[0111] In this embodiment of the disclosure, the similarity z of the current sample image to each preset category can be based on a classification function, such as softmax. i The process is performed to obtain the normalized result corresponding to the current sample image, which is the prediction probability obtained by predicting the current sample image.

[0112] Optionally, the predicted probability of the i-th category can be calculated based on the following formula (2):

[0113]

[0114] Here, τ represents the temperature parameter. During model training, the temperature parameter τ can be set to a larger value at the beginning of training. As training progresses, the temperature parameter τ can be reduced, which can also be called cooling operation, so that the model can converge faster.

[0115] Optionally, the temperature parameter τ and the number of training iterations T have the following formula relationship:

[0116]

[0117] Where τ0 represents the initial value of the temperature parameter.

[0118] S403: Determine the current classification loss value based on the difference between the normalization result corresponding to the current sample image and the classification label corresponding to the current sample image.

[0119] The normalization result corresponding to the current sample image represents the probability that the current sample image belongs to each preset category. The classification label corresponding to the current sample image represents the true probability y of the current sample image. Specifically, the current classification loss L can be calculated using the following crossover loss function formula (4). new , means as follows:

[0120]

[0121] Where, N new This indicates the total number of sample images in the current task; y represents the predicted probability of the i-th class in the n-th current sample image; i n This represents the true probability of the i-th class in the n-th current sample image.

[0122] S405: Obtain the historical similarity of each preset category corresponding to the unknown features of the historical task; the historical similarity is the similarity of the unknown features of the historical task in the historical update.

[0123] For example, the methods for obtaining the historical similarity of unknown features of a historical task corresponding to each preset category can include the following: First, the unknown features of the historical task can be obtained from memory. Then, the unknown features of the historical task are input into the corresponding historical visual adaptation network in the historical task to obtain initial historical visual features. These initial historical visual features are the visual features corresponding to the unknown features of the historical task in historical updates. Next, the similarity between the visual joint features fused with the initial historical visual features and the unknown features of the historical task and the historical category features is calculated to obtain the historical similarity of the unknown features of the historical task corresponding to each preset category. The historical category features are the category features corresponding to the historical sample images in historical updates. Alternatively, the initial historical visual features can be obtained from storage. Then, the similarity between the visual joint features fused with the initial historical visual features and the unknown features of the historical task and the historical category features is calculated to obtain the historical similarity of the unknown features of the historical task corresponding to each preset category. To improve data acquisition efficiency and reduce system resource consumption, the historical similarity of the unknown features of the historical task corresponding to each preset category stored in the historical task can be directly obtained from the corresponding memory. Of course, to implement this method, the historical similarity of the unknown features of the historical task corresponding to each preset category in the historical task needs to be stored in memory during the historical task process.

[0124] S407: Determine the historical classification loss value based on the difference between the similarity of each preset category corresponding to the unknown features of the historical task and the historical similarity.

[0125] In one exemplary implementation, please refer to Figure 4 Step S407 may include:

[0126] S4071: Determine the first historical classification loss value based on the classification labels corresponding to the unknown features of the historical task and the historical predicted classification results.

[0127] The first historical classification loss value L1 can be represented by the following formula (5):

[0128]

[0129] Where, N new This represents the number of unknown features in the historical task, which is also the number of historical sample images; y' i m This represents the classification label (true probability) of the i-th category of the m-th historical sample image; This represents the historical predicted classification result (historical predicted probability) for the i-th category of the m-th historical sample image.

[0130] S4073: Determine the second historical classification loss value based on the difference between the similarity of each preset category corresponding to the unknown features of the historical task and the historical similarity.

[0131] In an exemplary implementation, step S4073 may include: normalizing the similarity of each preset category corresponding to the unknown features of the historical task to obtain the prediction result corresponding to the unknown features of the historical task; determining the difference between the historical prediction result corresponding to the unknown features of the historical task and the prediction result corresponding to the unknown features of the historical task to obtain the prediction result difference value of each preset category corresponding to the historical task perception feature; the prediction result corresponding to the unknown features of the historical task is the prediction result corresponding to the unknown features of the historical task in the historical update; statistically processing the square of the prediction result difference value corresponding to each preset category of the unknown features of the historical task to obtain the target prediction result difference value of the unknown features of the historical task; and determining the second historical classification loss value based on the target prediction result difference value of the unknown features of the historical task.

[0132] For example, the second historical classification loss value L2 can be expressed by the following formula (6):

[0133]

[0134] Among them, y' i m This represents the prediction result of the i-th category corresponding to the unknown feature of the m-th historical task; This represents the historical prediction results for the i-th category corresponding to the unknown features of the m-th historical task; This represents the difference in prediction results for the i-th category corresponding to the perceived features of the historical task.

[0135] Since the second historical classification loss value in the above embodiment is calculated as the expected value (i.e., MSE value) of the difference between the prediction result corresponding to the unknown feature of the historical task and the historical prediction result corresponding to the unknown feature of the historical task, it is necessary to first normalize the similarity of each preset category corresponding to the unknown feature of the historical task, which increases the computational complexity and system resource overhead. Therefore, in order to simplify the step of calculating the second historical classification loss and reduce system resource overhead, in another exemplary embodiment, step S4073 may include: determining the difference between the similarity of each preset category corresponding to the unknown feature of the historical task and the historical similarity of each preset category corresponding to the unknown feature of the historical task, to obtain the similarity difference value of each preset category corresponding to the perceived feature of the historical task; performing statistical processing on the square of the similarity difference value of each preset category corresponding to the unknown feature of the historical task to obtain the target similarity difference value of the unknown feature of the historical task; and determining the second historical classification loss value based on the target similarity difference value of the unknown feature of the historical task.

[0136] For example, the second historical classification loss value L2 can be expressed by the following formula (7):

[0137]

[0138] Among them, z' i m This represents the similarity between the unknown features of the m-th historical task and the i-th category; This represents the historical similarity between the unknown features of the m-th historical task and the i-th category; This represents the similarity difference value between the perceived features of the historical task and the i-th category.

[0139] S4075: Determine the sum of the first historical classification loss value and the second historical classification loss value to obtain the historical classification loss value.

[0140] Historical classification loss value L old It can be expressed based on the following formula (8):

[0141]

[0142] S409: Determine the loss value based on the current classification loss value and the historical classification loss values.

[0143] In the above implementation, the historical classification loss value can be determined by calculating the cross-entropy loss value of the classification label corresponding to the unknown features of the historical task and the historical prediction classification result, as well as the expected value (i.e., MSE value) of the difference between the similarity of the unknown features of the historical task corresponding to each preset category and the historical similarity of the unknown features of the historical task corresponding to each preset category. The sum of the two is then determined as the historical classification loss value. Therefore, the smaller the historical classification loss value, the higher the accuracy of the network model. Since the classification loss in the historical update and the difference loss between the historical similarity and the similarity of the unknown features of the historical task are fused, the obtained historical classification loss can better describe the network model's learning of the historical task, improving the robustness of the network model after training. Furthermore, the loss value also includes the current classification loss, which can describe the network model's learning of the current task, improving the plasticity of the network model and thus mitigating the problem of catastrophic forgetting of the network model.

[0144] S213: Based on the loss value, update the network parameters of the visual adaptation network to be updated until the preset update termination condition is met, and obtain the updated visual adaptation network.

[0145] Throughout the training process, the network parameters (such as weights and biases) of the visual adaptation network can be optimized by calculating the loss based on the current sample image of the current task and the features corresponding to the unknown features of the historical tasks.

[0146] The preset update termination condition can be a condition that satisfies a preset number of iterations, a preset loss value, or a combination of both.

[0147] In one exemplary implementation, see [reference] Figure 4 The methods also include:

[0148] S215: Store the unknown features of the current task into the unknown feature set of the target task.

[0149] In the current task, the current training samples may include multiple current sample images. When the model processes multiple current sample images, they can be processed sequentially in a preset order. For example, the current sample images can be encoded to obtain unknown features of the current task. Each current sample image corresponds to one unknown feature of the current task. The multiple unknown features of the current task output by the model can be stored sequentially in the unknown feature set of the target task. Optionally, the unknown feature set of the target task can be stored in the server's memory or on other terminals or servers.

[0150] S217: If the number of unknown features of the current task in the unknown feature set of the target task is equal to the number of unknown features of the preset task, and a newly generated unknown feature of the current task is obtained, update the unknown feature set of the target task based on the newly generated unknown feature of the current task to obtain the sampled feature set; the number of unknown features of the current task in the sampled feature set is equal to the number of unknown features of the preset task.

[0151] In an exemplary implementation, step S217 may include: storing the newly generated current task unknown features into the target task unknown feature set to obtain a quasi-target task unknown feature set; randomly sampling a preset number of current task unknown features from the quasi-target task unknown feature set to obtain a sampled feature set. In this way, the target task unknown feature set stores at most a preset number plus one of the task unknown features, and always ensures that the final fixed-stored sampled feature set contains only the preset number of task unknown features. Compared to saving all task unknown features in the current task and then resampling, this reduces the storage space requirements and improves system data processing efficiency.

[0152] To better illustrate the beneficial effects of this embodiment, a specific example is given below:

[0153] The preset number of unknown features for the current task is set to 50. At the beginning of the current task, the number of unknown features in the target task unknown feature set is 0. As the task progresses and the current sample image is input, the network model outputs more and more unknown features for the current task. Each time the model outputs an unknown feature, it is stored in the target task unknown feature set until the number of unknown features in the target task unknown feature set equals 50. If the 51st unknown feature is received at this point, it can be stored in the target task unknown feature set, resulting in a quasi-target task unknown feature set containing 51 features. Then, 50 unknown features are randomly sampled from this quasi-target task unknown feature set to obtain a sampled feature set. This sampled feature set can be used as the target task unknown feature set. When the 52nd unknown feature is received, the process of storing it in the target task unknown feature set and randomly sampling it is repeated until no more new unknown features are received, resulting in the target sampled feature set. This allows for equal-probability sampling of uncertain data streams with low system resource consumption.

[0154] Additionally, if the number of currently unknown features in the target task's unknown feature set is less than the preset number of currently unknown features, and newly generated currently unknown features are obtained, these newly generated currently unknown features are stored in the target task's unknown feature set. However, if the number of currently unknown features in the target task's unknown feature set is less than or equal to the preset number of currently unknown features, and no newly generated currently unknown features are obtained, it may be that the training samples input to the model have been completed but do not meet the requirements of the preset task's unknown feature data. In this case, you can choose to stop updating the target sample image set or increase the number of training samples.

[0155] S219: After the visual adaptation network has been updated, update the unknown feature set of the historical task based on the sampled feature set.

[0156] In this embodiment, the unknown feature of a historical task is any one of the unknown features in the set of unknown features of historical tasks. When the visual adaptation network is updated, the sampled feature set can be stored in the set of unknown features of historical tasks, thereby completing the update of the set of unknown features of historical tasks. For each historical task, the method of obtaining its corresponding sampled feature set is described in steps S215-S217 above. That is, as the number of tasks increases, the amount of data in the set of unknown features of historical tasks also increases. For example, if the historical tasks include a first historical task A and a second historical task B, then the set of unknown features of historical tasks includes the unknown features of tasks collected in the first historical task A and the unknown features of tasks collected in the second historical task B. The number of unknown features corresponding to each task can be a fixed number. For example, if the first historical task A and the second historical task B each collect 50 unknown features, then the set of unknown features of historical tasks contains 100 unknown features. Alternatively, it can be set differently according to actual needs. For example, the number of unknown features collected in the first historical task A can be 50, and the number of unknown features collected in the second historical task B can be 80. There is no limitation here.

[0157] The historical sample image set obtained in the above manner not only contains the unknown features of each historical task, but also samples the data flowing out of the current task in real time using the above random equal probability sampling method for each task. This ensures that the target task unknown feature set stores at most a preset number plus 1 of the unknown features, and always guarantees that the final fixed storage sampled feature set contains only the preset number of unknown features. Compared with the method of saving all the unknown features of the current task and then sampling, this reduces the storage space requirements and helps improve the system's data processing efficiency.

[0158] To better demonstrate the beneficial effects of the above implementation methods, the following example illustrates a method for calculating the similarity between unknown features of a task and their corresponding category features based on different unknown features of the task, resulting in different similarities for the final test images. In this experiment, the CIFAR-100 dataset was used, with ResNet50 as the base network and a two-layer perceptron network structure (e.g., a fully connected layer-activation layer-fully connected layer structure) for the visual adaptation network. The number of past task replay samples (i.e., the number of historical sample images) was set to 0, 50, 200, and 500, respectively, to train the image recognition model, resulting in four trained image recognition models. The test images were then input into these four trained image recognition models, and the corresponding experimental results are shown in Table 1 below.

[0159]

[0160] In Table 1, 0, 50, 200, and 500 in the first row represent the number of historical sample images; z in the first column... a and z b These represent the two similarity values ​​corresponding to the test images, where z a The calculation formula (9) can be expressed as follows:

[0161]

[0162] Z b The calculation formula for z can be obtained based on the above formula (1); therefore, z a It is an unknown feature of the task. Category features of each preset category corresponding to the sample image The cosine similarity between them; and Z b It is a task-aware feature and unknown features of the task Combined visual features Category features of each preset category corresponding to the sample image The cosine similarity between them; it can be seen that, based on the proposed scheme, unknown features of the task are... and task-aware features The cosine similarity obtained by the visual-language joint learning method significantly exceeds that obtained by using only task perception. The visual language learning method presented in this paper demonstrates that updating and training the network model based on this scheme yields more accurate prediction results.

[0163] As can be seen from the above technical solutions of the embodiments of this application, the embodiments of this application acquire two modal data: the current sample image and the category features corresponding to each preset category of the current sample image. The category features of each preset category are obtained by inputting the category prompt text corresponding to each preset category into a text encoding network. The current sample image is first encoded using an image encoding network to obtain unknown features for the current task. Subsequently, a visual adaptation network to be updated is used to extract features from the unknown features for the current task and the unknown features for the historical task, respectively, to obtain the perceived features for the current task and the perceived features for the historical task. The similarity of the current sample image to each preset category and the similarity of the unknown features for each preset category are calculated and determined. The similarity of each preset category is used to determine the loss value. Based on the loss value, the network parameters of the visual adaptation network to be updated are updated until the preset update termination condition is met, resulting in the updated visual adaptation network. Since the network input includes both image and text modal data of the current task, it takes into account the semantic information contained in the natural language of the category label text, making better use of the contextual relevance in the text space and providing more effective information. It also includes task-unknown features from historical tasks, i.e., it includes historical data features. This allows the visual adaptation network to improve its plasticity while maintaining stability during continuous learning and updating, reducing the catastrophic forgetting of the visual adaptation network in visual cognitive tasks.

[0164] Corresponding to the incremental learning methods for visual joint features provided in the above embodiments, this application also provides an incremental learning device for visual joint features. Since the incremental learning device for visual joint features provided in this application corresponds to the incremental learning methods for visual joint features provided in the above embodiments, the implementation methods of the aforementioned incremental learning methods for visual joint features are also applicable to the incremental learning device for visual joint features provided in this embodiment, and will not be described in detail in this disclosure.

[0165] Please see Figure 5 The diagram shown is a structural schematic of an incremental learning device for visual joint features provided in an embodiment of this application. The device has the function of implementing the incremental learning method for visual joint features in the above-described method embodiment. The function can be implemented by hardware or by hardware executing corresponding software.

[0166] like Figure 5 As shown, the incremental learning device 500 for visual joint features may include:

[0167] The acquisition module 501 is used to acquire the current sample image and the category features of each preset category corresponding to the current sample image;

[0168] Encoding module 503 is used to encode the current sample image to obtain unknown features for the current task;

[0169] The feature extraction module 505 is used to extract features from the unknown features of the current task and the unknown features of the historical task based on the visual adaptation network to be updated, to obtain the current task perception features and the historical task perception features; the unknown features of the historical task are the unknown features of the task in the historical update.

[0170] The first similarity determination module 507 is used to determine the similarity of the current sample image to each preset category based on the current task perceived features, the current task unknown features, and the category features of each preset category corresponding to the current sample image.

[0171] The second similarity determination module 509 is used to determine the similarity between the unknown features of the historical task and each preset category based on the unknown features of the historical task, the perceived features of the historical task, and the category features of each preset category corresponding to the current sample image.

[0172] The loss value determination module 511 is used to determine the loss value based on the similarity of the current sample image to each preset category and the similarity of the unknown features of the historical task to each preset category.

[0173] The update module 513 is used to update the network parameters of the visual adaptation network to be updated based on the loss value until the preset update end condition is met, so as to obtain the updated visual adaptation network.

[0174] In one exemplary implementation, the first similarity determination module includes:

[0175] The first fusion module is used to fuse the current task's perceived features and the current task's unknown features to obtain the current joint visual features;

[0176] The third similarity determination module is used to determine the similarity between the category features of each preset category corresponding to the current sample image and the current visual joint features, so as to obtain the similarity of the current sample image to each preset category.

[0177] The second similarity determination module includes:

[0178] The second fusion module is used to fuse unknown features of historical tasks and perceived features of historical tasks to obtain joint historical visual features.

[0179] The fourth similarity determination module is used to determine the similarity between the category features of each preset category corresponding to the current sample image and the historical visual joint features, so as to obtain the similarity between the unknown features of the historical task and each preset category.

[0180] In one exemplary implementation, the loss value determination module includes:

[0181] The normalization processing module is used to normalize the similarity of the current sample image to each preset category, and obtain the normalization result corresponding to the current sample image; the normalization result represents the probability that the current sample image belongs to each preset category;

[0182] The first loss value determination module is used to determine the current classification loss value based on the difference between the normalization result corresponding to the current sample image and the classification label corresponding to the current sample image.

[0183] The first acquisition module is used to acquire the historical similarity of unknown features of historical tasks for each preset category; the historical similarity is the similarity of unknown features of historical tasks in historical updates;

[0184] The second loss value determination module is used to determine the historical classification loss value based on the difference between the similarity of each preset category corresponding to the unknown features of the historical task and the historical similarity.

[0185] The third loss value determination module is used to determine the loss value based on the current classification loss value and the historical classification loss values.

[0186] In one exemplary implementation, the second loss value determination module includes:

[0187] The fourth loss value determination module is used to determine the first historical classification loss value based on the classification labels corresponding to the unknown features of the historical task and the historical prediction classification results.

[0188] The fifth loss value determination module is used to determine the second historical classification loss value based on the difference between the similarity of each preset category corresponding to the unknown features of the historical task and the historical similarity.

[0189] The sixth loss value determination module is used to determine the sum of the first historical classification loss value and the second historical classification loss value to obtain the historical classification loss value.

[0190] In one exemplary embodiment, the fifth loss value determination module includes:

[0191] The difference determination module is used to determine the difference between the similarity of the unknown features of the historical task corresponding to each preset category and the historical similarity of the unknown features of the historical task corresponding to each preset category, so as to obtain the similarity difference value of the perceived features of the historical task corresponding to each preset category.

[0192] The statistics module is used to perform statistical processing on the square of the similarity difference value of each preset category corresponding to the unknown features of the historical task, so as to obtain the target similarity difference value of the unknown features of the historical task.

[0193] The seventh loss value determination module is used to determine the second historical classification loss value based on the target similarity difference value of unknown features in the historical task.

[0194] In one exemplary embodiment, the historical task unknown feature is any one of the task unknown features in the set of historical task unknown features; the apparatus further includes:

[0195] The first storage module is used to store the unknown features of the current task into the unknown feature set of the target task;

[0196] The first update module is used to update the target task unknown feature set based on the newly generated current task unknown features when the number of current task unknown features in the target task unknown feature set is equal to the preset number of task unknown features and new current task unknown features are obtained, so as to obtain a sampled feature set; the number of current task unknown features in the sampled feature set is equal to the preset number of task unknown features.

[0197] The second update module is used to update the unknown feature set of the historical task based on the sampled feature set after the visual adaptation network has been updated.

[0198] In one exemplary implementation, the first update module includes:

[0199] The second storage module is used to store the newly generated unknown features of the current task into the unknown feature set of the target task, so as to obtain the quasi-unknown feature set of the target task.

[0200] The sampling module is used to randomly sample a preset number of unknown features of the current task from the unknown feature set of the quasi-target task to obtain the sampled feature set.

[0201] It should be noted that the apparatus provided in the above embodiments is only illustrated by the division of the above functional modules when implementing its functions. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above. In addition, the apparatus and method embodiments provided in the above embodiments belong to the same concept, and the specific implementation process can be found in the method embodiments, which will not be repeated here.

[0202] This application provides an electronic device including a processor and a memory. The memory stores at least one instruction or at least one program, which is loaded and executed by the processor to implement any of the incremental learning methods for visual joint features provided in the above method embodiments.

[0203] Memory can be used to store software programs and modules. The processor executes various functional applications and data processing by running the software programs and modules stored in the memory. Memory can primarily include a program storage area and a data storage area. The program storage area can store the operating system, application programs required for the functions, etc.; the data storage area can store data created based on the use of the device, etc. Furthermore, memory can include high-speed random access memory, and can also include non-volatile memory, such as at least one disk storage device, flash memory device, or other volatile solid-state storage device. Accordingly, memory can also include a memory controller to provide the processor with access to the memory.

[0204] The method embodiments provided in this application can be executed in a computer terminal, server or similar computing device, that is, the above-mentioned electronic device may include a computer terminal, server or similar computing device. Figure 6 This is a hardware structure block diagram of an electronic device that runs an incremental learning method for visual joint features, as provided in an embodiment of this application. Figure 6 As shown, the server 600 can vary significantly due to different configurations or performance. It may include one or more Central Processing Units (CPUs) 610 (CPUs 610 may include, but are not limited to, microprocessors (MCUs) or programmable logic devices (FPGAs), a memory 630 for storing data, and one or more storage media 620 (e.g., one or more mass storage devices) for storing application programs 623 or data 622. The memory 630 and storage media 620 may be temporary or persistent storage. The program stored in the storage media 620 may include one or more modules, each module may include a series of instruction operations on the server. Furthermore, the CPU 610 may be configured to communicate with the storage media 620 and execute the series of instruction operations stored in the storage media 620 on the server 600. Server 600 may also include one or more power supplies 660, one or more wired or wireless network interfaces 650, one or more input / output interfaces 640, and / or one or more operating systems 621, such as Windows Server™, Mac OS X™, Unix™, Linux™, FreeBSD™, etc.

[0205] The input / output interface 640 can be used to receive or send data via a network. Specific examples of the network described above may include a wireless network provided by the communication provider of server 600. In one example, the input / output interface 640 includes a network interface controller (NIC), which can connect to other network devices via a base station to communicate with the Internet. In another example, the input / output interface 640 may be a radio frequency (RF) module for wireless communication with the Internet.

[0206] Those skilled in the art will understand that Figure 6 The structure shown is for illustrative purposes only and does not limit the structure of the aforementioned electronic device. For example, server 600 may also include... Figure 6 The more or fewer components shown, or having the same Figure 6 The different configurations shown.

[0207] Embodiments of this application also provide a computer-readable storage medium, which can be disposed in an electronic device to store at least one instruction or at least one program related to implementing an incremental learning method for visual joint features. The at least one instruction or the at least one program is loaded and executed by the processor to implement any of the incremental learning methods for visual joint features provided in the above method embodiments.

[0208] Embodiments of this application also provide a computer program product or computer program, which includes computer instructions stored in a computer-readable storage medium. A processor of an electronic device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the electronic device to perform any of the incremental learning methods for visual joint features provided in the above-described method embodiments.

[0209] Optionally, in this embodiment of the disclosure, the storage medium may include, but is not limited to, various media capable of storing program code, such as USB flash drives, read-only memory (ROM), random access memory (RAM), portable hard drives, magnetic disks, or optical disks.

[0210] It should be noted that the order of the embodiments described above is merely for descriptive purposes and does not represent the superiority or inferiority of the embodiments. Furthermore, specific embodiments have been described above. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps described in the claims can be performed in a different order than that shown in the embodiments and still achieve the desired result. Additionally, the processes depicted in the drawings do not necessarily require a specific or sequential order to achieve the desired result. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0211] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, the apparatus embodiments are basically similar to the method embodiments, so the description is relatively simple; relevant parts can be referred to the descriptions of the method embodiments.

[0212] Those skilled in the art will understand that all or part of the steps of the above embodiments can be implemented by hardware or by a program instructing related hardware. The program can be stored in a computer-readable storage medium, such as a read-only memory, a disk, or an optical disk.

[0213] The above description is only a preferred embodiment of this application and is not intended to limit this application. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the protection scope of this application.

Claims

1. An incremental learning method for visual joint features, characterized in that, The method includes: Obtain the current sample image and the category features of each preset category corresponding to the current sample image; the category features of each preset category corresponding to the current sample image are obtained by encoding the category hint text of the preset image category into a preset category space; The current sample image is encoded to obtain unknown features for the current task; Based on the visual adaptation network to be updated, feature extraction is performed on the unknown features of the current task and the unknown features of the historical task to obtain the current task perception features and the historical task perception features; the unknown features of the historical task are the unknown features of the task in the historical update. Based on the current visual joint features obtained by fusing the current task perceived features and the current task unknown features, and the category features of each preset category corresponding to the current sample image, the similarity of the current sample image to each preset category is determined; Based on the historical visual joint features fused with the historical task unknown features and the historical task perceived features, and the category features of each preset category corresponding to the current sample image, the similarity of the historical task unknown features to each preset category is determined. The loss value is determined based on the similarity of the current sample image to each preset category and the similarity of the unknown features of the historical task to each preset category. Based on the loss value, the network parameters of the visual adaptation network to be updated are updated until the preset update termination condition is met, and the updated visual adaptation network is obtained.

2. The method according to claim 1, characterized in that, The step of determining the similarity of the current sample image to each preset category based on the current visual joint features fused from the current task perceived features and the current task unknown features, and the category features of each preset category corresponding to the current sample image, includes: By fusing the current task-perceived features and the current task-unknown features, the current visual joint features are obtained; Determine the similarity between the category features of each preset category corresponding to the current sample image and the current visual joint features to obtain the similarity of the current sample image to each preset category; The determination of the similarity between the unknown features of the historical task and each preset category based on the historical visual joint features fused from the historical task unknown features and the historical task perceived features, and the category features of each preset category corresponding to the current sample image, includes: By fusing the unknown features of the historical task with the perceived features of the historical task, a joint historical visual feature is obtained; Determine the similarity between the category features of each preset category corresponding to the current sample image and the historical visual joint features to obtain the similarity between the unknown features of the historical task and each preset category.

3. The method according to claim 1, characterized in that, The step of determining the loss value based on the similarity of the current sample image to each preset category and the similarity of the unknown features of the historical task to each preset category includes: The similarity of the current sample image to each preset category is normalized to obtain the normalized result corresponding to the current sample image; the normalized result represents the probability that the current sample image belongs to each preset category; Based on the difference between the normalization result corresponding to the current sample image and the classification label corresponding to the current sample image, the current classification loss value is determined. Obtain the historical similarity of the unknown features of the historical task for each preset category; the historical similarity is the similarity of the unknown features of the historical task in historical updates; Based on the difference between the similarity of the unknown features of the historical task to each preset category and the historical similarity, the historical classification loss value is determined; The loss value is determined based on the current classification loss value and the historical classification loss value.

4. The method according to claim 3, characterized in that, The determination of historical classification loss based on the difference between the similarity of each preset category corresponding to the unknown features of the historical task and historical similarity includes: Based on the classification labels corresponding to the unknown features of the historical task and the historical prediction classification results, the first historical classification loss value is determined; Based on the difference between the similarity of the unknown features of the historical task corresponding to each preset category and the historical similarity, the second historical classification loss value is determined; The sum of the first historical classification loss value and the second historical classification loss value is determined to obtain the historical classification loss value.

5. The method according to claim 4, characterized in that, The step of determining the second historical classification loss value based on the difference between the similarity of the unknown features of the historical task corresponding to each preset category and the historical similarity includes: The difference between the similarity of the unknown features of the historical task corresponding to each preset category and the historical similarity of the unknown features of the historical task corresponding to each preset category is determined to obtain the similarity difference value of the perceived features of the historical task corresponding to each preset category. The square of the similarity difference value corresponding to each preset category of the unknown features of the historical task is statistically processed to obtain the target similarity difference value of the unknown features of the historical task. The second historical classification loss value is determined based on the target similarity difference value of the unknown features of the historical task.

6. The method according to any one of claims 1-5, characterized in that, The unknown feature of the historical task is any one of the unknown features in the set of unknown features of historical tasks; the method further includes: Store the unknown features of the current task into the unknown feature set of the target task; If the number of unknown features of the current task in the target task unknown feature set is equal to the preset number of unknown features of the task, and a newly generated unknown feature of the current task is obtained, the target task unknown feature set is updated based on the newly generated unknown feature of the current task to obtain a sampled feature set; the number of unknown features of the current task in the sampled feature set is equal to the preset number of unknown features of the task. Once the visual adaptation network update is complete, the historical task unknown feature set is updated based on the sampled feature set.

7. The method according to claim 6, characterized in that, The target task unknown feature set is updated based on the newly generated current task unknown features to obtain a sampled feature set, including: The newly generated unknown features of the current task are stored in the unknown feature set of the target task to obtain the quasi-unknown feature set of the target task; The sampled feature set is obtained by randomly sampling the current task unknown features of the preset number of task unknown features from the quasi-target task unknown feature set.

8. An incremental learning device for visual joint features, characterized in that, The device includes: The acquisition module is used to acquire the current sample image and the category features of the current sample image for each preset category; the category features of the current sample image for each preset category are obtained by encoding the category hint text of the preset image category into a preset category space; The encoding module is used to encode the current sample image to obtain unknown features for the current task; The feature extraction module is used to extract features from the unknown features of the current task and the unknown features of the historical task based on the visual adaptation network to be updated, respectively, to obtain the current task perception features and the historical task perception features; the unknown features of the historical task are the unknown features of the task in the historical update. The first similarity determination module is used to determine the similarity of the current sample image to each preset category based on the current visual joint features obtained by fusing the current task perceived features and the current task unknown features and the category features of each preset category corresponding to the current sample image; The second similarity determination module is used to determine the similarity between the unknown features of the historical task and each preset category based on the historical visual joint features fused with the unknown features of the historical task and the perceived features of the historical task and the category features of each preset category corresponding to the current sample image. The loss value determination module is used to determine the loss value based on the similarity of the current sample image to each preset category and the similarity of the unknown features of the historical task to each preset category. The update module is used to update the network parameters of the visual adaptation network to be updated based on the loss value until a preset update termination condition is met, thereby obtaining the updated visual adaptation network.

9. An electronic device, characterized in that, It includes a processor and a memory, wherein the memory stores at least one instruction or at least one program, the at least one instruction or the at least one program being loaded and executed by the processor to implement the incremental learning method of visual joint features as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores at least one instruction or at least one program, which is loaded and executed by a processor to implement the incremental learning method for visual joint features as described in any one of claims 1 to 7.

11. A computer program product, characterized in that, The computer program product includes at least one instruction or at least one program segment, which is loaded and executed by a processor to implement the incremental learning method for visual joint features as described in any one of claims 1 to 7.

Images