Continual learning method for training computer vision models

Abstract

Computer vision model learning software (3) being loadable in, and executable by, electronic processing resources (2) and designed to cause, when executed, the electronic processing resources (2) to become configured to implement a computer vision model comprising a main neural network (6) trained, on a current image dataset, to perform a computer vision task. The computer vision model learning software (3) is further designed to cause, when executed, the electronic processing resources (2) to become configured to receive one or more images to be analysed (4) and compute a computer vision task's output (9) by performing said computer vision task on the received images to be analysed (4) by means of the computer vision model. In addition, the computer vision model learning software (3) is further designed to cause, when executed, the electronic processing resources (2) to become configured to perform update operations to update the main neural network's (6) weights. Said update operations comprise forming an updated image dataset, different from the current image dataset, based on the received images to be analysed (4) and based on the computer vision task's output (9). In addition, said update operations comprise training an auxiliary twin neural network (7), being different from the main neural network (6) but having the same architecture of the main neural network (6), by adjusting the auxiliary twin neural network's (7) weights based on the updated image dataset to perform said computer vision task. Furthermore, the update operations comprise substituting the weights of the main neural network (6) with the weights of the auxiliary twin neural network (7) to update the main neural network (6) without interrupting its operation.

Description

[0001] "CONTINUAL LEARNING METHOD FOR TRAINING COMPUTER VISION MODELS"

[0002] CROSS-REFERENCE TO RELATED APPLICATIONS

[0003] This Patent Applications claims priority from European Patent Application No. 24177895.0 filed on May 24, 2024 and Italian Patent Application No. 102024000021954 filed on October 3, 2024, the entire disclosure of which is incorporated herein by reference.

[0004] TECHNICAL FIELD OF THE INVENTION

[0005] The invention relates in general to the computer vision field. In particular, the present invention relates to a method for training an artificial neural network to perform a computer vision task.

[0006] STATE OF THE RT

[0007] As is known, in the domain of computer vision, in particular of visual object detect ion / recognit ion, the common procedure is to train one or more neural networks on objects and / or domain-specific datasets. In detail, a neural network can be trained on a training dataset to become configured to perform a predefined computer vision task by adjusting its weights by executing a training method on said training dataset.

[0008] However, neural network weights are fixed and do not change during inference, resulting in poor adaptation capability in setups different from the training dataset on which said neural network was trained. This limitation could potentially lead to issues with the introduction of new class instances or in scenarios where variations occur during the inference phase, such as changes in the background of the images or the lighting conditions in the environment.

[0009] As is known, various solutions have been proposed to address this problem; examples include domain adaptation and open-world recognition techniques. Anyway, the proposed solutions are too general for a specific operation or require a hand-crafted network re-training for any change in the application setup, environment or system dynamics.

[0010] Hence, as is well known, the most used solution for adapting a neural network on newly acquired data would require a temporary stop of the neural network operation, in particular inference, to allow a human operator to manually collect, inspect and label the instances for a new dataset and then finetune the neural network based on said new instances .

[0011] Additionally, US 2019 / 303759 Al describes that physical sensor data can be generated by a vehicle operating in a real-world environment. This data is then used to train deep neural networks (DNNs) . After training, said DNNs may be tested in a simulated environment, in some examples using hardware configured for installation in a vehicle to execute an autonomous driving software stack, to control a virtual vehicle in the simulated environment or to otherwise test, verify, or validate the outputs of the DNNs. Before the DNNs use the data, virtual sensor data created by virtual sensors in the simulated environment may be encoded to match the format of the physical sensor data generated by the vehicle.

[0012] As is further known, a scientific article (KAI WANG ET AL: "Positive Pair Distillation Considered Harmful: Continual Meta Metric Learning for Lifelong Object Re- Identification", ARXIV.ORG, CORNELL UNIVERSITY LIBRARY, 2022-10-04) discusses the challenge of lifelong object reidentification, which involves incrementally learning from a stream of re-identification tasks to develop a representation that generalizes to previously unseen tasks and identities. The authors propose a method called Continual Meta Metric Learning to address this problem by using knowledge distillation and exploring the roles of positive and negative pairs. They introduce a novel distillation scheme called Distillation without Positive Pairs (DwoPP) , which removes positive pairs from the distillation process to enhance model updates and mitigate the antagonistic relationship between distillation and metric losses.

[0013] In addition, it is known that WO 2022 / 109583 Al discloses systems and methods for selecting data for training a machine learning model using active learning. These described methods include receiving a plurality of unlabelled sensor data logs corresponding to the surroundings of an autonomous vehicle. Said methods further include identifying one or more trends associated with a training dataset comprising a plurality of labelled data logs. A function is then determined for assigning an importance score to each of the unlabelled sensor data logs using the identified trends. This function is used to assign the importance score to each of the unlabelled sensor data logs. The methods also include selecting a subset of the sensor data logs that have an importance score greater than a threshold. This subset is then used for further training the machine learning model, which was initially trained using the training dataset, to generate an updated model. As is further known, GB 2 627 869 A discloses a method for classifying an image using a machine learning (ML) model. Said method comprising extracting features from an input image to generate a feature vector, and creating a statistical vector, comprising at least one statistical moment which is extracted from the feature vector, using a pooling module. In addition, the method further comprises a phase of concatenating the statistical vector to the feature vector. Furthermore, said method comprises classifying the input image, by: comparing the concatenated vector to prototype reference vectors which are stored on a user device, identifying the reference vector which is most similar to the concatenated vector, and assigning the classification of the identified prototype vector to the input image .

[0014] SUBJECT-MATTER AND SUMMARY OF THE INVENTION

[0015] The Applicant notes that known solutions can be improved. The Applicant observes that the known solutions allow a neural network to adapt to newly acquired data, enabling it to perform the computer vision task on new classes or to configure itself to perform a related, but different, computer vision task. However, the Applicant further observes that the known solutions exhibit several drawbacks. In particular, according to the known solutions, during the training of the neural network for the computer vision task, the network becomes unavailable for inference or any other operations except for the training itself. Consequently, the main application must wait for the manual fine-tuning of the neural network, leading to a loss of performance in the overall process and causing the unavailability of the neural network's service or operation during the training phase, like a maintenance work. This often requires a significant amount of time and sometimes also a considerable amount of money. In addition, the known solutions need expert operators who can understand when the application is performing poorly. Furthermore, according to the known solutions, expert operators manually collect the dataset and re-train the network, causing the entire service to be blocked. This blocking refinement results in a waste of time and money for commercial facilities that provide models for performing computer vision tasks as services. Hence, the Applicant notes that the known solutions fail to enable the fully autonomous continuous training of a neural network.

[0016] Furthermore, the known fine-tuning techniques, used to adapt neural networks to new setups for a computer vision task, are subject to the well-known issue called catastrophic forgetting. In detail, considering only new data could quickly lead to overfitting of the neural network in favour of the new data compared to the pre-trained, or initial, data. Consequently, relying solely on new inputs can negatively affect the existing knowledge of the neural network. Hypothetically, combining old and new data for retraining the neural network could enhance performance without compromising the detection and recognition capabilities of the old data; however, this solution is often unfeasible because old data may not be available to the end user, and combining a large amount of data can lead to memory capacity issues, increased training time, or the need for additional hardware capacity. Hence, the object of the invention is to provide a computer vision model learning software which at least partly solves the problems of the known prior art.

[0017] In detail, the provided solution is a computer vision model learning software that allows to autonomously, and almost instantaneously, update a neural network enabling it to adapt to a new setup; more in detail, to perform the computer vision task on new classes of instances. In detail, the provided solution allows to ensure that a neural network is autonomously adapted and updated to enhance and adjust its detection or recognition capabilities in real-time, without the need to stop the inference of the neural network while updating it. More in detail, the provided solution enables the neural network to be updated without the need for an expert human operator to manage the entire process.

[0018] Furthermore, the provided solution employs an unsupervised continual learning approach combined with an intelligent image pool collection module to effectively mitigate the issue of catastrophic forgetting. In addition, the capability to autonomously collect a new dataset for training the neural network enables a fully autonomous continuous training .

[0019] According to the invention, there is provided a computer vision model learning software and a computer vision model learning system as claimed in the appended claims.

[0020] According to the invention, there is provided a computer vision model learning software being loadable in, and executable by, electronic processing resources and designed to cause, when executed, the electronic processing resources to become configured to implement a computer vision model comprising a main neural network trained, on a current image dataset, to perform a computer vision task. The computer vision model learning software is further designed to cause, when executed, the electronic processing resources to become configured to receive one or more images to be analysed and compute a computer vision task' s output by performing said computer vision task on the received images to be analysed by means of the computer vision model. In addition, the computer vision model learning software is further designed to cause, when executed, the electronic processing resources to become configured to perform update operations to update the main neural network' s weights. Said update operations comprise forming an updated image dataset, different from the current image dataset, based on the received images to be analysed and based on the computer vision task' s output. In addition, said update operations comprise training an auxiliary twin neural network, being different from the main neural network but having the same architecture of the main neural network, by adjusting the auxiliary twin neural network' s weights based on the updated image dataset to perform said computer vision task. Furthermore, the update operations comprise substituting the weights of the main neural network with the weights of the auxiliary twin neural network to update the main neural network without interrupting its operation.

[0021] In particular, there is provided a computer vision model learning software designed to cause, when executed, the electronic processing resources to become configured to determine that a model update condition is, or is not, satisfied based on the computer vision task' s output; and to perform the update operations, to update the main neural network' s weights, when the model update condition is determined to be satisfied.

[0022] In particular, the computer vision task' s output 9 comprises data indicative of the computer vision model' s performance on said computer vision task performed on the received images to be analysed. In detail, the computer vision model learning software is designed to cause, when executed, the electronic processing resources to become configured to determine that the model update condition is, or is not, satisfied based on the data indicative of the computer vision model' s performance and based on a performance threshold.

[0023] In addition, the computer vision model learning software is possibly designed to cause, when executed, the electronic processing resources to become configured to concurrently perform said computer vision task on the received images to be analysed by means of the computer vision model' s main neural network; and perform the update operations without interrupting the operation of the main neural network.

[0024] The computer vision model learning software is further possibly designed to cause, when executed, the electronic processing resources to become configured to train the auxiliary twin neural network by adjusting its weights based on an unsupervised training algorithm.

[0025] In detail, the operation of forming the updated image dataset comprises the steps of selecting, from external electronic storing resources, external images based on the received images to be analysed and based on the computer vision task' s output; and forming the updated image dataset based on the selected external images and based on the current image dataset.

[0026] In more detail, the operation of forming the updated image dataset further comprises the steps of selecting, from the current image dataset, a plurality of current image dataset 's images to be retained based on the received images to be analysed and based on the computer vision task' s output; and forming the updated image dataset by adding to itself the selected external images and the selected current image dataset' s images to be retained.

[0027] The computer vision model learning software is possibly designed to cause, when executed, the electronic processing resources to become further configured to implement a selection model configured to compute a gain factor indicative of the contribution of an input image to said updated image dataset, for a purpose of learning said computer vision task, based on the received images to be analysed and the computer vision task' s output. Wherein the operation of forming the updated image dataset further comprises the steps of selecting, from external electronic storing resources, the external images based on said selection model; and selecting, from the current image dataset, the current image dataset 's images to be retained based on said selection model.

[0028] In detail, the selection model is further configured to determine a confidence value of the main neural network in performing the computer vision task on said input image; and to compute the gain factor, indicative of the contribution of said input image, based on the determined confidence value of the main neural network on said input image. In addition, the computer vision task is conveniently an object detection task. Optionally, the operation of forming the updated image dataset further comprises the step of detecting, for each of the images to be analysed, a number of objects represented in said image to be analysed, and determined to be associated with one or more predefined classes, by means of a classifier model distinct to the computer vision model. Furthermore, the operation of forming the updated image dataset further comprises the step of determining, for each of the images to be analysed, that a selection triggering condition is, or is not, satisfied for said image to be analysed based on the number of objects detected by means of the classifier model in said image to be analysed and based on the computer vision task' s output computed by performing the computer vision task on said image to be analysed by means of the computer vision model. In addition, the operation of forming the updated image dataset further comprises the step of selecting, as external images, the images to be analysed for which the selection triggering condition is determined to be satisfied. Optionally, the computer vision task is an object re-identification and tracking task, and wherein the selection model is further configured to determine a target representation representing features of an object selected to be identified and tracked by means of the main neural network; and to detect one or more objects represented in said input image and determined to be associated to one or more predefined classes. In addition, the selection model is configured to compute, for each of the detected objects, one or more object features of said detected object by means of the main neural network; and compute an object representation representing said detected object based on said object features. Furthermore, the selection model is configured to compute the gain factor, indicative of the contribution of said input image, based on one or more of the computed object representations and based on the target representation.

[0029] In particular, the operation of forming the updated image dataset comprises the steps of determining an updated image dataset' s size based on the updated image dataset; and forming the updated image dataset by incrementally adding selected images, either from the selected external images or from the selected current image dataset' s images to be retained, to itself as long as the updated image dataset' s size is less than or equal to a size threshold.

[0030] There is further provided, a computer vision model learning system comprising electronic processing resources storing, and configured to execute, the computer vision model learning software according to any one of the preceding claims .

[0031] BRIEF DESCRIPTION OF THE DRAWINGS

[0032] Figure 1 schematically shows a block diagram of a computer vision model learning system according to the present invention.

[0033] Figure 2 schematically shows a block diagram of the computer vision model learning system according to a first embodiment of the present invention.

[0034] Figure 3 and Figure 4 exemplary show comparisons of the neural network capabilities with respect to selected baselines .

[0035] Figure 5 schematically shows a block diagram of the computer vision model learning system according to a second embodiment of the present invention.

[0036] Figure 6 schematically shows an example of a pipeline for the computer vision task according to the second embodiment of the present invention.

[0037] DESCRIPTION OF PREFERRED EMBODIMENTS OF THE INVENTION

[0038] The invention will now be described in detail with reference to the accompanying figures, so as to allow a person skilled in the art to carry it out and use it. Possible changes to the embodiments described herein will be immediately evident to skilled people and the generic principles described herein can be applied to other embodiments and applications without for this reason going beyond the scope of protection of the invention as it is defined in the appended claims. Therefore, the invention cannot be considered as limited to the embodiments described and shown herein, but it has to be associated with the widest scope of protection possible in accordance with the features described and claimed herein.

[0039] If not specifically defined otherwise, all technical and scientific terms have the meaning commonly used by people ordinarily skilled in the industry to which the invention belongs. In case of conflict, the description - including the definitions provided therein - is binding. Furthermore, the examples are provided by mere way of explanation and, as such, should not be considered as limiting.

[0040] In particular, the block diagrams included in the accompanying figures and described below should not be interpreted as representation of structural features, namely as constructive limits, but they should be regarded as representation of functional features, i.e. intrinsic properties of the devices defined by the obtained effects, namely as functional limits, which can be implemented in different manners, hence so as to protects the functionality thereof (possibility of functioning) .

[0041] In order to make it easier for readers to understand the embodiments described herein, reference is made to some specific embodiments and a specific language will be used to describe them. The terms used in this document are aimed at exclusively describing particular examples and are not suited to limit the scope of protection of the invention.

[0042] Figure 1 schematically shows a block diagram of a computer vision model learning system 1 according to the present invention. In detail, Figure 1 schematically shows a block diagram of a computer vision model implemented according to the present invention. In more detail, it shows the architecture of the implemented computer vision model.

[0043] The computer vision model learning system 1 comprises electronic processing resources 2 storing, and configured to execute, a computer vision model learning software 3, or computer product, to autonomously, and preferably continuously, train a main neural network so that it learns to perform a computer vision task or application. In detail, said computer vision task can be any task that involves a computer vision operation. In a way of example, said computer vision task is an object re-identification task (in detail, the problem of matching objects across consecutive images or in a multi-camera system) or an object instance segmentation task (in detail, a task that involves identifying and delineating individual objects within an image while assigning a unique label to each pixel) .

[0044] The computer vision model learning software 3 is configured to cause, when executed, the electronic processing resources 2 to become configured to update the main neural network employing an unsupervised continual learning approach combined with an intelligent image pool collection module 5. In detail, the computer vision model learning software 3 is configured to cause, when executed, the electronic processing resources 2 to become configured to perform a plurality of operations in order to update the main neural network.

[0045] With regard to the computer vision model learning, it is emphasized that what matters are the operations that must be implemented to realize such functionality and not the hardware and software architectures with which such operations are implemented; these could be implemented through a concentrated architecture, for example a single electronic device, or through a distributed cooperative architecture, comprising several electronic devices in communication and cooperating with each other, according to a proprietary logical architecture that the manufacturer of the computer vision model learning software 3 will decide to adopt .

[0046] The electronic processing resources 2 are configured to receive one or more images to be analysed 4, or on which perform a computer vision task. In detail, electronic processing resources 2 are configured to receive the images to be analysed 4 from an electronic device of a user or from electronic storing resources, either internal or internal to the computer vision model learning system 1. In a way of example, said images to be analysed 4 are RGB images.

[0047] The electronic processing resources 2 are further configured to implement (or execute) a computer vision model comprising a main neural network trained, on a current image dataset, to perform (and in detail to carry out) a computer vision task; in detail, a predefined computer vision task. In particular, this main neural network is pre-trained to perform the specific computer vision task (for example image recognition or object detection) in a real-world environment. In particular, said main neural network architecture (or structure) depends on the computer vision task to be learned and to the samples' type of the current image dataset (in detail the initial image dataset) . In fact, the main neural network can be possibly designed with any neural network' s architecture.

[0048] In addition, the electronic processing resources 2 are configured to perform said computer vision task on the received images to be analysed 4 based on the computer vision model .

[0049] The electronic processing resources 2 are configured to compute a computer vision task' s output 9 by performing said computer vision task on the received images to be analysed 4 by means of the computer vision model. In particular, the electronic processing resources 2, in detail the application block 8, are configured to compute an application specific output, to be outputted to a final user of the service, based on the computer vision task' s output 9; in a way of example, the computer vision task' s output 9 corresponds to the application specific output. In detail, the computer vision task' s output 9 is the output of the computer vision model. In particular, in order to perform said computer vision task, the electronic processing resources 2 are configured to input said received images to be analysed 4 to the computer vision model and to determine an output, obtained based on said received images to be analysed 4, from the computer vision model. Optionally, but preferably, the electronic processing resources 2 are configured to compute pre-processed images by executing a preprocessing algorithm on the received images to be analysed 4; and to input said pre-processed images to the computer vision model in order to receive its output.

[0050] In detail, the computer vision task' s output 9 comprises target data obtained by performing said computer vision task; in a way of example, the output comprises a modified image, a label or a description of the received and analysed image. In detail, if the computer vision task is an object detection task, the computer vision task' s output 9 comprises a representation of one or more objects detected in the received and analysed image by executing the computer vision model. Optionally, the output of the computer vision task 9 may be a segmented map that delineates all detected objects, accurately segmented within the received image. Alternatively, if the computer vision task is an object reidentification and tracking task, the computer vision task' s output 9 may be an annotated trajectory map that tracks the movement and re-identification of objects across multiple frames in the received video sequence.

[0051] According to an aspect of the present invention, the computer vision task' s output 9 comprises data indicative of the computer vision model' s performance in performing said computer vision task on the received images to be analysed 4. Preferably, the computer vision task' s output 9 further comprises data, or a value, indicative of the computer vision model' s confidence in providing said computer vision task' s output 9. In a way of example, the data indicative of the computer vision model' s confidence is a confidence interval, which provides a range within which the true value of the computer vision model' s prediction is expected to lie, thereby indicating the reliability of the output. According to a different example, the data indicative of the computer vision model' s confidence is a confidence value that quantifies the certainty of the computer vision model in its prediction, allowing users to assess the trustworthiness of the results.

[0052] Furthermore, the electronic processing resources 2 are configured to perform update operations to update, in particular to almost instantaneously update, the main neural network' s 6 weights; in detail, to adapt the main neural network to a new setup (in more detail, to perform the computer vision task on new different classes or types of images) . Preferably, the electronic processing resources 2 to become configured to concurrently perform said computer vision task on the received images to be analysed 4 by means of the computer vision model' s main neural network and perform the update operations without interrupting the operation of the main neural network.

[0053] In particular, the electronic processing resources 2 are configured to determine that a model update condition is, or is not, satisfied based on the computer vision task' s output 9, and to perform the update operations, to update the main neural network' s 6 weights, when the model update condition is determined to be satisfied. In detail, the electronic processing resources 2 are configured to perform, or execute, the update operations whenever the model update condition is determined to be satisfied.

[0054] Preferably, the electronic processing resources 2 are configured to determine that the model update condition is, or is not, satisfied based on the satisfaction of a model update performance condition and / or based on the satisfaction of a model update confidence condition. According to an aspect of the present invention, the electronic processing resources 2 are configured to determine that the model update condition is satisfied when the model update performance condition or the model update confidence condition is determined to be satisfied. According to a different aspect of the present invention, the electronic processing resources 2 are optionally configured to determine that the model update condition is satisfied when at least one between the model update performance condition, the model update confidence condition and a model update error condition is determined to be satisfied .

[0055] In detail, the electronic processing resources 2 are configured to determine that the model update performance condition is, or is not, satisfied based on the data indicative of the computer vision model' s performance and based on a performance threshold; in particular, a predefined and stored performance threshold value. In particular, the model update performance condition is determined to be satisfied if the main neural network 6 is determined to fail in performing the computer vision task on the received images to be analysed 4. In particular, the electronic processing resources 2 are configured to determine a value indicative of the computer vision model' s performance (in a way of example the accuracy of the computer vision model) based on the data indicative of the computer vision model' s performance; and are configured to determine the satisfaction of the model update performance condition by comparing said value to the performance threshold. In detail, said data indicative of the computer vision model' s performance comprise the value indicative of the computer vision model' s performance. In a way of example, said model update performance condition is satisfied if the value indicative of the computer vision model' s performance is inferior to the performance threshold, indicating that the computer vision model is performing poorly on the received images to be analysed 4.

[0056] In detail, the electronic processing resources 2 are configured to determine that the model update confidence condition is, or is not, satisfied based on the data indicative of the computer vision model' s confidence and based on a confidence threshold; in particular, a predefined and stored confidence threshold value. In particular, the electronic processing resources 2 are configured to determine a confidence value indicative of the computer vision model' s confidence, as it is included in the computer vision task' s output 9; and are configured to determine the satisfaction of the model update confidence condition by comparing said confidence value to the confidence threshold. In a way of example, said model update confidence condition is satisfied if the confidence value is determined to be inferior to the confidence threshold, indicating that the computer vision model lacks certainty in its predictions performed on the received images to be analysed 4.

[0057] In addition, the electronic processing resources 2 are configured to determine that the model update error condition is satisfied if the computer vision task' s output 9 includes data indicating the occurrence of an error during the execution of the computer vision task.

[0058] Optionally, the electronic processing resources 2 are configured to perform the update operations periodically, based on a predefined time interval, independently of the satisfaction of the model update condition.

[0059] Said update operations comprise an operation of forming an updated image dataset, different from the current image dataset, based on the received images to be analysed 4 and based on the computer vision task' s output 9. In detail, the operation of forming the updated image dataset allows the electronic processing resources 2 to perform a different training (with respect to the initial one) due to the difference between the current image dataset and said updated image dataset. In particular, the electronic processing resources 2 are configured to implement an autonomous image pool collection module 5 designed to form, or generate, the updated image dataset based on the received images to be analysed 4 and based on the computer vision task' s output 9 in order to perform a continual learning training, in detail unsupervised training, of the main neural network 6. Optionally, but preferably, the electronic processing resources 2 are configured to collect the updated image dataset in real-time. In detail, the electronic processing resources 2 are configured to form the updated image dataset without any human intervention; in more detail, in order to enable a continual learning of the main neural network 6.

[0060] In detail, the operation of forming the updated image dataset comprises the steps of selecting, from external electronic storing resources, external images based on the received images to be analysed 4 and based on the computer vision task' s output 9; and forming the updated image dataset based on the selected external images and based on the current image dataset. In more detail, the electronic processing resources 2 are configured to search for, and to select, the external images on online databases or on local databases stored in electronic storing resources internal to the computer vision model learning system 1. In a way of example, the electronic processing resources 2 are configured to scrape the external images from online sites. In a way of example, the electronic processing resources 2 are configured to determine the external images based on a predefine scraping algorithm. In particular, before forming the updated image dataset, the electronic processing resources 2 are configured to execute the preprocessing algorithm on said external images in order to convert their format to match the format of the images of the current image dataset .

[0061] Preferably, forming the updated image dataset comprises the operation of selecting, from the current image dataset, a plurality of current image dataset 's images to be retained, or maintained, based on the received images to be analysed 4 and based on the computer vision task' s output 9. Alternatively, the electronic processing resources 2 are possibly configured to select, from the current image dataset, a plurality of current image dataset 's images to be discarded based on the received images to be analysed 4 and based on the computer vision task' s output 9; and to select the images to be retained based on the images to be discarded .

[0062] In addition, the electronic processing resources 2 are designed to form the updated image dataset by adding to itself the selected external images and the selected current image dataset' s images to be retained in detail in order to maintain the main neural network 6 capacity to recognize past objects while learning new ones. In detail, this allows to contrast the catastrophic forgetting problem (the tendency of an artificial neural network to abruptly and drastically forget previously learned information upon learning new information) . In particular, the operation of forming the updated image dataset allows for a neural network fine-tuning on newly collected samples maintaining the information of some of the previously collected ones, maintaining a good trade-off between the old and new sample data collected facing catastrophic forgetting issues. In particular, selecting both external images and current image dataset' s images allows to obtain a good plasticity (capacity of a network to functionally modify itself in response to experience without losing past knowledge) of the main neural network 6.

[0063] According to an aspect of the present invention, the electronic processing resources 2 are designed to implement a selection model 5A configured to compute a gain factor indicative of the contribution, or importance, of an input image to said updated image dataset, for a purpose of learning said computer vision task, based on the received images to be analysed 4 and the computer vision task' s output 9.

[0064] According to said aspect of the present invention, the selection model 5A is configured to compute, or estimate, said gain factor based on a predefined heuristic and / or based on a learning-based technique configured to estimate how retaining or discarding an input image would affect the knowledge contained into the updated image dataset. In particular, the gain factor is a quantity, or a value, indicating the extent to which retaining or discarding the input image would impact the performance of the main neural network 6 trained on the current image dataset.

[0065] Preferably, the selection model 5A is configured to determine a confidence value of the main neural network 6 in performing the computer vision task on said input image; and compute the gain factor, indicative of the contribution of said input image, based on the determined confidence value of the main neural network 6 on said input image.

[0066] In particular, the electronic processing resources 2, in detail the selection model 5A, are configured to determine a data recency status indicative of the fact the input image is new or old data and to compute the gain factor for the input image based on this determined data recency status; new data refers to an external image, while old data refers to data already included in the current image dataset. In detail, the selection model 5A is configured to determine if the input image is a new, or an old, data by searching for the input image in the current image dataset.

[0067] In more detail, if the input image is determined to be an old data the selection model 5A is configured to determine (or receive) the confidence value of the main neural network 6 on said input image based on the corresponding computer vision task' s output 9. In particular, this output comprises the confidence values of the main neural network 6 on all the old images. If the input image is determined to be a new data, the electronic processing resources 2 are possibly configured to compute the confidence value based on the main neural network 6 and said input image in order to compute the gain factor based on the computed confidence value of the main neural network 6 on said input image. In detail, the electronic processing resources 2 are configured to execute the main neural network 6 on said input image and to receive the computer vision task' s output 9 for said input image; and to determine the confidence value of the main neural network 6 on said input image because it is included into the computer vision task' s output 9.

[0068] According to an aspect of the present invention, the gain factor is said confidence value of the main neural network 6. According to a different aspect of the present invention, the electronic processing resources 2 are configured to adjust the gain factor based on the confidence value of the main neural network 6 on said input image by means of a predefine weighting algorithm. In detail, the selection model 5A is optionally configured to compare said confidence value with a confidence threshold and to adjust the gain factor based on said comparison. In more detail, the selection model 5A is configured to compute the difference between the confidence value and the confidence threshold by comparing said confidence value with said confidence threshold; and to adjust the gain factor based on the computed difference. In this way, the electronic processing resources 2 are possibly configured to select the images which the main neural network 6 can recognize but with less confidence, with respect to said confidence threshold, to maintain some past knowledge and strengthen its confidence.

[0069] Optionally, the selection model 5A is configured to compute the current proportion of collected external images and collected images to be retained in the updated image dataset based on the number of external images and the number of the images to be retained in the updated image dataset; and to compute said gain factor based on said proportion and based on the determined data recency status of the input image. More in detail, the selection model 5A is configured to weight, or to adjust, said gain factor in function of said proportion and the determined data recency status in order to maintain a balanced trade-off between the old and the new images collected, thereby preventing forgetting issues .

[0070] Optionally, the selection model 5A is configured to compute a gain factor based on an input image and an input image dataset (in detail, the current image dataset) . According to an aspect of the present invention, the selection model 5A is further configured to determine, or identify, a plurality of images that are representatives of the input image dataset; and to compute said gain factor based on the representatives of the input image dataset. In detail, the selection model 5A is further configured to determine, or identify, the representatives of the input image dataset based on a clustering algorithm; in a way of example, a k-means algorithm. In detail, the selection model 5A is configured to compute the gain factor for an input image based on a similarity value, computed based on a similarity distance algorithm, indicative of the distance between the latter and said representatives of the input image dataset. Optionally, the selection model 5A is configured to compute, for each one of the representatives of the input image dataset, a similarity value between the input image and said representative based on the input image and said representative image. In addition, the selection model 5A is configured to compute the gain factor based on the computed similarity values; for example, by performing an average of said computed similarity values.

[0071] In addition, the selection model 5A is further configured to output a data indicative of the computed gain factor; in particular, the output comprises the computed gain factor.

[0072] In addition, the operation of forming the updated image dataset further comprise the steps of selecting, from external electronic storing resources, the external images based on said selection model 5A; and selecting, from the current image dataset, the current image dataset 's images to be retained based on said selection model 5A. In a way of example, the electronic processing resources 2, by relying on the selection model 5A, are configured to select the external images and the images to be retained based on the most representative images from the current image dataset. In particular, the electronic processing resources 2 are configured to input one image, either external or from the current image dataset, into the selection model 5A and to receive an output from said selection model 5A. Furthermore, said electronic processing resources 2 are configured to select, or not select, said input image to be part of the updated image dataset based on the output received from the selection model 5A. In detail, said electronic processing resources 2 are configured to determine that a selection condition is, or is not, satisfied for said input image based on the selection model 5A' s output. In more detail, the electronic processing resources 2 are configured to determine the computed gain factor based on the selection model 5A' s output; for example, it is comprised within the output, or it is computed by performing a format conversion. Optionally, said selection condition is satisfied if the computed gain factor is higher than, or equal to, a gain threshold; wherein, in detail, the images for which said selection condition is satisfied are images which can improve, or at least maintain, the main neural network 6 performance. In detail, said gain threshold can be predefined or can be computed possibly based on a plurality of several gain thresholds computed on different input images; in a way of example, said gain threshold is the average of the several computed gain thresholds. If said selection condition is determined to be satisfied, the electronic processing resources 2 are configured to select said input image and add, or include, it into the updated image dataset.

[0073] Preferably, the operation of determining the updated image dataset comprises determining an updated image dataset' s size (current size) based on the updated image dataset. In detail, said operation of determining the updated image dataset comprises forming the updated image dataset by incrementally adding selected images, either from the selected external images or from the selected current image dataset' s images to be retained, to itself as long as the updated image dataset' s size is less than or equal to a size threshold. In particular, the electronic processing resources 2 are configured to cease, or stop, the selection and addition of images to the updated image dataset when the updated image dataset' s size is more than or possibly equal to the size threshold. In fact, the electronic processing resources 2 are configured to retain a finite quantity of the updated image dataset samples (in detail the most representative) to avoid infinite dataset size and memory storage increase. In detail, the electronic processing resources 2 are designed to determine that a training triggering condition is satisfied if the size of the updated image dataset is higher or equal to the size threshold; and to determine that the training triggering condition is not satisfied if the updated image dataset' s size is less than said size threshold. In more detail, the electronic processing resources 2 are designed to determine that a training triggering condition is satisfied if the updated image dataset is already formed and ready to train a neural network .

[0074] In detail, said size threshold is the current image dataset' s size. Alternatively, the electronic processing resources 2 are optionally configured to compute and define said size threshold based on the current image dataset' s size and / or based on data indicative of the computational resources and / or capabilities available for the electronic processing resources 2. Alternatively, the size threshold is a predefined value stored in the electronic storing resources.

[0075] Furthermore, the update operations comprise an operation of training (or fine tuning) an auxiliary twin neural network 7, being different from the main neural network 6 but having the same architecture of the main neural network 6 (in detail being different only for the weights) , by adjusting the auxiliary twin neural network' s 7 weights based on the updated image dataset to perform said computer vision task.

[0076] In detail, the electronic processing resources 2 are configured to trigger the training (or to execute the operation of training) of the auxiliary twin neural network 7 based on the updated image dataset if the training triggering condition is determined to be satisfied. In more detail, the electronic processing resources 2 are configured to wait before training the auxiliary twin neural network 7 based on the updated image dataset until the training triggering condition is satisfied. In particular, the auxiliary twin neural network' s 7 training is triggered solely when the image pool collection module 5 has acquired a number of images, or samples, higher than the size threshold .

[0077] In detail, the operation of training an auxiliary twin neural network 7 comprises the operation of divide the updated image dataset into a (in more detail predefined) number of batches and start the training of the auxiliary twin neural network 7 on said batches. In more detail, the operation of training an auxiliary twin neural network 7 comprises the operation of divide the updated image dataset into a training set, a test set and optionally a validation set and start the training of the auxiliary twin neural network 7 at least on the training set, the test set.

[0078] Preferably, the electronic processing resources 2 are configured to train the auxiliary twin neural network 7 by adjusting its weights based on an unsupervised training algorithm. In detail, said unsupervised training algorithm, when executed, allows to compute one or more performance (or loss) values, indicative of the performance of the auxiliary twin neural network 7 in performing the computer vision task on said received images, and to adjust its weights based on the computed loss values.

[0079] In particular, the electronic processing resources 2 are configured to perform a plurality of training epochs (optionally, the number of epochs is predefined) in order to train the auxiliary twin neural network 7. In detail, the electronic processing resources 2, at each training epoch, are configured to determine a performance (or loss) value indicative of the performance of the auxiliary twin neural network 7 in performing the computer vision task on a plurality of images stored in the test dataset specifically designed to evaluate the auxiliary twin neural network 7 during its training. In particular, this performance value is then used to adjust the auxiliary twin neural network' s 7 weights, ensuring that the model continues to improve its accuracy and efficiency in handling the computer vision task. Additionally, the electronic processing resources 2 are optionally configured to periodically update the test dataset in order to include new images to said test dataset, providing a comprehensive evaluation of the auxiliary twin neural network' s 7 capabilities over time.

[0080] Possibly, the computer vision model further comprises said auxiliary twin neural network 7; more in detail, said auxiliary twin neural network 7 is parallel, and possibly independent, to the main neural network 6 of the computer vision model .

[0081] Preferably, said electronic processing resources 2 are configured to train the auxiliary twin neural network 7 while the main neural network 6 is operative and provides a service of computer vision to one or several users. In detail, the operation of training the auxiliary twin neural network 7 is a parallel operation with respect to any operation performed by the main neural network 6.

[0082] The update operations further comprise an operation of substituting the weights of the main neural network 6 with the weights of the auxiliary twin neural network 7 to update the main neural network 6 without interrupting (or stopping) its operation, in detail its inference operation. This procedure lets the main neural network 6 adjust (by substituting the weights) its weights to new object instances and domains without stopping the whole application, when it is in inference mode and the underlying application is running .

[0083] In detail, the update operations comprise the operation of determining if a training stopping condition is, or is not, satisfied based on the performance value of the auxiliary twin neural network 7 computed at the current training epoch and / or based on the current training epoch. In detail, said training stopping condition is satisfied when the electronic processing resources 2 determine that the training of the auxiliary twin neural network 7 can be stopped. In a way of example, the electronic processing resources 2 are configured to determine that the training stopping condition is satisfied if the determined performance value (in detail the overall performance of the auxiliary twin neural network 7) is higher than a certain performance threshold. In a different way of example, the electronic processing resources 2 are configured to determine that the training stopping condition is satisfied if the current training epoch is a number higher than a predefined number of epochs.

[0084] If said training stopping condition is determined to be satisfied, the electronic processing resources 2 are configured to perform the operation of substituting the weights of the main neural network 6 with the weights of the auxiliary twin neural network 7 to update the main neural network 6. In detail, the electronic processing resources 2 are configured to copy on-the-fly the auxiliary twin neural network' s 7 weights in the main neural network 6 enabling a fast adaptation of the main neural network 6 on the new environment setup. In detail, thanks to GPU (Graphics Processing Unit) memory performance, the operation of substituting the weights of the main neural network 6 with the weights of the auxiliary twin neural network 7 is almost instantaneous and introduces negligible delays in the overall process. In addition, the neural network duplication could be also used as a backup of the main neural network 6 architecture in case of blocking failures of such architecture. In detail, even if having an additional architecture in parallel to the original one could require additional hardware and software capabilities in the implementation, the benefits in terms of time, money and management surpass these minimal initial investments.

[0085] According to a first embodiment of the present invention, the computer vision task is an object re-identification and tracking task. In detail, this task aims to detect, localize and then track an object of interest (namely a target to be localized) on an RGB image; this is performed by reidentifying the object in consecutive frames especially when the object is subject to partial / total occlusions and changes its appearance.

[0086] Figure 2 schematically shows a block diagram of the computer vision model learning system 1 according to said first embodiment of the present invention. According to said first embodiment, the electronic processing resources 2, in more detail the application block 8, are configured to provide as output the detection of the selected object of interest on an input image (in a way of example, the re-identified target is highlighted with its white segmentation contours) .

[0087] According to said first embodiment of the present invention, the computer vision model preferably comprises an instance segmentation neural network 6A configured to provide a rough initial tracking estimate, and the main neural network 6 preferably is a feature extractor neural network configured to determine object features (or characteristics) based on the rough initial tracking estimate. In particular, the instance segmentation neural network 6A is configured or trained to identify various objects represented in an input image and to crop the image to generate different corresponding sub-images, each representing a distinct object.

[0088] Wherein, the main neural network 6 is conveniently designed to compress the determined object features into a feature vector (in detail with a fixed length) to be outputted. In detail, the main neural network 6 is designed to determine several object features and to generate corresponding feature vectors to be outputted (in detail to the application block 8) . Wherein, the auxiliary twin neural network 7, having the same architecture of the main neural network 6, is a feature extractor neural network.

[0089] In particular, the instance segmentation neural network 6A is trained to output data indicative of the rough initial tracking estimate to the main neural network 6 in order to enable the latter to perform the computer vision task.

[0090] In detail, the instance segmentation network 6A is trained once; in more detail, after the initial training, the instance segmentation network' s 6A weights remain fixed. In more detail, according to said first embodiment of the present invention, the electronic processing resources 2 are configured to update, with new weights, only the feature extractor neural network.

[0091] Preferably, the image pool collection module 5 is configured to create a local database comprising target and distractors; wherein, the term distractor refers to each object which is in the same class as the target but is a different entity (all the objects which are not target, in other words objects not of interest for the task) . The image pool collection module 5 is optionally configured to randomly select, from the distractor images (not associated with an identifier of the target) , a plurality of images; in detail, a balanced amount of images with respect to the target images. This randomness allows to consider a generalizable appearance of the available distractors.

[0092] Preferably, the electronic processing resources 2 are configured to train the computer vision model, and in particular the auxiliary twin neural network 7, based on a distance-based loss function (in detail a triplet loss) designed to learn embeddings that are closer for similar input data and farther for dissimilar ones. In detail, using such a distance-based loss function would force the network to produce embeddings which are more similar to the features coming from the target and more dissimilar from the ones belonging to the distractors. In more detail, this training procedure would allow the auxiliary twin neural network 7 to learn the distinctive characteristics of the target to distinguish it from the distractors.

[0093] According to said first embodiment of the present invention, the selection model 5A is configured to determine a target representation representing features of an object selected to be identified and tracked by means of the main neural network 6; wherein, possibly the target representation will be used for the re-identification step. According to a preferred aspect of the first embodiment of the present invention, the electronic processing resources 2, in detail the application block 8, are configured to compute the target representation by computing a statistical model (preferably a statistical Gaussian model) of the target based on one or more feature vectors received in input; in particular, from the main neural network 6. In detail, the target representation is the computed statistical model.

[0094] In more detail, the electronic processing resources 2 are configured to select (based on a predefined algorithm) an object to be identified, and eventually tracked, as the target between the objects detected by means of the main neural network 6 in images already analysed 4. In more detail, the images to be analysed 4 are a sequence, or an ordered set, of images representing an area to be monitored, in which an object can be identified and / or tracked as target. More in particular, the electronic processing resources 2 are configured to receive the images to be analysed 4 from a camera arranged, into an environment, to frame a predefined area and capture images of that predefined area; and possibly to determine the sequence of said images to be analysed 4 based on the captured images and the corresponding time instants at which the camera captured said images. Hence, the electronic processing resources 2 are configured to determine, and in detail to collect, the computer vision task' s output 9 computed on one or more received images already analysed 4; and to compute (in detail as a statistical model) the target representation based on said computer vision task' s outputs 9. In particular, the electronic processing resources 2 are configured to determine a feature vector for each computer vision task' s output 9 and to compute the target representation based on the determined feature vectors.

[0095] Alternatively, according to a different aspect of the first embodiment, the electronic processing resources 2 are optionally configured to receive a data indicative of the target from a user' s device and are configured to compute said target representation by extracting its features based on the main neural network 6.

[0096] According to said first embodiment of the present invention, the selection model 5A is configured to detect (in detail executing the instance segmentation network 6A) one or more objects represented in said input image (in detail the current image to be analysed 4) and determined to be associated to one or more predefined classes. In addition, the selection model 5A is configured to compute, for each of the detected objects, one or more object features of said detected object by means of the main neural network 6; and compute an object representation representing said detected object based on said object features. Preferably, said object representations are feature vectors. In detail, the selection model 5A is configured to receive the object features from the main neural network 6 (determined by means of the latter) . In detail, the selection model 5A is configured to input an image to be analysed 4 to the main neural network 6 to cause it to compute the object features, and then to compute the feature vector based on said object features .

[0097] According to said first embodiment of the present invention, the selection model 5A is further configured to compute the gain factor, indicative of the contribution of said input image, based on one or more of the computed object representations and based on the target representation.

[0098] According to a preferred aspect of the first embodiment of the present invention, the selection model 5A is configured to compute the gain factor by comparing the object representation (in detail a feature vector of the input image) with the target representation (in detail the statistical model) based on a similarity distance algorithm. In particular, wherein, the similarity distance algorithm is designed to compute a value indicative of the similarity, or distance, between the object representation and the target representation; more in detail, the gain factor corresponds to said computed value indicative of the similarity.

[0099] In detail, the selection model 5A is configured to select a final object representation between the object representations, or to compute one different object representation based on all the object representations. The selection model 5A is further configured to compute the gain factor by applying the similarity distance algorithm between the final object representations and the target representation. In detail, said similarity distance algorithm is a Mahalanobis distance (De Maesschalck, Roy, Delphine Jouan-Rimbaud, and Desire L. Massart. "The mahalanobis distance." Chemometrics and intelligent laboratory systems 50.1 (2000) : 1-18. ) ; which is designed to compare a feature vector (the object representation) of the target with the statistical Gaussian model (the target representation) adopted for the re-identification task. In particular, the selection model 5A computes the gain factor based on the extent to which object features, determined to be associated with the input image, provide information for training with the current image dataset. In a way of example, the object features which provide the greatest information to the training step are the features which are the farthest from the target model and the least re-identifiable ones but still belong to the target of interest .

[0100] In a way of example, the first embodiment of the present invention has been tested using a network architecture specialized in people re-identification and has proven to be effective for tracking specific targets on images surpassing the current state of the art. Specifically, the first embodiment was compared with a MOT (Multi-Object Tracking) algorithm (for example, a yolo tracking) and a reidentification and tracking algorithm. The experiment compared the algorithms with a custom dataset in a single person tracking task, i.e. a selected person to be tracked in the image plane while partial and total occlusions or visual appearance changes occur. As can be seen in Figure 3 the mean tracking time of the systems has been compared and the approach of the first embodiment of the present invention outperforms both systems. Instead, in Figure 4, the reidentification capabilities of the implementation of the first embodiment of the present invention have been compared with the ones of the other re-identification framework. In detail, the first embodiment of the present invention, employing the unsupervised continual learning approach presented, can recover the errors that the other algorithms have during inference resulting in a perfect reidentification and tracking task in all the videos of the dataset .

[0101] According to a second embodiment of the present invention, the computer vision task is an object detection task. This second embodiment of the present invention can be implemented for any application which requires the detection and the mask of an object on the image, e.g. , surveillance, object detection for 3D localization, logistics or others.

[0102] Figure 5 schematically shows a block diagram of the computer vision model learning system 1 according to said second embodiment of the present invention.

[0103] According to the second embodiment of the present invention, the operation of forming the updated image dataset further comprises a step of detecting, for each of the images to be analysed 4, a number of objects represented in said image to be analysed 4, and determined to be associated to one or more predefined classes (specifically, types of objects to identify) , by means of a classifier model distinct (or different) , and separate, to the computer vision model. In detail, the electronic processing resources 2 are configured to execute the classifier model to detect the objects represented in the image to be analysed 4, passed to the computer vision model by the electronic processing resources 2, and to compute and output a data indicative of the presence or absence of objects of said predefined classes in the image to be analysed 4; in particular, a data indicative of the number of objects detected, associated to one or more predefined classes, in said image to be analysed 4. In more detail, the classifier model is configured to compute a data indicative of the presence or absence of objects based on the detected objects. In a way of example, the data indicative of the presence or absence of objects comprises a positive value if an input image is determined to comprise, or represent, objects of predefined classes; alternatively, it comprises a negative value if an input image is determined not to comprise, or represent, objects of predefined classes. In particular, the electronic processing resources 2 are configured to implement the classifier model configured, in detail trained, to receive an input image, to detect one or more objects, represented in said input image, determined to be associated to one or more predefined classes. In detail, said classifier model is trained based on a predefined dataset comprising several images, representing objects of predefined classes, in order to be enabled to determine if an input image represents at least on object of one of the predefined classes. In more detail, said classifier model comprises a neural network trained based on the predefined dataset to detect objects of the predefined images, and wherein said classifier model is designed to have a different neural network' s architecture with respect to the computer vision model. In more detail, the classifier model neural network has a number of layers inferior with respect to the number of layers of the computer vision model' s neural network. In a way of example, while the classifier model is less accurate in detecting objects, it requires fewer computational resources 2 to operate compared to the computer vision model .

[0104] According to the second embodiment of the present invention, the operation of forming the updated image dataset further comprises a step of determining, for each of the images to be analysed 4, that a selection triggering condition is, or is not, satisfied for said image to be analysed 4 based on the number of objects detected by means of the classifier model in said image to be analysed 4 and based on the computer vision task' s output 9 computed by performing the computer vision task on said image to be analysed 4 by means of the computer vision model.

[0105] In detail, the electronic processing resources 2 are configured to determine that said selection triggering condition is or is not satisfied by comparing the classifier model' s output, in particular the number of objects detected, with the computer vision task' s output 9.

[0106] According to an aspect of the present embodiment of the present invention, the electronic processing resources 2 are configured to determine if the computer vision task' s output 9 is indicative of the presence or of the absence of objects detected by means of the computer vision model; and are configured to determine if the classifier model' s output is indicative of the presence or of the absence of objects detected by means of the classifier model. In particular, the electronic processing resources 2 are configured to determine that the computer vision model' s output is indicative of the presence of objects detected by means of the computer vision model if it comprises data of at least one object; and otherwise it is indicative of the absence of objects (in particular it is indicative of the absence of objects if the computer vision model failed to detect objects in the image) . In more detail, the electronic processing resources 2 are configured to determine that said selection triggering condition is satisfied if the number of objects detected by means of the classifier model in said image to be analysed 4 is at least one, and if the computer vision task' s output 9, computed by performing the computer vision task on said image to be analysed 4 by means of the computer vision model, is indicative of the absence of detected objects in said image to be analysed 4; in more detail, otherwise the selection triggering condition is determined to be not satisfied. In particular, the electronic processing resources 2 are configured to select, and to collect, the images not detected by the main neural network 6 if it is determined that a trigger (for example a camera trigger) was activated but no objects were localized by the main neural network 6.

[0107] According to a different aspect of the present embodiment of the present invention, the electronic processing resources 2 are configured to determine the number of objects detected by means of the computer vision model based on the computer vision task' s output 9; and are configured to determine the number of objects detected by means of the classifier model based on the classifier model' s output. In more detail, the electronic processing resources 2 are configured to determine that said selection triggering condition is satisfied if the number of objects detected by means of the classifier model in said image to be analysed 4 is different than the number of objects detected by means of the computer vision model; and, possibly, to determine that said selection triggering condition is not satisfied if the number of objects detected by means of the classifier model in said image to be analysed 4 corresponds to the number of objects detected by means of the computer vision model .

[0108] In addition, also according to said second embodiment, the automatic image collection module is configured to form the updated image dataset based the received images to be analysed 4 and the output detection of the main neural network 6.

[0109] In addition, according to the second embodiment, the operation of forming the updated image dataset further comprises a step of selecting, as external images, the images to be analysed 4 for which the selection triggering condition is determined to be satisfied. In particular, the selection triggering condition is satisfied for the images where the computer vision model is determined to fail to perform the computer vision task, in order to select the images that improve the quality of the updated image dataset for the purpose of learning the computer vision task. Specifically, if the computer vision model fails to perform the task on an image, it can be assumed that this image is important for learning the computer vision task.

[0110] In particular, the selection model 5A is further configured to determine if the selection triggering condition is or is not satisfied, and to compute the gain factor based on the satisfaction of the selection triggering condition. In a way of example, said gain factor is equal to one if the selection triggering condition is determined to be satisfied and is equal to zero if the selection triggering condition is determined to not be satisfied. Furthermore, preferably, the step (or operation) of selecting, as external images, the images to be analysed 4 for which the selection triggering condition is determined to be satisfied is performed by the selection model 5A.

[0111] In detail, it can be observed that the accuracy of the classifier model is not paramount; rather, its speed relative to the computer vision model is crucial for promptly initiating the image selection process. In more detail, the primary objective of employing this classifier model is to ascertain whether the selection triggering condition is satisfied and, if so, to trigger the image selection operation accordingly.

[0112] According to said second embodiment, the main neural network 6 and the auxiliary twin neural network 7 are preferably instance segmentation networks (with the same architecture, but possibly with different weights) . Optionally, the electronic processing resources 2 are further configured to perform a blobs clustering inside the received images to be analysed 4 and, due to the precise spatial overlapping between RGB and the received images to be analysed 4, the locations of the objects can be mapped and saved in pixel coordinates of the RGB images. In addition, optionally, to complete the automatic labelling, for each post-localized, but not detected, object in the images a label class is assigned utilizing an additional classification network module which processes the segmented objects cut from the received images to be analysed 4.

[0113] Figure 6 schematically shows an example of a pipeline for the computer vision task according to the second embodiment of the present invention.

[0114] For example, the second embodiment was tested in a logistics scenario, where objects were moving on a rolling belt and a camera (in detail, a type of depth camera that provides both depth (D) and colour (RGB) data as the output in real-time) was mounted on top of it, in a perpendicular direction. In detail, the second embodiment of this invention allows to recognize and localize objects which are passing under the camera over the rolling belt. More in detail, through a photocell which simply detects a surface which is over the rolling belt, a trigger is sent to the RGB-D camera for image acquisition when an object is passing inside a tunnel. More in detail, for refining an instance segmentation network, the location and the class of each object are labelled inside each image which will be used for automatic steps of training. In addition, the Applicant noted that there was no performance degradation when the objects or the environment changed their appearances, for example, new parcels or different lighting during the day.

[0115] In a way of example, the present invention can also be applied in human-robot interaction tasks; for example, to accomplish a human-following scenario, i.e. , the robot follows a specific person (the target) in a human-populated environment. Moreover, for example, this invention is further intended to be used in a package' s recognition setup for Autonomous Robotic Grasping in a logistic scenario.

[0116] The present solution described herein offers several advantages .

[0117] In particular, the proposed invention makes use of an auxiliary twin neural network 7 to perform the training iterations in parallel to the operation (and activity) of the main neural network 6, allowing the normal continuation of the underlying application which executes the main neural network 6. In this way, the provided solution enables the main neural network 6 to be updated without the need for an expert human operator to manage the entire process. This feature is particularly beneficial for critical applications which cannot be stopped for fine-tuning the network because it would result in a loss of performance, time and money.

[0118] In particular, substituting the weights of the main neural network 6 with the weights of the auxiliary twin neural network 7 to update the main neural network 6 is an operation that allows the main neural network 6 process to smoothly continue without any need to stop.

[0119] In addition, the Applicant notes that the proposed invention allows also to overcome the known catastrophic forgetting problem. In detail, the Applicant further notes that the proposed software architecture allows to improve or expand the network capabilities without affecting the previous neural network knowledge.

[0120] Furthermore, the Applicant notes that the present invention enables the completely autonomous continual learning of an artificial neural network for a computer vision task.

[0121] In addition, in the field of neural network applications, this architecture can incorporate an adaptation module that autonomously refines the network without requiring any human intervention, continuously improving application results without interrupting the overall process. The reported examples, particularly the one involving object detection on a conveyor belt in industrial environments, clearly demonstrate how this solution can be employed and the advantages it can bring to a production site.

[0122] The Applicant notes that the general composition of the architecture allows for its applicability in different fields and tasks other than the ones exposed; in addition, it can be applied to any application involving neural networks resulting in a highly valuable architecture for different tasks. In particular, the proposed invention can be used every time a real-time, continuous adaptable, finetuned network is needed on specific domains or environments.

Claims

CLAIMS1. Computer vision model learning software (3) being loadable in, and executable by, electronic processing resources (2) and designed to cause, when executed, the electronic processing resources (2) to become configured to :- receive one or more images to be analysed (4) ;- implement a computer vision model comprising a main neural network (6) trained, on a current image dataset, to perform a computer vision task; and- compute a computer vision task' s output (9) by performing said computer vision task on the received images to be analysed (4) by means of the computer vision model; characterized in that the computer vision model learning software (3) is further designed to cause, when executed, the electronic processing resources (2) to become configured to perform update operations to update the main neural network' s (6) weights; wherein said update operations comprise:- forming an updated image dataset, different from the current image dataset, based on the received images to be analysed (4) and based on the computer vision task' s output (9) ;- training an auxiliary twin neural network (7) , being different from the main neural network (6) but having the same architecture as the main neural network (6) , by adjusting the auxiliary twin neuralnetwork' s (7) weights based on the updated image dataset to perform said computer vision task; and- substituting the weights of the main neural network (6) with the weights of the auxiliary twin neural network (7) to update the main neural network (6) without interrupting its operation.

2. Computer vision model learning software (3) according to claim 1, and designed to cause, when executed, the electronic processing resources (2) to become configured to:- determine that a model update condition is, or is not, satisfied based on the computer vision task' s output (9) ; and- perform the update operations, to update the main neural network' s (6) weights, when the model update condition is determined to be satisfied.

3. Computer vision model learning software (3) according to claim 2, where the computer vision task' s output (9) comprises data indicative of the computer vision model' s performance on said computer vision task performed on the received images to be analysed (4) ; and where the computer vision model learning software (3) is designed to cause, when executed, the electronic processing resources (2) to become configured to determine that the model update condition is, or is not, satisfied based on the data indicative of the computer vision model' s performance and based on a performance threshold.

4. Computer vision model learning software (3) according to any one of the preceding claims, and designed to cause, when executed, the electronic processing resources (2) to become configured to concurrently:- perform said computer vision task on the received images to be analysed (4) by means of the computer vision model' s main neural network (6) ; and- perform the update operations without interrupting the operation of the main neural network (6) .

5. Computer vision model learning software (3) according to any one of the preceding claims, and designed to cause, when executed, the electronic processing resources (2) to become configured to train the auxiliary twin neural network (7) by adjusting its weights based on an unsupervised training algorithm.

6. Computer vision model learning software (3) according to any one of the preceding claims, wherein the operation of forming the updated image dataset comprises the steps of:- selecting, from external electronic storing resources, external images based on the received images to be analysed (4) and based on the computer vision task' s output (9) ; and- forming the updated image dataset based on the selected external images and based on the current image dataset.

7. Computer vision model learning software (3) according to claim 6, wherein the operation of forming the updated image dataset further comprises the steps of:- selecting, from the current image dataset, a plurality of current image dataset 's images to be retained based on the received images to be analysed (4) and based on the computer vision task' s output ( 9) ; and- forming the updated image dataset by adding to itself the selected external images and the selected current image dataset' s images to be retained .

8. Computer vision model learning software (3) according to claim 7, and designed to cause, when executed, the electronic processing resources (2) to become further configured to implement a selection model (5A) configured to compute a gain factor indicative of the contribution of an input image to said updated image dataset, for a purpose of learning said computer vision task, based on the received images to be analysed (4) and the computer vision task' s output (9) ; wherein the operation of forming the updated image dataset further comprises the steps of:- selecting, from external electronic storing resources, the external images based on said selection model (5A) ; andselecting, from the current image dataset, the current image dataset 's images to be retained based on said selection model (5A) .

9. Computer vision model learning software (3) according to claim 8, wherein the selection model (5A) is further configured to:- determine a confidence value of the main neural network (6) in performing the computer vision task on said input image; and- compute the gain factor, indicative of the contribution of said input image, based on the determined confidence value of the main neural network (6) on said input image.

10. Computer vision model learning software (3) according to any one of the claims from 6 to 9, wherein the computer vision task is an object detection task, and wherein the operation of forming the updated image dataset further comprises the steps of:- detecting, for each of the images to be analysed (4) , a number of objects represented in said image to be analysed (4) , and determined to be associated to one or more predefined classes, by means of a classifier model distinct to the computer vision model ;- determining, for each of the images to be analysed (4) , that a selection triggering condition is, or is not, satisfied for said image to be analysed (4) based on the number of objects detected bymeans of the classifier model in said image to be analysed (4) and based on the computer vision task' s output (9) computed by performing the computer vision task on said image to be analysed (4) by means of the computer vision model; and- selecting, as external images, the images to be analysed (4) for which the selection triggering condition is determined to be satisfied.

11. Computer vision model learning software (3) according to claim 8 or 9, wherein the computer vision task is an object re-identification and tracking task, and wherein the selection model (5A) is further configured to :- determine a target representation representing features of an object selected to be identified and tracked by means of the main neural network;- detect one or more objects represented in said input image and determined to be associated with one or more predefined classes;- compute, for each of the detected objects, one or more object features of said detected object by means of the main neural network; and compute an object representation representing said detected object based on said object features; and- compute the gain factor, indicative of the contribution of said input image, based on one or more of the computed object representations and based on the target representation.

12. Computer vision model learning software (3) according to any one of the claims from 7 to 11, wherein the operation of forming the updated image dataset comprises the steps of: - determining an updated image dataset' s size based on the updated image dataset; and- forming the updated image dataset by incrementally adding selected images, either from the selected external images or from the selected current image dataset' s images to be retained, to itself as long as the updated image dataset' s size is less than or equal to a size threshold.

13. Computer vision model learning system (1) comprising electronic processing resources (2) storing, and configured to execute, the computer vision model learning software (3) according to any one of the preceding claims.

Images