Method and system for neural network object recognition for image processing

Abstract

The present disclosure relates to methods and systems for neural network object recognition for image processing, including custom training databases and adapting instance segmentation neural networks for performing customizations.

Description

Background Technology

[0001] Computer vision provides visual capabilities to computers or automated machines. Therefore, the goal of computer vision is to provide such systems with the ability to infer the physical world by understanding what is seen, for example, in 3D and from images captured by a camera. In other words, applications in robotics, virtual reality (VR), augmented reality (AR), and merged reality (MR) may require understanding the world around a robot or person providing the viewpoint in these applications. For example, a robot needs to understand what it sees in order to manipulate (grasp, move, etc.) objects. VR, AR, or MR applications, for instance, need to understand the world around a person providing the viewpoint so that, as the person moves within that world, they are shown how to avoid obstacles in that world. This capability also allows such computer vision systems to add semantically believable virtual objects to the world environment. Thus, a system that understands it is looking at a light can understand the light's purpose and operation. Other computer vision applications also utilize this semantic object recognition, such as video editing, autonomous vehicles, smart homes, and security systems.

[0002] For these purposes, many semantic object recognition techniques, including object classification, detection, and segmentation, are now based on neural networks. However, many traditional neural network object recognition techniques are trained on datasets that are too small relative to the number of potential objects in the world, and therefore often miss the expected objects. However, network datasets are often also very large, making training neural networks both expensive and time-consuming. The situation is exacerbated when such neural networks are often not adaptable to adding the desired target objects to the neural network database for a specific user. Attached Figure Description

[0003] The materials described herein are shown in the accompanying drawings by way of example rather than limitation. For simplicity and clarity, the elements shown in the drawings are not necessarily drawn to scale. For example, the dimensions of some elements may be exaggerated relative to others for clarity. Furthermore, reference numerals are repeated between drawings where deemed appropriate to indicate corresponding or similar elements. In the drawings: Figure 1 This is a schematic diagram of an image processing device according to at least one implementation of this article; Figure 2 This is a flowchart of an object recognition method for image processing according to at least one implementation of this paper; Figures 3A-3C This is a detailed flowchart of a recognition method for image processing according to at least one implementation of this paper; Figure 4This is a schematic flowchart of local and global neural network adaptation for object segmentation according to at least one implementation of this paper; Figure 5 This is a schematic flowchart of a merged object segmentation mask method based on at least one implementation of this paper; Figure 6 It is a sequence of example video images containing the objects to be segmented; Figure 7 It is aimed at Figure 6 The example of a sequence of video image sequences that produces a sequence of object segmentation masks; Figures 8A-8D The example image is used to illustrate a user-defined dataset entry for an object recognition method in image processing, based on at least one implementation of this paper. Figure 9 This is a schematic diagram of the example system; Figure 10 This is a schematic diagram of another example system; and Figure 11 Another example device is shown, all arranged according to at least some implementations of this disclosure. Detailed Implementation

[0004] One or more implementations will now be described with reference to the accompanying drawings. While specific configurations and arrangements are discussed, it should be understood that this is for illustrative purposes only. Those skilled in the art will recognize that other configurations and arrangements may be employed without departing from the spirit and scope of this specification. It will be apparent to those skilled in the art that the techniques and / or arrangements described herein can also be used in a variety of other systems and applications besides those described herein.

[0005] While the following description illustrates various implementations that can be embodied in architectures such as, for example, System-on-a-Chip (SoC) architectures, the implementations of the technologies and / or arrangements described herein are not limited to specific architectures and / or computing systems, and can be implemented by any architecture and / or computing system for similar purposes. For example, various architectures employing, for example, multiple integrated circuit (IC) chips and / or packages, and / or various computing devices, commercial devices, and / or consumer electronics (CE) devices (such as imaging devices that may or may not be used for computer vision tasks, digital cameras, smartphones, webcams, video game panels or consoles, set-top boxes, tablets, etc.) can implement the technologies and / or arrangements described herein, where any of them may have light projectors and / or sensors for performing object detection, depth measurement, and other tasks. Furthermore, while the following description may set forth many specific details, such as logical implementations, types and interrelationships of system components, logical partitioning / integration choices, etc., the claimed subject matter can be implemented without these specific details. In other cases, some materials, such as, for example, control structures and complete sequences of software instructions, may not be shown in detail so as not to obscure the material disclosed herein. The materials disclosed herein can be implemented in hardware, firmware, software, or any combination thereof.

[0006] The materials disclosed herein can also be implemented as instructions stored on a machine-readable medium or memory, which can be read and executed by one or more processors. A machine-readable medium can include any medium and / or mechanism for storing or transmitting information in a machine-readable (e.g., computing device) form. For example, a machine-readable medium can include read-only memory (ROM); random access memory (RAM); disk storage media; optical storage media; flash memory devices; electrical, optical, acoustic, or other forms of propagated signals (e.g., carrier waves, infrared signals, digital signals, etc.) and others. In another form, a non-transitory article, such as a non-transitory computer-readable medium, can be used with any of the foregoing examples or other examples, except that it does not include the transient signal itself. It includes elements other than the signal itself that can temporarily hold data in a "transitory" manner, such as RAM, etc.

[0007] In this specification, references to "an implementation," "implementation," "example implementation," etc., indicate that the described implementation may include a specific feature, structure, or characteristic, but each implementation may not necessarily include that specific feature, structure, or characteristic. Furthermore, these phrases do not necessarily refer to the same implementation. Moreover, when a specific feature, structure, or characteristic is described in conjunction with an implementation, it is asserted that the influence of such feature, structure, or characteristic on other implementations, whether explicitly described herein or not, is within the knowledge of a person skilled in the art.

[0008] Systems, items, and methods for providing neural network object recognition for imaging processing.

[0009] As mentioned, computer vision is frequently used to infer the physical world. Applications in robotics, virtual reality (VR), augmented reality (AR), merged reality (MR), and other computer vision-based automation applications may require understanding the world around a camera sensor, whether the sensor is mounted on a robot, at the user's point of view (POV), or on other devices such as autonomous vehicles. For example, these systems may need to understand what they see in order to manipulate (grasp, move, etc.) objects. VR / AR / MR applications may need to understand the world to avoid obstacles as the user moves and to add semantically plausible virtual objects to the environment. In other respects, semantic object recognition can be used for many different applications that require identifying objects within an image.

[0010] To perform these tasks, many machine learning techniques are being used, such as deep learning-based methods, which train neural networks to perform computer vision tasks for object recognition, such as object classification, object detection, spatial object segmentation, and semantic object segmentation (or recognition). Object recognition techniques may include multiple such operations that often overlap or are combined to be performed by a single algorithm or neural network technique.

[0011] Traditional object recognition techniques have several drawbacks. First, many traditional deep learning networks are trained on predefined datasets with a fixed number of object types. Currently, it's impossible to construct a dataset containing all objects in the world, making it difficult to train a general-purpose network to encompass everything. For example, Mask Region Convolutional Neural Networks (Mask R-CNN) detect and segment 80 categories, and YOLOv9000 detects 9000 objects. While 9000 is a large number for current datasets and classifiers, it remains relatively small compared to the millions of object types in the world. Therefore, from one perspective, the training database for traditional object recognition neural networks is far too small.

[0012] Furthermore, some techniques, known as one-shot video object recognition and segmentation, are used for small datasets and user-defined object understanding scenarios. In this technique, the user can request any object in an image to be identified and segmented. This one-shot technique is typically trained using a single annotated frame that semantically labels the object within a sequence of video frames. However, this one-shot neural network technique is prone to overfitting due to the single annotated frame and cannot handle object variations in appearance, shape, etc., within its dataset. Moreover, this one-shot technique is limited because it cannot handle novel scenes, including new objects and backgrounds that it has not yet encountered.

[0013] Secondly, from another perspective, the training databases for neural networks in traditional object recognition and segmentation techniques are too large. In particular, the performance of deep learning techniques largely depends on the aforementioned large amounts of carefully annotated training data. In many applications, the objects of interest vary from task to task, and the neural network must be trained to handle all tasks. Therefore, building separate datasets for each individual task, for example, by training a general neural network for each specific task, can be extremely expensive and time-consuming, and thus impractical.

[0014] Third, traditional neural network object recognition and segmentation techniques are not adaptable. User needs for certain object types are constantly changing. For example, in the field of elderly care, seniors can use robots with computer vision to help identify or even dispense medications. When such actions are required, these systems typically lack any way for users to add images of new bottle types (new bottle shapes and printed labels on the bottles) to the neural network database.

[0015] Many traditional techniques attempt to address these problems. This includes OSVOS (One-Shot Video Object Segmentation) (see Caelles, S. et al., “One-shot Video Object Segmentation”, CVPR (2017)). OSVOS uses a semi-supervised approach for video object segmentation. For each video, only the first frame is annotated. This traditional technique pre-trains a segmentation neural network on large datasets and then fine-tunes the network on a given annotation. This approach works well with simple videos but is prone to overfitting on the initial annotated frames, and therefore cannot handle large changes or deformations of objects in the foreground or significant changes in the background. Furthermore, this technique does not provide users with a way to customize and extend the network.

[0016] A one-time segmentation technique that uses augmented data to fine-tune deep neural networks is Lucid DataDreaming (see Khoreva et al., “Lucid Data Dreaming for Object Tracking”, CVPR workshop, (2017)). It synthesizes training data by simulating foreground and background changes in lighting, deformation, motion, etc., and then synthesizes new data for training. This is achieved through in-painting on a dynamic background. However, the transformations in this technique are relatively limited because it cannot adequately segment large changes in both foreground and background objects (referred to here as unexperienced objects) that have not yet been included in the neural network training database. Large deformations are difficult to simulate, and unknown new objects are difficult to predict. Therefore, this method is not robust to large appearance changes (especially on backgrounds). Furthermore, Lucid DataDreaming does not provide users with the ability to customize and expand the neural network training database.

[0017] Another technique does perform automatic growth on the neural network training database, but the added data is unreliable. Specifically, Online Adaptive Video Object Segmentation (OnAVOS) (see Voigtlaender, P. et al., “Online adaptation of convolutional neural networks for video object segmentation”, BMVC (2017)) is an updated online adaptation version of the OSVOS database that adds foreground variations to the training database using its own output in the form of a self-looping scheme. Performance for appearance changes is slightly improved by using more training data, but the accuracy of the added training data cannot be determined. Inaccurate labels can lead to artifacts, and segmentation errors can accumulate as videos are analyzed and the training data evolves. Therefore, artifacts are likely to be unnecessarily considered as positive samples for the training database, and as videos are analyzed, artifacts can then spread throughout the training iterations.

[0018] To address these issues, this paper discloses a one-off, semi-supervised object recognition and segmentation system and method that operates efficiently on small training datasets and can be easily customized by the user to add new objects of interest. The disclosed method collects and accumulates training data as the input video is analyzed instance-by-instance and frame-by-frame for training a neural network training database, which itself automatically generates reliable annotations for training the instance segmentation neural network during a custom setup mode. The instance segmentation neural network is a semantic segmentation neural network, which typically provides a segmentation mask for individual instances, one instance at a time, and here the instance can be a user-defined instance. The instance segmentation neural network provides fine (as opposed to coarse) pixel-level boundaries and a semantically recognized label during runtime or inference mode. However, during the custom setup mode described herein, semantic recognition is not yet required because the instance labels are already known during the custom setup mode, as explained below.

[0019] In this paper, the instance segmentation neural network is adapted to each frame by fine-tuning itself. This is achieved by providing three specific features during the custom setup mode. First, the user can add desired objects of interest to the neural network training database; second, a general knowledge (or universal) neural network is used to provide candidate masks; and third, the instance segmentation neural network is trained on the database using local and global neural network adaptation techniques. During custom mode, the instance segmentation neural network provides segmentation masks, which can be binary masks, and these masks are compared with candidate masks to determine the quality of the candidate masks for selection and addition to the training database. Once trained, the instance segmentation neural network is used during runtime (or inference) to provide annotated output masks as semantic object recognition results.

[0020] More specifically, a user-customized object segmentation solution or custom setup mode is proposed, in which the end user of the image capture device and image processing system (e.g., a smartphone) can define their own object of interest, for example, by providing only one annotation per object on the first frame of a video sequence of one or more objects. Regardless of the object the user desires, this method only requires one frame of the input video sequence to be annotated (or an initial mask to be provided) to automatically generate the corresponding training data and adapt the instance segmentation neural network to the object variations. The user does not need to manually construct excessively large predefined and fixed training datasets for the desired objects and for custom training. This custom setup mode is conveniently performed by the end user to customize the neural network training database with the objects to be identified according to the end user's needs. This is performed by the user, after, for example, the manufacturer has pre-trained the neural network by default, but before actual use (or runtime or inference) to identify those desired objects. The custom setup mode can also be referred to as a network fine-tuning mode.

[0021] Such systems and methods also include operations of general knowledge or universal neural networks to provide candidate masks for the desired object. Thus, such a network receives a user video sequence with first annotated frames as input. A general knowledge network is then used to provide reliable, automatic generation of training data for the neural network. The general knowledge network generates possible training data in the form of candidate masks, and this training data is robust to appearance changes, large transformations, occlusion, etc. Then, a quality metric is determined by selecting candidate masks that pass certain criteria to select only qualified data for training.

[0022] During pre-training, the general knowledge neural network is based on one or more large general datasets and is retrained or fine-tuned in a class-insensitive manner to adapt the general neural network training database to any object type. Since one (or some other selected low number) frames are annotated on the input video sequence, and therefore the target object on (one or more) of the input annotated frames is augmented (or modified) to form many training frames for each object to avoid overfitting. Furthermore, the general knowledge neural network is fine-tuned by limiting the augmented data to a small subset of the neural network by layers and / or epochs (where an epoch is a single pass of all training data through the network). Thus, the general knowledge network will be able to identify desired objects of interest, but it will also be able to detect and segment other novel objects.

[0023] Unlike traditional deep neural networks with fixed capabilities after training, the instance segmentation neural network here grows in capability as data is accumulated and added to the neural network training database based on a two-level network adaptation strategy. The instance segmentation neural network is automatically and gradually (or in other words, frame-by-frame) updated during customized fine-tuning and adapted to new scene and object variations in both the foreground and background. This is achieved using both local and global adaptation techniques, where local adaptation emphasizes training samples from nearby frames (relative to frames further away in time along the video sequence) for training based on the current frame, while global adaptation uses samples from all or substantially all frames of the video sequence. Specifically, in local network adaptation, the instance segmentation network is primarily fine-tuned on nearby frames to achieve good performance on the next frame, which is immediately beneficial, allowing the method to select good candidate masks during the quality measurement phase. In global network adaptation, the instance segmentation neural network is fine-tuned on all generated data (for instances and for all frames in the video sequence analysis) to adapt to different variations. This can include using some samples from the same samples used by local adaptation from the training database, allowing training to be performed using multiple epochs of the same object. This results in improved accuracy of the neural network by utilizing all training data to train on object variations.

[0024] Therefore, it can be said that, compared to traditional methods, this method iterates between neural network adaptation and data generation operations to provide better quality (more accurate) on objects with greater variation. By accumulating training data in a customized mode using the proposed method, the instance segmentation neural network is fine-tuned on this data to improve performance and is instructed to emphasize the segmentation of (one or more) target objects by training on the accumulated data of the objects. As mentioned above, using the instance segmentation network in conjunction with a general knowledge (universal) neural network (instead of using a general neural network alone) results in significantly fewer false alarms and better contours on object boundaries. Compared to both traditional methods and general knowledge networks alone, the instance segmentation network significantly improves performance based on both data generation and network adaptation.

[0025] refer to Figure 1Image processing system 100 may include object recognition system 102 to perform the object recognition methods described herein. Image processing system 100 may also have image and annotation input unit 104, which may or may not be considered part of object recognition system 102. Object recognition system 102 may optionally have initial instance detection unit 106 as described below, but additionally has data generation unit 108, instance segmentation unit 110, and neural network training database 112. In customized or fine-tuned mode, object recognition system 102 performs an iterative loop or circular training process to train instance segmentation neural network on objects added to the system by the user during this mode. To perform this process, data generation unit 108 receives an image with the desired object and provides candidate masks for the object. Quality evaluation is performed to determine which candidate mask is the most accurate mask, and the selected mask is then added to training database 112 to make the added mask or data reliable. Instance segmentation unit 110 obtains the added objects and performs an adaptation process to train instance segmentation neural network 120 with those new objects and outputs segmentation masks for those new objects. Then, quality assessment is performed using these segmentation masks by comparing them with candidate masks of the same instance.

[0026] Now, let's first discuss the pre-training mode of system 100 in more detail. Data generation unit 108 has a general neural network (or simply net) unit 114 operating a general neural network (Gen. Net) 116, and instance segmentation unit 110 has an instance segmentation neural network (net) unit 118 running an instance segmentation neural network (Ins. Seg. Net) 120. Both units can include pre-training modules for generating initial or default neural networks, or they can communicate with remote pre-training modules to generate initial neural networks 116 and 120. The initial neural networks 116 and 120 are then uploaded, respectively, to a general knowledge network unit 114 and an instance segmentation network unit 118 on a computing device with the logic and software of an object recognition system. The general knowledge network unit 114 can be fine-tuned remotely or on the device. Pre-training can include initial training on a public (or general knowledge or universal) dataset, followed by fine-tuning the network on augmented data for each instance. Details of pre-training are provided below via process 300.

[0027] In the custom setup mode, the user (or a user-initiated automated program) records one or more target objects to be identified during runtime or inference mode via video recording. The image and annotation input unit 104 receives images from the user in the form of a video sequence, and then inputs annotations from the user for the first frame of the video sequence, but these can be different frames or a much smaller number of frames than the total number of frames in the video sequence. In one form, the uploaded first image is displayed to the user on the interface, and the user draws boundaries around the instances (or objects) to be identified (e.g., using a different color for each object), and then inputs annotations for each identified instance. This automatically generates a count of the number of instances to be trained during the custom setup mode. If other instances appear in the uploaded video sequence, these instances are ignored. Furthermore, if one or more instances disappear from the recorded field of view during the video sequence, it will not affect object recognition. For example, suppose there are two instances in the input video sequence. After the first frame of the video sequence, the custom setup mode segments these two target instances regardless of the actual number of instances. Therefore, if one of these two instances disappears from the image, the custom setup mode object segmentation is still performed twice, once for each object. For missing instances, the output of the instance segmentation neural network provides an empty result. In one form, the user can be instructed to record one or more objects preferably from different perspectives. The image data can be preprocessed by this unit or other units sufficient for object recognition operations, as described below.

[0028] Alternatively, after the user uploads the video sequence, the initial instance detection unit 106 then analyzes the frames of the video sequence, particularly the first frame of the video sequence, and can provide at least a coarse object segmentation for each frame, provided that the position of each object or instance on a single first frame is identified and counted.

[0029] Subsequently, the object recognition system 102 performs object recognition (including instance segmentation) instance by instance by analyzing the entire video sequence to fit instance segmentation of one instance at a time, as described in detail below. The analysis of the video sequence can then be repeated for each instance, or the video sequence can be analyzed separately in parallel for each instance.

[0030] Specifically, for a single instance, each frame of the video sequence is analyzed, and this is repeated iteratively multiple times for each instance, for example, three times. The general knowledge network unit 114 inputs the images from the video sequence frame by frame into the general knowledge neural network 116, and provides a set of candidate masks j for the current frame t and for a single instance, where each candidate mask is specified as... C tj .

[0031] The instance segmentation network unit 118 also receives frames from the video sequence and analyzes the image data of those frames, one frame at a time, for the same instance being analyzed by the general knowledge network unit 114. The output of the instance segmentation neural network 120 is a segmentation mask. M t One form of this is a binary segmentation mask. Both candidate masks and segmentation masks are provided to a quality evaluation unit 122, which may be part of a data generation unit 108. The quality evaluation unit 122 compares these masks and determines which candidate mask is most similar to the segmentation mask. This can be determined by comparing the difference between the segmentation mask and each candidate mask against a threshold. In one form, this involves comparing a quality score against a threshold formed by a color histogram and calculating a weighted Jaccard similarity coefficient or index (or simply Jaccard) between the candidate and the segmentation mask, as described below. This score is then compared against a threshold or other criterion to select one of the candidates as the best candidate to be added to the training database 112, also referred to herein as the added mask, which can become a sample to be used to adapt the instance segmentation neural network (or ISNN) 120. As an example, the training database 112 only stores the added mask, and the initial pre-training datasets for the general knowledge neural network and the instance segmentation mask remain separate from the training database 112. These initial datasets can be fixed and / or updated individually over time relative to the custom setup pattern described in this paper.

[0032] Moving on to the adaptation of the instance segmentation neural network 120, two adaptation techniques are used: local adaptation and global adaptation. Local adaptation adapts the instance segmentation neural network to output a fine-grained segmentation mask on the target frame based on training database samples from other frames, according to the temporal proximity of the sample frame to the current target frame being analyzed along the video sequence. The closer the frames are in time, the more likely the instance segmentation neural network (or ISNN) 120 samples will be adapted to the samples of that frame. The local adaptation unit 124 will determine which training database samples to select for adaptation of the instance segmentation neural network, as described below.

[0033] Global adaptation unit 126 adapts all samples from all frames of the video sequence (for the current instance) to ISNN 120. Global adaptation unit 126 performs this adaptation after a single iteration, tracked by iteration counter 128, has been analyzed for the entire video sequence or a specified multi-frame portion of the video sequence. Similarly, this is performed for a set number of iterations for a single instance. This process is then repeated for each instance.

[0034] In online runtime (inference) mode, runtime (inference) input 130 is provided to instance segmentation network unit 118 for running ISNN 120. Conditional random field (CRF) classifier 132 collects binary masks for each instance and combines them into a single frame to output a frame-specific instance mask that can be used by other applications, whether computer vision applications or other applications.

[0035] refer to Figure 2 A process 200 for object recognition in imaging processing is provided. In the illustrated implementation, process 200 may include one or more uniformly numbered operations, functions, or actions 202 to 212. As a non-limiting example, this document may refer to Figure 1 Example image processing system 100 or Figure 9 The process 200 is described using an example image processing system 900 (where relevant).

[0036] Process 200 may include “obtaining image data of frames from a captured video sequence, including the content of at least one object to be identified and added to a neural network training database for object recognition or segmentation, so as to customize the neural network training database by including at least one desired object for a specific user” 202. This operation may include the user activating the customized setup mode by either uploading the video sequence to an application or program, or by initiating the mode via a signal, electronic switch, or other indication on such an application or program. This may involve the user capturing a video of one or more objects that the user wants his / her device (e.g., a smartphone) to be able to recognize. The user may manually capture the images using a smartphone camera or other device, or the computer vision system may capture the images automatically, for example, on a system that sets up a camera for such an arrangement. An interface may be provided to guide the user on how to capture images and input labels or annotations for objects in the images. In one form, annotations may only mark objects on the first image of the video sequence. The result is a video sequence of image data, which will be used to customize the training neural network for an object recognition or segmentation system. This operation may also include obtaining the video sequence in the form of preprocessed raw image data with RGB, YUV, or other color space values, in addition to brightness values ​​for multiple frames of the video sequence. This operation can also include obtaining depth data when depth data is used for segmentation analysis.

[0037] Alternatively, although the examples discussed herein will receive video sequences from a user, it will be understood that the decision to first capture video for object recognition training purposes can be fully automatic, where a computer vision device or application may be unable to identify objects in an image and decide to prompt the user or other application to make identification annotations or labels, and then the image and annotations are provided to the object recognition system presented herein. For example, class-insensitive object detection and / or segmentation methods may be performed first (see, for example, Uijlings, JRR et al., “Selective Search for Object Recognition”, International Journal of Computer Vision, Technical Report (2012); and Vande Sande, Koen EA et al., “Segmentation as Selective Search for Object Recognition”, International Journal of Computer Vision, Vol. 104(2), pp. 154–171 (2013)). Then, for each detected and segmented object, a conventional object recognition, detection, and / or segmentation algorithm can be run, such as counting objects trained on a custom setup (see, for example, Redmon, Joseph et al., “Yolov3: An Incremental Improvement”, arXiv (2018)). This algorithm can also be used for the aforementioned automatic segmentation using the initial instance detection unit 106. If automatic recognition and instance counting fail, the user can be prompted on the interface to manually identify and annotate the desired objects in the captured video.

[0038] Process 200 may optionally include "inputting frames into a general knowledge neural network associated with a general database to output multiple candidate masks associated with instances of objects" 204. This may include, for example, generating a general neural network based on a known general database. The database, and consequently the neural network, can be fine-tuned during a preprocessing mode to improve the accuracy of the neural network, thereby identifying more variations of objects already in the database and improving the accuracy of identifying objects not yet present in the database. Subsequently, image data from the user can be received during a customized or fine-tuning mode. In one form, the image data has been run through an initial segmentation algorithm to provide a count of objects or instances in the image. The general knowledge neural network can then analyze the image instance by instance and output a set of candidate masks for each individual instance.

[0039] Then, process 200 may include "customizing the training database by adding an added mask to the training database, and the added mask is one of a plurality of candidate masks" 206, and therefore, the method then selects one of the candidate masks to add to the training database. Each mask may be at a binary, grayscale scale, or other scale that reveals which pixels are part of an object and which pixels do not indicate the object's boundary.

[0040] This operation may also include performing a quality evaluation to provide a quality score for each candidate mask. One approach is to include a comparison of one candidate mask with a segmentation mask from an instance segmentation neural network. In one form, a score is given for each candidate mask. As an example, the score may include a comparison of color histograms and a calculation of a weighted Jaccard, but other algorithms may also be used. The scores can be compared to a threshold or other criteria to determine which candidate mask to add to the training database.

[0041] Once a candidate mask is selected, the candidate mask, the corresponding image or frame of the object (or instance) forming the mask, and the annotations for the object or instance are stored in the training database as part of a custom dataset.

[0042] Simultaneously, process 200 may also include "inputting frames into an instance segmentation neural network to output a segmentation mask" 208, and this may include providing the same user video sequence to the ISNN, which also operates on epochs one instance at a time. The ISNN receives individual frames as input frame by frame and outputs a segmentation mask for each instance and for each frame. During the custom setup mode, the ISNN performs spatial recognition but not semantic recognition because the ISNN is operating on a single instance basis and annotations or labels are known. Also during this custom setup mode, the segmentation mask is used to compare with candidate masks for quality evaluation, as described above. Details of the ISNN's structure are provided below.

[0043] To train the ISNN for customized objects added to the training database, process 200 may further include "modifying the instance segmentation neural network to form a modified instance segmentation neural network, wherein said modification is at least partially based on at least one added mask in the training database" 210. As an example, this could include a local adaptation operation or a global adaptation operation. In one form, both operations are used. This involves running the ISNN on samples (or added masks) that have been added to the training database but are provided over multiple frames. For local adaptation, local adaptation is performed for each frame, and the closer a frame is along the video sequence to the current frame being analyzed, the more likely the ISNN is to adapt by using samples from that frame or by adding a mask. For example, each sample from the same instance and within the five closest frames may have a 70% chance of being used, while samples from frames that are temporally farther from the current target frame may only have a 30% chance of being used. Alternatively, a fixed interval or other fixed interval can be used, such that each sample of the most recent 10 frames is used, then every other frame for the next 10 frames, then every 5 frames for the next 100 frames, and so on.

[0044] A global adaptation is run once at each iteration across the entire video sequence, and an ISNN is run on samples of the same instance for each frame in the video sequence. This replicates the samples, allowing both the global and local adaptations to use the same samples from the training database, which, as mentioned above, leads to stronger recognition of those specific samples.

[0045] Process 200 may include “performing object recognition during inference runtime using a modified instance segmentation neural network” 212. This refers to an online runtime (or inference) mode following training during a custom setup mode, such that an image is now being captured for object recognition itself. During this mode, an ISNN output segmentation mask is used as the final object recognition mask. This can be provided to a classifier that collects all instances of the same frame and forms a single mask for the entire frame, as described below. This frame-based mask can then be provided for further fine-tuning, or it can be provided to other applications using object recognition (e.g., autonomous vehicle systems, various reality systems (virtual, augmented, etc.)) or other applications identifying objects in an image (e.g., the drug recognition program mentioned herein).

[0046] Therefore, this semi-supervised video object segmentation method provides a user-accessible customizable setup mode that performs data generation and neural network adaptation. The system does not require a large, domain-specific (annotated) dataset for training. Instead, given only one annotation, the system generates reliable training data from unlabeled videos. Furthermore, as more data accumulates during the customizable setup mode, the instance segmentation neural network improves its object recognition capabilities. In this mode, as frames of the video sequence are analyzed, the instance segmentation neural network gradually adapts to changes in both the foreground and background data. Based on these two advantages, the proposed method can be applied using small datasets and user-defined scene object understanding.

[0047] refer to Figures 3A-3C A process 300 for an object recognition method for imaging processing is provided. In the illustrated implementation, process 300 may include one or more operations, functions, or actions 302 to 366, numbered evenly. As a non-limiting example, this document may refer to Figure 1 Example image processing system 100 or Figure 9 The system 900 (where applicable) describes the process 300.

[0048] During the pre-training mode, process 300 may initially include “performing pre-training of a general knowledge object recognition neural network” 302. A general knowledge (or universal) neural network is used to generate candidate instance masks. The input to this neural network is images of a video sequence, where only the first image (or a small number of other frames, such as at intervals, etc.) is annotated, and the output is a set of candidate masks. To learn general knowledge about the appearance, shape, and other variations of objects in a category-insensitive manner so that the general neural network database is adaptable to any object type, the neural network can first be trained on large, universal datasets such as ImageNet (see Russakovsky, O. et al., “Large Scale Visual Recognition Challenge”, airXiv: 1409.0575v3 (2015); Fei, L. et al., “Analysis of Large-Scale Visual Recognition”, BayArea Vision Meeting (2013)) and COCO (Lin, TY et al., “Microsoft COCO: CommonObjects in Context”, ECCV (2017)). In this implementation, the general knowledge neural network has an architecture in the form of Mask R-CNN.

[0049] Because one (or some other selected low number) of annotated frames are used on the input pre-trained video sequence, the target object on the input annotated frames is augmented (or modified) to form thousands of training frames for each object to avoid overfitting. Similarly, by way of an example, the augmented data can be used only on the last two layers of the general knowledge neural network for fine-tuning and for a few epochs. As a result, due to training on the augmented data, the general knowledge neural network is able to handle more variations of the objects it has already trained on, but it is also able to detect and segment other new objects that it has not yet experienced. Then, by simulating foreground changes such as lighting, deformation, etc., the new data is synthesized into the general knowledge neural network using the Clear Data Dream methodology (as described above). The pre-training dataset of the general knowledge neural network is kept separate from the training database 112 used for instance segmentation training during the custom setup mode as described above.

[0050] Similarly, process 300 may initially include “performing pre-training of the instance segmentation neural network” 304. Here, the instance segmentation neural network (ISNN) uses a parent network in OSVOS as the instance segmentation neural network. Weights pre-trained on ImageNet (as described above) are used as the base instance segmentation neural network, and this network is fine-tuned on the binary segmentation dataset of DAVIS (see Caelles, S. et al., “The 2018 DAVIS Challenge on Video Object Segmentation”, arXiv preprint arXiv: 1803.00557 (2018)). As with general knowledge neural networks, a training instance of an annotated image is augmented to hundreds of additional training images. The instance segmentation neural network is then fine-tuned to fit each instance based on its pre-trained augmented dataset. Furthermore, this paper uses the Clear Data Dream method (as described above) to synthesize new data by simulating foreground changes such as lighting, deformation, etc. The instance segmentation network is denoted as N in this paper for process 300 and is referred to as Algorithm 1 in the summary described below.

[0051] The pre-training modes for the two neural networks can occur at least partially offline and remotely from the camera providing the video sequences during the custom setup mode, and remotely from the mobile device or other computing device or server that will perform the object recognition analysis described herein. In one form, the initial training of the neural network using a public or general dataset is performed offline, but augmentation and fine-tuning of the images during pre-training can be performed on a mobile device. In another form, although all or part of the preprocessing is remote, both the custom setup and runtime (inference) modes are performed on the same device, and in one form, on the same device as the camera capturing the video sequences for the custom setup mode. In yet another form, all three modes are performed on the same device, which may or may not be the same device that captured the images for the video sequences.

[0052] During the customization (or fine-tuning) mode, process 300 may include “acquiring image data of a video sequence” 306. This may include “acquiring images” 308. This involves the user identifying one or more objects that he / she wants to be recognized by his / her device (whether it is a smartphone, tablet, computer, or webcam, or any other camera that could be a specific object recognition system such as on a vehicle or VR goggles, etc.). The user can then manually position the camera to record the object video and, in one form, obtain different views of the object from different angles. More than one object may be captured in the camera’s field of view. The video sequence should capture as many different viewpoints of the object as possible, and the video should have at least approximately 300 to 500 frames (10 to 20 seconds of video at 25 fps). Alternatively, the object may be placed in front of a preset arrangement of one or more computer-controlled cameras, or the camera arrangement may be placed next to one or more objects, and the user may activate one or more cameras to automatically capture images of the object. This includes multiple cameras or a single camera that automatically moves to capture different views of the object, or, for example, an object on a turntable. Websites, mobile applications, or other screen interfaces can be used by users to activate customized setup modes and to confirm and analyze video sequences from the user, as well as identify desired objects. Alternatively, activation of this camera arrangement can be automatic or as described above.

[0053] Process 300 may include “performing preprocessing” 310. Whether the uploaded image is to be displayed to a user so that the user can select and identify objects to be recognized, or whether initial object recognition is to be performed automatically, sufficient preprocessing can be applied to the image for object recognition operations. Preprocessing may include, as needed, de-mosaicing, noise reduction, pixel linearization, shadow compensation, resolution reduction, vignetting removal, and / or 3A-related operations, including automatic white balance (AWB), autofocus (AF), and / or automatic exposure (AE) modifications. Furthermore, the color and brightness pixel values ​​of the image can be provided in many different additional forms, such as gradients, histograms, etc. Additionally, when used, depth image data can be determined by a stereo camera system (e.g., utilizing an RGBD camera) that captures images of the same or moving scene from multiple angles. This system can perform multiple calculations to determine the 3D space of the scene in the image and the depth dimension for each point, pixel, feature, or object in the image. Other methods for determining 3D space from a single camera are also possible, such as time-of-flight, structured, or coded light techniques.

[0054] Then operation 306 can also include “Get Annotation” 312. The video sequence should have one annotation frame, and in one form, only the first frame has an annotation. Optionally, more frames can have annotations, but this may be less efficient and require more unnecessary semantic computation. In one form, the user provides a single label as an annotation for each object or instance to be identified on the annotation frame by drawing a boundary or edge around each object to be identified, and a field for inputting the label may appear for each such object. For each boundary of different objects, the interface may have different boundary type cues for the user, for example, distinguishing objects from each other by color or line type (such as dashed lines). The annotation can be a single word or multiple words. Websites, applications, or interfaces for inputting images may also provide a screen for inputting annotations for each object on the first image of the video sequence.

[0055] Then, process 300 may include "obtaining instance counts" 314. Instance counting can be performed automatically and in real-time as the user is identifying objects and entering annotations. Alternatively, instance counting can be performed automatically once the user uploads the video, or object selection can also be performed automatically. In this case, once the images are received and preprocessed sufficiently for object detection, initial object segmentation can be applied to determine how many objects are on a single image in the video sequence and to provide a coarse object location for each object. This can be performed using known algorithms mentioned in this paper.

[0056] Regardless of how the instance count is obtained, the total number of instances on a single image can be built up to the maximum number of instances (the maximum number of instances).I =i). This system can be described as per instance. I Generate individual instance segmentation neural networks. Therefore, assume the video contains... I If there are 1 target instance, then the instance partitioning unit contains 1 target instance. I Each segmentation network, and custom object recognition will be run separately. I Alternatively, users can define regions of an image for each object on the interface described above, for example, by placing bounding boxes around each object in the image and then indicating the total number of objects to be identified and trained on the interface.

[0057] Then, process 300 may include "setting up an instance" I =1”316 to set the instance count and start with the first instance on the image. During custom settings mode, the same video sequence (whether in parallel or sequential) will be analyzed separately for each instance.

[0058] Then, process 300 may include "set iter=1" 318 to set the iteration count. Specifically, the method iteratively performs the following operations: repeating the analysis of the video sequence for each iteration and performing multiple iterations for each instance, such that each instance has multiple analysis iterations. Each iteration should further fine-tune the instance segmentation neural network. The number of iterations is determined experimentally and methodologically, with three iterations used. Each iteration includes operations from both the data generation unit and the instance segmentation unit.

[0059] For iteration I The first operation is to "initialize the training database". D "320, where the training data set is initialized by adding input annotations to the dataset in the database." D This will also iterate instance segmentation neural networks. N iter Configured as an instance segmentation neural network N Same (here among them) N In its initial or default form, but subsequently adapted through local and global adaptations as explained below. First, the default instance segmentation neural network... N It is formed through pretreatment.

[0060] Then, process 300 may include "set frame t=1" 322, where frames can be indexed from t=1 to T, and for the first frame 1 of the video sequence and for the current iteration iter=1 and instance. I =1 to begin analysis.

[0061] Next, process 300 may include "inputting frames t of the video sequence into a method specific to the instance".I In the general knowledge neural network, "324" refers to the operation of a general knowledge network unit that receives input frames and generates training data, and in a form that occurs on the user's device and for each frame during the custom setup mode. Alternatively, the neural network processing during the custom setup mode can be performed remotely, for example, on a server, such that the input image is sent to the server and the network's output is sent back to the user's device. As mentioned above, the general knowledge neural network can be a pre-trained Mask R-CNN network, but many other network architectures can also operate. The output of the general knowledge neural network is a candidate mask C with K candidates. tj A set of masks, where j = 1 to K. Each mask is binary, grayscale, or another scale used to indicate the pixel-level location of object boundaries. Thereafter, process 300 may include "obtaining candidate masks for frame t from a general knowledge neural network" 326.

[0062] Moving to the instance segmentation unit for several operations, process 300 may include "inputting the same frames of the video sequence into the instance segmentation neural network" 328; thus, frame 1 is also initially input into the instance segmentation neural network. The structure of the instance segmentation neural network is as described above for preprocessing, but in other cases it will be modified as it adapts to new data from the data generation unit, as described herein. Due to instance-based operations, the instance segmentation neural network is adapted individually for each instance and evolves frame-by-frame and iteratively for each instance, as described below. The instance segmentation neural network can provide a mask at a binary or other grayscale or color scale, and when binary, it may be referred to herein as a binary segmentation network or simply a segmentation network. Therefore, each instance can have a different segmentation network (or instance segmentation neural network (ISNN)).

[0063] The output of the instance segmentation neural network is a segmentation mask. M t It displays the boundaries of instances or objects and associates them with annotations for the corresponding objects. In one form, a mask... M t This can be a binary mask. This mask is used for quality assessment. Therefore, process 300 can include "obtaining a single mask". M t 330.

[0064] Process 300 may include "determining the best quality candidate mask" 332, which refers to a quality metric or evaluation used to measure how likely it is that the annotations, shape, and appearance of the candidate mask are correct. To this end, process 300 may include "determining the mask..." M t and each candidate mask C tjThe comparison score between candidates and masks is 334. The quality score represents the score for each candidate and mask. M t The similarity between them. Using an example from this article, two features can be used for measurement: candidate and mask. M t Jaccard The similarity is calculated using the intersection of the union and the color histogram. The quality score is represented by a weighted sum of these two similarities: in, score tj It refers to the quality score of the j-th candidate in the t-th frame. f It's a color histogram. C tj It is the first t Candidates at frame, M t It is a mask for an instance segmentation neural network. It is Jaccard, and β It is set in this implementation method β The weighting at =1.5 (which was determined experimentally).

[0065] Process 300 may include "comparing the score to a standard" 336. As an example, comparing the score to a threshold... th quality The threshold is determined through a trial-and-error process. This threshold can be fixed throughout the analysis for all instances and iterations, or it can vary depending on the instance or iteration to add more samples to the training dataset. In other words, the reason for modifying the threshold is to try to select only reliable samples to add to the training database at each iteration. For example, in the early stages, the output of the instance segmentation neural network may not be very accurate. In this case, a high threshold is used to limit the selection of training data to those samples with very high confidence values. In later stages, as the instance segmentation neural network becomes more and more accurate, the threshold can be lowered to add more samples to the training dataset, as more samples will be accurate. The selected top candidate mask... d t It is a candidate mask with the highest quality score, and this score is compared with a threshold. If the quality score is higher than the threshold... th quality Then process 300 may include "adding the best candidate mask to the training database" 338, where the best candidate mask d t Attached to the training database (or training dataset)D = D U d t This is also known as masking. The best candidate masks are accumulated in the training dataset from the first frame to the last frame as the video evolves, forming a customized masked dataset, thus customizing the training database.

[0066] Now we turn to fine-tuning and adapting the instance segmentation neural network, using training data accumulated at the training database to update the instance segmentation neural network at two levels: local adaptation for achieving good segmentation on a single frame, and global adaptation for learning and adapting to dynamic frame-to-frame changes in the video.

[0067] Therefore, process 300 may include “performing local adaptation” 340, specifically adapting the instance segmentation neural network (ISNN) to the added masks in the training database. As the video evolves in each frame and local adaptation is performed at each iteration, the local adaptation is adapted to the network. N iter Fine-tune the instance segmentation neural network at the i-th iteration.

[0068] To perform local adaptation, process 300 may include "using a larger number of samples when the frame is closer to the target frame being analyzed" 342, such that local adaptation uses a high sample rate for data from nearby frames and a low sample rate for data from distant frames. Specifically, for each iteration of a single instance, there is one instance (or object or associated added mask) on a frame. Added masks (or instances or objects) in frames closer to the current target frame being analyzed along the video sequence have a higher probability of being selected as training data for fine-tuning (or adapting) the instance segmentation neural network. Therefore, instances from closer frames are more likely to be selected as training data. In other words, for a single instance, fine-tuning may include hundreds or thousands of frames in the video sequence being analyzed for multiple iterations. At each frame, an instance sample can be selected from the training database (or dataset) to fine-tune the instance segmentation neural network. Each instance in the dataset is selected with a different probability. In one example used in this paper, a probability of 0.7 is assigned to the 5 closest frames along the video sequence, and a probability of 0.3 is assigned to other more distant frames. Therefore, if fine-tuning requires 100 frames, the system selects instance samples from the training database 70 times from the most recent 5 frames and selects instance samples 30 times from the other frames. The probability values ​​(0.7 and 0.3 in the current example) are determined experimentally.

[0069] Process 300 may include “performing adaptation by inputting an image corresponding to the added mask and simultaneously using the added mask as a supervised output” 344. When the top candidate mask (or added mask) is saved to the training database, the corresponding color image is also saved. These images are the input training data used to fine-tune or adapt the instance segmentation neural network, while the saved added mask is a supervised spatial mask output with known annotations because the instances are known.

[0070] Process 300 may include querying "last frame t?" 346, which determines whether the last frame of the video sequence has been reached. If the last frame has not been reached, process 300 sets frame t 348 as the next frame in the video sequence and loops back to operation 324 to repeat the semantic network adaptation for the next frame, still using the same instance. I And still in the same iteration iter It will be carried out internally.

[0071] If the last frame in the video sequence has been reached, process 300 may include “performing global adaptation” 350, specifically “obtaining masking data from a training database of all frames in the video sequence” 352, which is specifically for the current instance being analyzed. Global neural network adaptation learns the dynamic changes of each instance across the entire or substantially the entire video. Therefore, in each iteration iter At this point, after analyzing all frames, each instance segmentation neural network... N Fine-tuned on all (or nearly all) of the added masks for each frame in the video sequence (or in other words, the entire training dataset of added masks for the video sequence of the current instance).

[0072] Then the adapted instances are segmented into a neural network. N Used as the initial segmentation network in the next iteration. For this purpose, process 300 may include "running the globally modified instance segmentation neural network as the initial instance segmentation neural network for the next iteration" 356.

[0073] refer to Figure 4 As a summary diagram 400 of the adaptation operation, the segmentation network can be updated at two levels: local adaptation and global adaptation. Local adaptation is performed in each frame t=1 to T, as shown in video sequence 402, as the video evolves. Local adaptation fine-tunes the segmentation network on training data from nearby frames (stored in database 404) to achieve good segmentation results. After processing all frames, global adaptation is performed by fine-tuning the segmentation network for the current instance on all training data 404 from the video sequence to learn the dynamic changes of the video. Both global and local adaptations are iterative, as summarized in Algorithm 1 below, and as described above, where... NIt is a segmentation network initially trained offline and / or during preprocessing, but will be updated at the end of each iteration, and N iter This refers to the segmentation neural network that is updated for each frame. Pyramid 406 represents the adapted instance segmentation neural network, which, following operation 356 above, provides the latest segmentation neural network for the next iteration of frame analysis. N .

[0074] Process 300 may include "updating the quality metric threshold" 358, and this is performed by reducing the threshold determined experimentally for the reasons described above. As an example, the threshold may be reduced by 10%. As more global adaptations are performed through more iterations, the instance segmentation neural network output has increasingly better contours, making it possible to reduce the threshold to add more samples to the training dataset, as described above.

[0075] Then, process 300 may include querying “maximum iteration iter?” 360 to determine if the last iteration has been reached. If not, the iteration counter is incremented by 1 at counting operation 362, and the process loops back to operation 320 to initialize the training database and begin analysis again with the first frame of the video sequence. Initialization of the training database includes the tasks described above for operation 320. If the last iteration has been reached, process 300 may include querying “last instance”. I The system checks if the last instance has been analyzed ("364"). If so, the process ends, the instance segmentation neural network has been adapted or modified, and it is now ready for runtime inference mode.

[0076] If the last instance has not yet been analyzed, process 300 may include "obtain the first frame to train the next instance" 366, and the process loops back to operation 318 to set the iteration count back to 1 and begin customized neural network training for the next instance in the video sequence.

[0077] Algorithm 1, as a summary of the process of customizing or fine-tuning the settings mode, can be stated as follows: refer to Figure 5In subsequent runtime or inference modes, a technique can be used to combine instance segmentation masks into a single frame-based mask for use by other applications. An example of this technique is the fully connected Conditional Random Field (CRF) classifier (see Krähenbühl, P. et al., “Efficient inference in fully connected CRFs with gaussian edge potentials,” Advances in neural information processing systems, pp. 109-117 (2011)). While the segmentation network computes a binary mask for each instance, the CRF merges all binary masks into a single mask as the final object recognition output for the frame. The merged single frame-based mask will have, for example, labels 1 to L for all instances on the frame. This is shown in the mask merging flowchart 500, which shows the input frame or image 502 that is fed into the instance segmentation neural network 504. At the instance segmentation neural network 504, separate segmentation masks 506 and 508 are generated for each instance on the frame (here, a person and a bicycle), which are then merged into a single mask 510 via CFR.

[0078] Subsequently, applications can utilize this mask for a variety of reasons mentioned in this paper. Since video object segmentation is a fundamental computer vision task, it has important applications in fields such as video editing, smart homes, robotics, and autonomous driving. The proposed method can also be embedded in tablets, laptops, Chromebooks, smartphones, PCs, smart TVs, and more. Through this technology, these devices can provide many value-added services for segmentation-related applications.

[0079] refer to Figure 6-7 The results are video sequence 600 and example results 700 of object recognition performed using the custom patterns presented in this paper. Evaluation was performed using the DAVIS 2018 challenge dataset. It has 150 video sequences, with 90 for training, 30 for evaluation, and 30 for testing. There are 10,459 annotated frames and 376 objects. For the test dataset, only the first frame of each video is annotated. The resulting segmentation masks 702 to 705 in sequence 700 show good segmentation results based on each corresponding frame 602 to 605.

[0080] refer to Figures 8A-8DThe images present examples of applying the disclosed methods using small datasets and user-defined objects for scene understanding. For example, an elderly person might want a robot to label medicine bottles for daily medication dispensing because, for instance, the person might have difficulty handling the medication with their hands. Therefore, the robot must understand the labeling of each medication. Thus, the user first... (The sentence is incomplete and requires more context to translate accurately.) Figure 8A Object 802, in this case, is a medicine bottle. As discussed above, the first frame of the video sequence can be labeled manually, interactively, or automatically. Based on one-time annotations, the disclosed method or robot automatically generates training data in the form of a color image 804 showing object 806 and a corresponding candidate mask 808 showing object 810. The robot or method then automatically generates training data based on the unlabeled video and updates the instance segmentation neural network based on the generated data. During the testing phase, data for the same object is generated by the user's device, rather than, for example, during offline preprocessing, and the instance segmentation neural network is continuously adapted as data accumulates.

[0081] Test results from another scenario (not user-provided video) Figure 8C Images 812 to 822, taken from different perspectives of bottle 824, are shown in the results, which are displayed in a shadow mask. In the test results, images 826 and 830 (…) Figure 8D The results show that although only the label on the front of bottle 828 was captured, the robot could still identify and segment the back of the bottle 832, which is significantly different from the front. Furthermore, the method remained effective even when the scene changed.

[0082] Furthermore, it can be performed in response to instructions provided by one or more computer program products. Figure 4 and 5Any one or more operations. Such a program product may include a signal-bearing medium that provides instructions, which, when executed by, for example, a processor, can provide the functionality described herein. A computer program product may be provided in any form of one or more machine-readable media. Thus, for example, a processor including one or more processor cores may perform one or more operations of the example processes described herein in response to program code and / or instructions or instruction sets transmitted to the processor from one or more computer or machine-readable media. In general, a machine-readable medium may deliver software in the form of program code and / or instructions or instruction sets that can cause any device and / or system to perform as described herein. A machine or computer-readable medium may be a non-transitory article or medium (e.g., a non-transitory computer-readable medium) that can be used with any of the foregoing examples or other examples, except that it does not include transient signals themselves. It includes those elements other than signals themselves that can temporarily store data in a “transitory” manner, such as RAM, etc.

[0083] As used in any implementation described herein, the term "module" refers to any combination of software logic, firmware logic, and / or hardware logic configured to provide the functionality described herein. Software may be embodied as a software package, code, and / or instruction set or instructions, and "hardware" as used in any implementation described herein may, for example, individually or in any combination, include hardwired circuit systems, programmable circuit systems, state machine circuit systems, and / or fixed-function firmware storing instructions executed by the programmable circuit system. These modules may, collectively or individually, be embodied as circuit systems forming part of a larger system, such as integrated circuits (ICs), system-on-a-chip (SoCs), etc. For example, modules may be embodied in logic circuit systems for implementation via software, firmware, or hardware of the coding system discussed herein.

[0084] As used in any implementation described herein, the term "logic unit" refers to any combination of firmware logic and / or hardware logic configured to provide the functionality described herein. A logic unit may, collectively or individually, be embodied as part of a circuit system, such as an integrated circuit (IC), a system-on-a-chip (SoC), etc. For example, a logic unit may be embodied in a logic circuit system for the firmware or hardware implementation of the coding system discussed herein. Those skilled in the art will understand that operations performed by hardware and / or fixed-function firmware may optionally be implemented via software, which may be embodied as a software package, code, and / or instruction set or instructions, and will also understand that a logic unit may also utilize a portion of software to implement its functionality.

[0085] As used in any implementation described herein, the term "component" can refer to a module or logical unit, as those terms described above. Therefore, the term "component" can refer to any combination of software logic, firmware logic, and / or hardware logic configured to provide the functionality described herein. For example, those skilled in the art will understand that operations performed by hardware and / or firmware can optionally be implemented via a software module, which can be embodied as a software package, code, and / or instruction set, and will also understand that logical units can also utilize a portion of software to implement their functionality.

[0086] refer to Figure 9 An example image processing system 900 is arranged according to at least some implementations of this disclosure. In various implementations, the example image processing system 900 may have an imaging device 902 to form or receive captured image data. This can be implemented in various ways. Thus, in one form, the image processing system 900 may be one or more digital cameras or other image capturing devices, and in this case, the imaging device 902 may be camera hardware and camera sensor software, modules, or components. In other examples, the image processing system 900 may have an imaging device 902 that includes or may be one or more cameras, and a logic module 904 may be remotely communicating with or otherwise communicatively coupled to the imaging device 902 for further processing of the image data.

[0087] Therefore, the image processing system 900 can be a single camera or located on a multi-camera device, either of which can be a smartphone, tablet, laptop, or other mobile device, and they include computer vision cameras and sensors on robots, VR, AR, or MR headsets, etc. Additionally, the system 900 can be a device with one or more cameras, wherein processing occurs at one of the cameras or at a separate processing location in communication with the cameras, whether on or off the device and regardless of whether the processing is performed on a mobile device.

[0088] In any of these cases, the technology may include a camera, such as a digital camera system, a dedicated camera device, or an imaging phone or tablet, or other video camera, or some combination thereof. Thus, in one form, imaging device 902 may include camera hardware and optics, including one or more sensors and autofocus, zoom, aperture, ND filter, auto exposure, flash, and actuator control. These controls may be part of a sensor module or component for operating the sensor, which may be used to generate an image for the viewfinder and capture still images or videos. Imaging device 902 may also have lenses, an image sensor with an RGB Bayer color filter, an analog amplifier, an A / D converter, other components for converting incident light into a digital signal, and / or combinations thereof. The digital signal may also be referred to herein as raw image data.

[0089] Attached to or replacing the use of red-green-blue (RGB) depth cameras and / or microphone arrays to locate who is speaking, other forms include imaging devices of camera sensor types (e.g., webcams or webcam sensors or other complementary metal-oxide-semiconductor (CMOS) image sensors). Camera sensors can also support other types of electronic shutters, such as global shutters attached to or replacing rolling shutters, and many other shutter types. In other examples, RGB depth cameras and / or microphone arrays can be used as alternatives to camera sensors. In these examples, in addition to the camera sensor, the same sensor or a separate sensor and a light projector, such as an IR projector, can be provided to provide a separate depth image that can be used for triangulation with the camera image. Additionally, the imaging device can have any other known techniques for providing depth maps using multiple cameras or imaging devices or a single imaging device.

[0090] In the example shown and in the relevant sections of this document, logic module 904 may include a raw image processing unit 906 and a preprocessing unit 908, wherein the raw image processing unit 906 performs preprocessing such as de-mosaicing on the image data, and the preprocessing unit 908 performs further preprocessing tasks as described above to perform sufficient segmentation, which may or may not include generating a depth map or depth image to form a three-dimensional space, wherein pixels or points have three-dimensional (x, y, z) coordinates on the resulting depth map or depth image representing the three-dimensional space (or a 2D image or set of images of the same scene).

[0091] The logic module may also include an object recognition unit 909 for performing many of the operations described herein. Thus, for example, the segmentation unit 909 may have a custom mode unit 910 and a runtime mode unit 911. The custom mode unit 910 may have an initial segmentation unit 912, a data generation unit 913, and an instance segmentation unit 916. Similarly, as described above, the data generation unit 913 may have a general knowledge neural network unit 914 and a quality evaluation unit 915, as described above. The instance segmentation unit 916 has an instance segmentation neural network unit 917, a local adaptation unit 918, and a global adaptation unit 919, also as described above. The training database 925 may be stored in a memory storage device 924.

[0092] The image processing system 900 may also have one or more processors 920, which may include a dedicated image signal processor (ISP) 922, such as an Intel Atom, other GPUs, and / or dedicated purpose hardware running, for example, neural networks. The image processing system 900 may also have one or more memory storage devices 924, one or more displays 928 for providing images 930, a codec 932, and an antenna 926. In one example implementation, the image processing system 900 may have a display 928, at least one processor 920 communicatively coupled to the display, and at least one memory 924 communicatively coupled to the processor. The codec 932 may be an encoder, a decoder, or both. As an encoder 932, and utilizing the antenna 934, an encoder may be provided to compress image data for transmission to other devices that can display or store the image. It should be understood that, as a decoder, the codec may receive and decode image data for processing by the system 900 to receive an image for object recognition, as a supplement to or alternative to the image initially captured by the device 900. Additionally, the processed image 930 can be displayed on the display 928 or stored in the memory 924. As shown, any of these components may be able to communicate with each other and / or with portions of the logic module 904 and / or the imaging device 902. Therefore, the processor 920 can be communicatively coupled to both the imaging device 902 and the logic module 904 for operating those components. Through a method, although as... Figure 9 The image processing system 900 shown may include a specific set of blocks or actions associated with a particular component or module, but these blocks or actions may be associated with components or modules that are different from the specific components or modules shown herein.

[0093] refer to Figure 10The example system 1000 described herein operates one or more aspects of the image processing system described herein. As will be understood from the nature of the system components described below, these components may be associated with or used to operate one or more parts of the image processing system 1000 described above, thereby enabling the operation of the methods described herein. In various implementations, system 1000 may be a media system, but system 1000 is not limited to this context. For example, system 1000 may be incorporated into a digital still camera, a digital video camera, a mobile device with camera or video capabilities, such as an imaging phone, a webcam, a personal computer (PC), a laptop computer, an ultra-laptop computer, a tablet computer with multiple cameras, a touchpad, a portable computer, a handheld computer, a PDA, a personal digital assistant (PDA), a cellular phone, a combined cellular phone / PDA, a television, a smart device (e.g., a smartphone, a smart tablet, or a smart TV), a mobile internet device (MID), a messaging device, a data communication device, etc.

[0094] In various implementations, system 1000 includes a platform 1002 coupled to display 1020. Platform 1002 can receive content from content devices or other similar content sources, such as one or more content service devices 1030 or one or more content delivery devices 1040. A navigation controller 1050, including one or more navigation features, can be used to interact with, for example, platform 1002 and / or display 1020. Each of these components is described in more detail below.

[0095] In various implementations, platform 1002 may include any combination of chipset 1005, processor 1010, memory 1012, storage device 1014, graphics subsystem 1015, application 1016, and / or radio device 1018. Chipset 1005 can provide communication between processor 1010, memory 1012, storage device 1014, graphics subsystem 1015, application 1016, and / or radio device 1018. For example, chipset 1005 may include a storage adapter (not shown) capable of providing communication with storage device 1014.

[0096] The processor 1010 can be implemented as a Complex Instruction Set Computer (CISC) or Reduced Instruction Set Computer (RISC) processor; an x86 instruction set compatible processor; a multi-core or any other microprocessor or central processing unit (CPU). In various implementations, the processor 1010 can be one or more dual-core processors, one or more dual-core mobile processors, etc.

[0097] The memory 1012 can be implemented as a volatile memory device, such as, but not limited to, random access memory (RAM), dynamic random access memory (DRAM), or static RAM (SRAM).

[0098] Storage device 1014 can be implemented as a non-volatile storage device, such as, but not limited to, disk drives, optical disc drives, tape drives, internal storage devices, attached storage devices, flash memory, battery-backed SDRAM (synchronous DRAM), and / or network-accessible storage devices. In various implementations, storage device 1014 may include techniques that enhance storage performance protection for valuable digital media, for example, when multiple hard drives are included.

[0099] The graphics subsystem 1015 can perform processing on images such as still images or videos for display. For example, the graphics subsystem 1015 can be a graphics processing unit (GPU) or a visual processing unit (VPU). The graphics subsystem 1015 and the display 1020 can be communicatively coupled using analog or digital interfaces. For example, this interface can be any of a high-definition multimedia interface, display port, wireless HDMI, and / or wireless HD-compatible technologies. The graphics subsystem 1015 can be integrated into the processor 1010 or the chipset 1005. In some implementations, the graphics subsystem 1015 can be a standalone card communicatively coupled to the chipset 1005.

[0100] The graphics and / or video processing techniques described herein can be implemented in various hardware architectures. For example, graphics and / or video functions can be integrated within a chipset. Alternatively, discrete graphics and / or video processors can be used. As yet another implementation, graphics and / or video functions can be provided by a general-purpose processor, including a multi-core processor. In other implementations, these functions can be implemented in consumer electronic devices.

[0101] Radio device 1018 may include one or more radio devices capable of transmitting and receiving signals using a variety of suitable wireless communication technologies. Such technologies may involve communication across one or more wireless networks. Example wireless networks include (but are not limited to) wireless local area networks (WLANs), wireless personal area networks (WPANs), wireless metropolitan area networks (WMANs), cellular networks, and satellite networks. When communicating across such networks, radio device 1018 may operate according to one or more applicable standards of any version.

[0102] In various implementations, display 1020 may include any type of television monitor or display. Display 1020 may include, for example, a computer screen, a touchscreen display, a video monitor, a television-like device, and / or a television. Display 1020 may be digital and / or analog. In various implementations, display 1020 may be a holographic display. Furthermore, display 1020 may be a transparent surface capable of receiving visual projections. Such projections can convey various forms of information, images, and / or objects. For example, such projections may be visual overlays for mobile augmented reality (MAR) applications. Under the control of one or more software applications 1016, platform 1002 may display user interface 1022 on display 1020.

[0103] In various implementations, one or more content service devices 1030 may be hosted by any national, international, and / or independent service, and thus accessible, for example, via the Internet from platform 1002. One or more content service devices 1030 may be coupled to platform 1002 and / or display 1020. Platform 1002 and / or one or more content service devices 1030 may be coupled to network 1060 to transmit (e.g., send and / or receive) media information to and from network 1060. One or more content delivery devices 1040 may also be coupled to platform 1002 and / or display 1020.

[0104] In various implementations, one or more content service devices 1030 may include cable TV boxes, personal computers, networks, telephones, internet-enabled devices or appliances capable of transmitting digital information and / or content, and any other similar devices capable of transmitting content unidirectionally or bidirectionally between the content provider and platform 1002 and / or display 1020 via network 1060 or directly. It is understood that content may be transmitted unidirectionally and / or bidirectionally to and from any component in the content provider and system 1000 via network 1060. Examples of content may include any media information, such as video, music, medical, and gaming information.

[0105] One or more content service devices 1030 may receive content such as cable television programs, including media information, digital information, and / or other content. Examples of content providers may include any cable or satellite television or radio or internet content provider. The examples provided are not intended to limit the implementations according to this disclosure in any way.

[0106] In various implementations, platform 1002 may receive control signals from navigation controller 1050, which has one or more navigation features. For example, the navigation features of controller 1050 may be used to interact with user interface 1022. In some implementations, navigation controller 1050 may be a pointing device, which can be a computer hardware component (specifically, a human interface device) that allows users to input spatial (e.g., continuous and multidimensional) data into a computer. Many systems, such as graphical user interfaces (GUIs) and televisions and monitors, allow users to control a computer or television and provide data to the computer or television using physical gestures.

[0107] Movement of navigation features of controller 1050 can be replicated on the display (e.g., display 1020) by movement of pointers, cursors, focus rings, or other visual indicators displayed on the display. For example, under the control of software application 1016, navigation features located on navigation controller 1050 can be mapped to virtual navigation features displayed on user interface 1022. In some implementations, controller 1050 may not be a separate component but may be integrated into platform 1002 and / or display 1020. However, this disclosure is not limited to the elements or context shown or described herein.

[0108] In various implementations, the driver (not shown) may include technology that allows a user to immediately turn platform 1002 (such as a television) on and off, for example, after initial startup, by touching a button (when enabled). Program logic may allow platform 1002 to stream content to a media adapter or other content service device(s) 1030 or content delivery device(s) 1040, even when the platform is “off”. Furthermore, for example, chipset 1005 may include hardware and / or software support for 8.1 surround sound audio and / or high-definition (7.1) surround sound audio. The driver may include a graphics driver for an integrated graphics platform. In some implementations, the graphics driver may include a Peripheral Component Interconnect (PCI) Fast Graphics Card.

[0109] In various implementations, any one or more components shown in system 1000 can be integrated. For example, platform 1002 and one or more content service devices 1030 can be integrated, or platform 1002 and one or more content delivery devices 1040 can be integrated, or platform 1002, one or more content service devices 1030, and one or more content delivery devices 1040 can be integrated. In various implementations, platform 1002 and display 1020 can be integrated units. For example, display 1020 and one or more content service devices 1030 can be integrated, or display 1020 and one or more content delivery devices 1040 can be integrated. These examples are not intended to limit this disclosure.

[0110] In various implementations, System 1000 can be implemented as a wireless system, a wired system, or a combination of both. When implemented as a wireless system, System 1000 may include components and interfaces suitable for communication via a wireless shared medium, such as one or more antennas 1003, transmitters, receivers, transceivers, amplifiers, filters, control logic, etc. Examples of wireless shared media may include portions of the wireless spectrum, such as the RF spectrum. When implemented as a wired system, System 1000 may include components and interfaces suitable for communication via a wired communication medium, such as input / output (I / O) adapters, physical connectors for connecting I / O adapters to the corresponding wired communication medium, network interface cards (NICs), disk controllers, video controllers, audio controllers, etc. Examples of wired communication media may include wires, cables, metal leads, printed circuit boards (PCBs), backplanes, switching structures, semiconductor materials, twisted pairs, coaxial cables, optical fibers, etc.

[0111] Platform 1002 can establish one or more logical or physical channels to transmit information. The information may include media information and control information. Media information can refer to any data representing content for a user. Examples of content include data from a voice conversation, video conferencing, streaming video, email (“email”) messages, text messages, social media formats, voicemail messages, alphanumeric symbols, graphics, images, video, text, etc. Data from a voice conversation can be, for example, voice information, silence periods, background noise, comfort noise, tone, etc. Control information can refer to any data representing commands, instructions, or control words for an automated system. For example, control information can be used to route media information through the system or to instruct nodes to process media information in a predetermined manner. However, these implementations are not limited to... Figure 10 In the elements or context shown or described.

[0112] refer to Figure 11The small shape factor device 1100 is one example in which different physical styles or shape factors can be embodied in systems 900 or 1000. Through this method, device 1100 can be implemented as a mobile computing device with wireless capabilities. For example, a mobile computing device can refer to any device having a processing system and a mobile power source or power supply such as one or more batteries.

[0113] As described above, examples of mobile computing devices may include digital still cameras, digital video cameras, mobile devices with camera or video capabilities, such as imaging phones, webcams, personal computers (PCs), laptops, ultra-laptops, tablets, touchpads, portable computers, handheld computers, PDAs, personal digital assistants (PDAs), cellular phones, combined cellular phones / PDAs, televisions, smart devices (e.g., smartphones, smart tablets, or smart TVs), mobile internet devices (MIDs), messaging devices, data communication devices, and the like.

[0114] Examples of mobile computing devices may also include computers arranged for human wear, such as wrist computers, finger computers, ring computers, glasses computers, clip-on computers, armband computers, shoe computers, clothing computers, and other wearable computers. For example, in various embodiments, a mobile computing device may be implemented as a smartphone capable of performing computer applications as well as voice and / or data communications. Although some embodiments are described by way of example using a mobile computing device implemented as a smartphone, it is understood that other embodiments may also be implemented using other wireless mobile computing devices. These implementations are not limited to this context.

[0115] like Figure 11 As shown, device 1100 may include a housing having a front 1101 and a rear 1102. Device 1100 includes a display 1104, an input / output (I / O) device 1106, and an integrated antenna 1108. Device 1100 may also include navigation features 1112. I / O device 1106 may include any suitable I / O device for inputting information into the mobile computing device. Examples of I / O device 1106 may include an alphanumeric keypad, numeric keypad, touchpad, input keys, buttons, switches, microphone, speaker, voice recognition device, and software, etc. Information may also be input into device 1100 via microphone 1114, or may be digitized by a voice recognition device. As shown, device 1100 may include a camera 1105 (e.g., including at least one lens, aperture, and imaging sensor) and an illuminator 1110 (e.g., those described herein), which are integrated into the rear 1102 of device 1100 (or elsewhere). These implementations are not limited to this context.

[0116] The various forms of devices and processes described herein can be implemented using hardware elements, software elements, or a combination of both. Examples of hardware elements may include processors, microprocessors, circuits, circuit elements (e.g., transistors, resistors, capacitors, inductors, etc.), integrated circuits, application-specific integrated circuits (ASICs), programmable logic devices (PLDs), digital signal processors (DSPs), field-programmable gate arrays (FPGAs), logic gates, registers, semiconductor devices, chips, microchips, chipsets, etc. Examples of software may include software components, programs, applications, computer programs, application programs, system programs, machine programs, operating system software, middleware, firmware, software modules, routines, subroutines, functions, methods, procedures, software interfaces, application programming interfaces (APIs), instruction sets, computational code, computer code, code segments, computer code segments, words, values, symbols, or any combination thereof. Determining whether an embodiment is implemented using hardware elements and / or software elements can vary based on any number of factors, such as desired computational speed, power levels, thermal tolerance, processing cycle budget, input data rates, output data rates, memory resources, data bus speeds, and other design or performance constraints.

[0117] One or more aspects of at least one embodiment can be implemented by representative instructions stored on a machine-readable medium, the representative instructions representing various logics within a processor that, when read by a machine, cause the machine to manufacture the logic to perform the techniques described herein. Such a representation (referred to as an "IP core") can be stored on a tangible machine-readable medium and provided to various customers or manufacturing facilities for loading into a manufacturing machine that actually manufactures the logic or processor.

[0118] While certain features presented herein have been described with reference to various implementations, this specification is not intended to be construed as limiting. Therefore, various modifications and other implementations of the implementations described herein that are readily apparent to those skilled in the art to the purposes of this disclosure are considered to be within the spirit and scope of this disclosure.

[0119] The following example is relevant to other implementations.

[0120] According to one example implementation, a computer-implemented object recognition method for image processing includes: acquiring image data of frames from a captured video sequence, including the content of at least one object to be recognized and added to a neural network training database of objects, so as to customize the neural network training database by including at least one desired object of a specific user; inputting the frames into a general knowledge neural network associated with a general database to output a plurality of candidate masks associated with instances of objects; customizing the training database by adding an additional mask, the additional mask being one of the plurality of candidate masks selected; inputting the frames into an instance segmentation neural network to output a segmentation mask; modifying the instance segmentation neural network to form a modified instance segmentation neural network, wherein the modification is at least partially based on at least one additional mask in the training database; and performing object recognition during inference runtime by using the modified instance segmentation neural network.

[0121] In another implementation, the method may further include: determining a quality score for each candidate mask by comparing each candidate mask with a segmentation mask output from the instance segmentation neural network; selecting a candidate mask with the highest quality score and satisfying at least one threshold criterion as an addend mask; calculating the quality score based at least on a color histogram comparison; calculating the quality score based at least on a value associated with Jaccard; and locally adapting the instance segmentation neural network to the addend mask from the training database based on how close the frame of the addend mask is to the current target frame being analyzed along the video sequence, wherein the closer the frame of the addend mask is to the current target frame, the more likely the instance segmentation neural network is to be adapted to the addend mask, wherein the addend mask at the five frames closest to the current target frame has approximately a 70% chance of being used to adapt the instance segmentation neural network, while frames farther than the five closest frames have a smaller chance of being used to adapt the instance segmentation neural network, wherein frames farther than the five closest frames have approximately a 30% chance of being used to adapt the instance segmentation neural network. The method includes, in addition to local adaptation, globally adapting the instance segmentation neural network to an additive mask for substantially every frame of the video sequence, such that the additive mask for local adaptation is used more than once. The method further includes adapting the instance segmentation neural network using supervised training by using additive masks from the training database as the output of the instance segmentation neural network and using frames of image data with corresponding individual additive masks as the input of the instance segmentation neural network; wherein, when input to the general knowledge neural network, only one frame in the video sequence has annotations for one or more objects on that frame; and wherein, when input to the instance segmentation neural network, only one frame in the video sequence has annotations for one or more objects on that frame.

[0122] In other implementations, a computer-implemented system comprising: at least one display; at least one memory; and at least one processor communicatively coupled to the display and the memory, and operating by: acquiring image data of frames of a captured video sequence, including the content of at least one object to be identified and added to a neural network training database of objects, so as to customize the neural network training database by including at least one desired object of a specific user; inputting these frames into a general knowledge neural network associated with a general database to output a plurality of candidate masks associated with instances of objects; customizing the training database by adding an additional mask to the training database, the additional mask being one of the plurality of candidate masks selected; inputting the frames into an instance segmentation neural network to output a segmentation mask; modifying the instance segmentation neural network to form a modified instance segmentation neural network, wherein the modification is at least partially based on at least one additional mask in the training database; and performing object recognition during inference runtime by using the modified instance segmentation neural network.

[0123] The system may further include a processor that operates by: receiving activation from a user for initiating a customized settings mode, the customized settings mode including the acquisition of the image data, inputting the frames into the general knowledge neural network and the instance segmentation neural network, and operating the mode to perform the modification of the instance segmentation neural network; wherein the general knowledge neural network and the instance segmentation neural network operate collaboratively to analyze the same plurality of frames of the video sequence, one instance at a time, such that the video sequence is repeatedly analyzed and analyzed at least once for each instance; wherein the video sequence is repeatedly analyzed for each instance to form a target The processor adapts the instance segmentation neural network for each instance through multiple iterations; wherein the processor operates by: locally adapting the instance segmentation neural network while a single frame is being analyzed, the local adaptation being repeated for multiple frames during a single iteration, and globally adapting the instance segmentation neural network once at the end of each iteration, wherein the local adaptation includes adapting the instance segmentation neural network using an added mask from the training database based on the position of the masked frame relative to the current target frame being analyzed along the video sequence, and wherein the global adaptation includes adapting the instance segmentation neural network to the added mask for substantially every frame of the video sequence.

[0124] As another implementation, at least one non-transitory computer-readable medium stores instructions thereon that, when executed, cause a computing device to operate by: acquiring image data of frames from a captured video sequence, including the content of at least one object to be identified and added to a neural network training database of objects, so as to customize the neural network training database by including at least one desired object of a specific user; inputting these frames into a general knowledge neural network associated with a general database to output a plurality of candidate masks associated with instances of objects; customizing the training database by adding an added mask, which is one of the plurality of candidate masks selected; inputting the frames into an instance segmentation neural network to output a segmentation mask; modifying the instance segmentation neural network to form a modified instance segmentation neural network, wherein the modification is at least partially based on at least one added mask in the training database; and performing object recognition during inference runtime by using the modified instance segmentation neural network.

[0125] The instructions may also cause the computing device to include, wherein the instructions cause the computing device to operate by: adapting the instance segmentation neural network for a single instance in a multi-iterative manner, wherein the adaptation is performed for multiple frames of the video sequence at each iteration, such that the instance is adapted to each instance being analyzed; wherein the instructions cause the computing device to operate by: determining a quality score for each candidate mask by comparing each candidate mask with a segmentation mask output by the instance segmentation neural network; and locally and globally adapting the instance segmentation neural network to perform the instance segmentation neural network... The network is modified in the following ways: locally, the local adaptation includes adapting the instance segmentation neural network using an added mask from the training database based on the position of the masked frame relative to the current target frame being analyzed along the video sequence; globally, the local adaptation includes adapting the instance segmentation neural network to the added mask for substantially every frame of the video sequence; and during runtime after customizing the instance segmentation neural network to a custom mode, running the instance segmentation neural network to output an instance segmentation mask for one instance of a frame, the output being repeated for each instance on the frame; and combining the output instance segmentation masks into a single frame-based mask.

[0126] In another example, at least one machine-readable medium may include a plurality of instructions that, in response to being executed on a computing device, cause the computing device to perform a method according to any of the above examples.

[0127] In yet another example, an apparatus may include a component for performing a method according to any of the above examples.

[0128] The examples above may include specific combinations of features. However, the examples above are not limited to this, and in various implementations, the examples above may include: performing only a subset of these features, performing different orders of these features, performing different combinations of these features, and / or performing additional features beyond those explicitly listed. For example, all the features described with respect to any example method may be implemented for any example device, example system, and / or example article, and vice versa.

Claims

1. A non-transitory computer-readable medium storing instructions that, in response to being executed on a computing device, cause the computing device to perform an operation comprising the following steps: Acquire image data of frames from a video sequence for one or more target objects to be identified, and annotations for instances of the objects in the first frame of the video sequence; The frame is input into a first neural network associated with a database to output one or more candidate masks associated with the instance to the database; The boundaries of the instance are determined based on the annotations for that instance; Calculate a similarity score for each candidate mask based on the boundary of the instance and the one or more candidate masks; From the one or more candidate masks, select one candidate mask whose similarity score meets at least one criterion as the selected mask, and update the database by adding the selected mask to the database; The frame is input into a second neural network to output a segmentation mask for the instance; as well as Based on the segmentation mask for the instance, output object recognition results for the one or more target objects to be identified.

2. The non-transitory computer-readable medium of claim 1, further comprising instructions that, in response to being executed on the computing device, cause the computing device to: The image data of the frames in the video sequence are preprocessed, wherein, The preprocessing of the image data includes identifying the location of the instance in the first frame.

3. The non-transitory computer-readable medium of claim 1, further comprising instructions that, in response to being executed on the computing device, cause the computing device to: The database is initialized by adding the annotation to the database.

4. The non-transitory computer-readable medium of claim 1, further comprising instructions that, in response to being executed on the computing device, cause the computing device to: The database is updated frame by frame.

5. The non-transitory computer-readable medium according to claim 1, wherein, The at least one criterion includes the similarity score being higher than a predetermined threshold.

6. The non-transitory computer-readable medium according to any one of claims 1 to 5, wherein, The first neural network has an architecture in the form of a Masked Region Convolutional Neural Network (Mask R-CNN).

7. A calculation method, comprising: Acquire image data of frames from a video sequence for one or more target objects to be identified, and annotations for instances of the objects in the first frame of the video sequence; The frame is input into a first neural network associated with a database to output one or more candidate masks associated with the instance to the database; The boundaries of the instance are determined based on the annotations for that instance; Calculate a similarity score for each candidate mask based on the boundary of the instance and the one or more candidate masks; From the one or more candidate masks, select one candidate mask whose similarity score meets at least one criterion as the selected mask, and update the database by adding the selected mask to the database; The frame is input into a second neural network to output a segmentation mask for the instance; as well as Based on the segmentation mask for the instance, output object recognition results for the one or more target objects to be identified.

8. The calculation method according to claim 7 further includes: The image data of the frames of the video sequence is preprocessed, wherein the preprocessing of the image data includes identifying the location of the instance in the first frame.

9. The calculation method according to claim 7, further comprising: The database is initialized by adding the annotation to the database.

10. The calculation method according to claim 7, further comprising: The database is updated frame by frame.

11. The calculation method according to claim 7, wherein, The at least one criterion includes the similarity score being higher than a predetermined threshold.

12. The calculation method according to any one of claims 7 to 11, wherein, The first neural network has an architecture in the form of a Masked Region Convolutional Neural Network (Mask R-CNN).

13. A computing system, comprising: A memory that stores instructions; and One or more processors, In response to being executed by the one or more processors, the instructions cause the one or more processors to perform the computation method of any one of claims 7 to 12.

14. A computing device comprising means for performing the computing method according to any one of claims 7 to 12.

15. A computer program product comprising instructions that, when executed by one or more processors, cause the one or more processors to perform the computation method of any one of claims 7 to 12.

Images