Machine-learning-assisted self-improving object-identification system and method

Abstract

To provide a system that precisely identifies an object arranged in a specified area.SOLUTION: An object-identification system 100 captures an image in an area by an image sensor 106 of a sensor module 102, and detects and recognizes an object using a deep-layer neural network (DNN) 112. A controller 104 is configured to: identify and register an area visitor; submit the image to the neural network; perform association with the detected object and retraining in a case of failure in recognition; further trace the position of the object and acquire label information in the case of failure in recognition; and perform association with an image version and recording to an image database 122.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] (CROSS REFERENCE TO RELATED APPLICATIONS) This application claims priority to and the benefit of co-pending U.S. Provisional Patent Application No. 62 / 734,491, entitled "Machine-Learning-Assisted Self-Improving Object-identification System and Method," filed September 21, 2018, the entire contents of which are incorporated herein by reference for all purposes.

[0002] FIELD OF THEINVENTION The present invention relates generally to machine learning systems and methods for identifying objects. [Background technology]

[0003] Various consumer goods and retail businesses are attempting to improve the shopping experience for customers by automating the purchasing and checkout process. Such automation involves deploying systems that can identify what items a customer has taken off the shelf. Some systems use video surveillance and image processing technology to identify these items. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] US Patent No. 09911290 [Patent Document 2] Japanese Patent Application Publication No. 08-096138 [Patent Document 3] JP 2016-139176 A [Patent Document 4] JP 2016-201105 A [Non-patent literature]

[0005] [Non-Patent Document 1] Hiromasa Yoshimoto and Tsutomu Tamukai, Construction of a general object recognition system for home service robots using deep learning with depth images, Technical Report of the Institute of Electronics, Information and Communication Engineers, 2017, SIS2017-55 Summary of the Invention [Problem to be solved by the invention]

[0006] However, proper detection and identification of items within a captured image can be affected by a variety of factors, such as lighting conditions, shadows, obstructed visibility, and the location and position of the item on the shelf, and inconsistent results render such a system ineffective. [Means for solving the problem]

[0007] All embodiments and features described below can be combined in any technically possible manner.

[0008] In one aspect, the present invention relates to an object identification system comprising an image sensor configured to capture images of objects disposed within an area designated to hold objects, a deep neural network trained to detect and recognize objects in the images that resemble the objects held within the designated area, and a controller in communication with the image sensor and the deep neural network that receives the images captured by the image sensor. The controller includes one or more processors configured to register identities of persons who visit the area designated to hold objects, submit images to the deep neural network, associate the registered identities of persons with objects detected in the images submitted to the deep neural network, retrain the deep neural network using the submitted images if the deep neural network is unable to recognize the objects detected in the submitted images, and track the location of the detected objects while they are within the area designated to hold objects.

[0009] The controller may be further configured to, in response to the deep neural network being unable to recognize the detected object in the submitted image, obtain labeling information for the detected object, associate the labeling information with a version of the image submitted to the deep neural network, and store the version of the image and the associated labeling information in an image database used to retrain the deep neural network. The human input acquisition module may be configured to, in response to a request from the controller, obtain labeling information from a user when the deep neural network is unable to recognize the detected object in the submitted image.

[0010] The controller may be further configured to find areas in the image where changes occur when the deep neural network is unable to recognize the detected object, generate a version of the image focused on the changes in the area, and submit the version of the image to the deep neural network to determine whether the deep neural network is able to recognize the detected object in a second version of the image. The controller may be further configured to obtain labeling information about the detected object regardless of whether the deep neural network recognizes the detected object in the submitted version of the image, associate the obtained labeling information with the version of the image submitted to the deep neural network, and store the version of the image and the associated labeling information in an image database used to retrain the deep neural network. Additionally, the controller may be further configured to obtain the labeling information from the deep neural network if the deep neural network recognizes the detected object in the submitted version of the image.

[0011] The deep neural network may be a first deep neural network, and the system may further include a second deep neural network configured to operate in parallel with the first deep neural network, wherein each of the first and second deep neural networks generates an output based on image data obtained from the image, and the image data obtained by the first deep neural network differs from the image data obtained by the second deep neural network.

[0012] The object identification system may further comprise a depth sensor having a field of view substantially matching the field of view of the image sensor, the depth sensor acquiring depth pixel values ​​of an image within the field of view of the depth sensor, and during training or object recognition, when an image is submitted to the deep neural network, the depth pixel value and less than three pixel values ​​taken from the group consisting of R (red), G (green), and B (blue) are submitted to the deep neural network as image data.

[0013] The deep neural network may reside on a remote server system, and the controller may further comprise a network interface for communicating with the deep neural network on the server system.

[0014] In another aspect, the invention relates to a method for identifying and tracking objects, the method comprising the steps of registering the identity of a person who visits an area designated to hold objects, capturing an image of the area designated to hold objects, submitting a version of the image to a deep neural network trained to detect and recognize objects in the image that resemble those objects held in the designated area, detecting the objects in the version of the image, associating the registered identity of the person with the detected object, retraining the deep neural network using the version of the image if the deep neural network is unable to recognize the detected object, and tracking the location of the detected object while it is within the area designated to hold objects.

[0015] The method may further include, in response to the deep neural network being unable to recognize the detected object in the version of the image, obtaining labeling information for the object detected in the version of the image, associating the labeling information with the version of the image, and storing the captured version of the image and the associated labeling information in an image database used to retrain the deep neural network. The step of obtaining labeling information for the object detected in the version of the image, in response to the deep neural network being unable to recognize the detected object in the version of the image, may include obtaining the labeling information from user-supplied input.

[0016] The method may further include locating areas in the version of the image where changes appear when the deep neural network is unable to recognize the object detected in the first version of the image, generating a second version of the image focused on the areas where changes were found, and submitting the second version of the image to the deep neural network to determine whether the deep neural network is able to recognize the object detected in the second version of the image. The method may further include obtaining labeling information about the object detected in the first version of the image regardless of whether the deep neural network recognizes the object detected in the second version of the image, associating the labeling information with the first version of the image, and storing the captured first version of the image and the associated labeling information in an image database used to retrain the deep neural network. The step of obtaining labeling information about the object detected in the version of the image may include obtaining the labeling information from the deep neural network if the deep neural network recognizes the object detected in the version of the image.

[0017] The step of submitting a version of the image to the deep neural network may include submitting the depth pixel value and less than three pixel values ​​taken from the group consisting of R (red), G (green), and B (blue) to the deep neural network as image data.

[0018] The method may further include submitting image data obtained from a version of the image to a first deep neural network and a second deep neural network in parallel, wherein the image data submitted to the first deep neural network is different from the image data submitted to the second deep neural network.

[0019] In another aspect, the invention relates to a sensor module comprising an image sensor configured to capture images within a field of view of the image sensor and a depth sensor having a field of view substantially coinciding with the field of view of the image sensor. The depth sensor is configured to obtain an estimated depth value for the image captured by the depth sensor. The sensor module further comprises a controller in communication with the image sensor and the depth sensor to receive image data associated with the image captured by the image sensor and the estimated depth value associated with the image captured by the depth sensor. The controller includes one or more processors configured to register the identity of a person who visits an area designated to hold an object; submit image data associated with images captured by the image sensor and estimated depth values ​​associated with images captured by the depth sensor to a deep neural network trained to detect and recognize objects in images that resemble these objects held in the designated area; associate the registered identity of the person with the objects detected in the image data and estimated depth values ​​submitted to the deep neural network; and store versions of the images captured by the image sensor and depth sensor for use in subsequent retraining of the deep neural network if the deep neural network is unable to recognize the detected objects.

[0020] The controller may further comprise a cloud interface for communicating with the deep neural network over a network.

[0021] The sensor module may further comprise a human input acquisition module configured to acquire labeling information from a user in response to a request from the controller when the deep neural network is unable to recognize the detected object based on the submitted image data and the estimated depth value. [Brief description of the drawings]

[0022] The above and further advantages of the present invention can be better understood by referring to the following description in conjunction with the accompanying drawings, in which like numerals indicate like structural elements and features in the various views, and in which the drawings are not necessarily to scale, emphasis instead being placed upon illustrating the principles of the invention.

[0023] [Figure 1] 1 illustrates an embodiment of a machine learning assisted object identification system that includes a controller and an image sensor having a field of view of a support surface designated to hold an object. [Diagram 2] FIG. 2 is a block diagram of one embodiment of the controller of FIG. 1. [Diagram 3] FIG. 1 is a functional block diagram of an embodiment of a machine learning assisted object identification system that includes an artificial intelligence module in communication with a local machine learning module. [Figure 4] FIG. 1 is a functional block diagram of an embodiment of a machine learning assisted object identification system that includes an artificial intelligence module in communication with a remote machine learning module. [Diagram 5] FIG. 2 is a functional block diagram of another embodiment of a machine learning assisted object identification system including an artificial intelligence module in communication with a remote machine learning module. [Figure 6] FIG. 1 is a block diagram of one embodiment of an object tracking module including a deep neural network or DNN. [Figure 7] FIG. 2 is a block diagram of one embodiment of a computer vision module. [Figure 8] 1 is a flow chart of one embodiment of a process for machine learning assisted object identification. [Figure 9] 1 is a flow chart of one embodiment of a process for identifying an object in a captured image and an optional multi-pass authentication process. [Figure 10]1 is a flowchart of one embodiment of a process for retraining a DNN with images in which the DNN does not initially identify objects. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0024] The object identification systems and methods described herein combine computer vision, machine learning, and deep neural networks (DNNs) to enable accurate identification and tracking of people and objects. Initially, the DNN may be a blank slate and unable to identify objects without human assistance, or may be trained on a predefined set of images to give the DNN a baseline. To give the DNN a baseline object identification ability, a human needs to train the DNN on a predefined training set of images. After that initial training, the DNN's ability to identify objects is continually improved for subsequent trainings. These subsequent trainings are based on images in which the DNN was initially unable to identify the objects. The objects in these images become identifiable and thus valuable for retraining the DNN, either because of the human-supplied information that identifies the objects in the images, or because of a multi-verification process that focuses the DNN's detection activity on areas in the images where changes have been detected.

[0025] 1 illustrates an embodiment of a machine learning assisted object identification system 100 having at least one sensor module 102. Each sensor module 102 includes a controller 104 in communication with one or more color (e.g., RGB) image sensors 106, and optionally one or more depth sensors 108 and light sources 110, shown in phantom, and optionally a radio-frequency (RF) transceiver (not shown). The controller 104 implements a deep neural network (DNN) 112 for use in object recognition and a computer vision module 114 for detecting changes in images, which are described in more detail below.

[0026] In one embodiment, each sensor module 102 is a self-contained electronic unit capable of registering persons visiting an object holding area, capturing images, processing images, detecting objects, machine learning assisted self-improving object recognition, tracking objects, and providing light guidance if so configured. In other embodiments, one or more of these functions are performed remotely (i.e., not at the sensor module), e.g., the functions of object detection, machine learning assisted self-improving object recognition, and object tracking may be performed at a remote computing site in communication with the sensor module over a network.

[0027] The sensor modules 102 may be deployed in a fixed position near the support surface 116 in the object holding area, or may be mobile and embodied in a mobile device. As an example of a fixed deployment, the sensor modules 102 may be dropped from a top in a surveillance configuration so that every corner of the enterprise site is covered. These sensor modules are small and non-invasive, and can track the identity and path of individuals throughout an enterprise, for example, as described in U.S. Patent Application Publication No. 2018 / 0164112 (A1), published June 14, 2018, the entire application of which is incorporated herein by reference.

[0028] Mobile embodiments of the sensor module include, but are not limited to, a smartphone, a tablet computer, a wearable computing device, or any other portable computing device configured with one or more processors, an RGB camera, wireless communication capabilities, an optional depth sensor, an optional light source, and software for implementing image processing functions, object detection, tracking, and recognition functions, self-improving machine learning functions, and optional light guide functions described herein. The software may be embodied in a downloaded application (app) that may be stored on the mobile device. A mobile person or machine may in fact carry an object identification device that can recognize objects captured by the camera(s) of the mobile device. For example, a person with such a device may activate the software to approach a table (i.e., support surface) holding various objects, point the device (i.e., its camera(s)) at each object, capture an image of the object, and communicate the type (identity) of the object. To obtain the identity of the object, the mobile device may communicate with a remote server hosting a DNN, send the image to the remote server, and receive the identity of the object.

[0029] Each image sensor 106, which may also be referred to herein as an optical sensor, provides color information, and each depth sensor 108 provides an estimated depth for each pixel of the captured image. The image sensor 106 and depth sensor 108 may be embodied in a single camera, such as, for example, Microsoft's Kinect™, or in separate cameras. The image sensor and optional depth sensor are disposed facing a support surface 116. Examples of support surfaces include, but are not limited to, desktop tables, shelves, and floor spaces. Generally, the support is disposed in or at an object-holding area. The object-holding area may be, for example, a supermarket, warehouse, inventory, room, cupboard, hallway, cupboard, locker, each with or without fixed access. Examples of objects to be identified and tracked include, but are not limited to, packages, parcels, boxes, equipment, tools, food products, bottles, jars, and cans. (People can also be identified and tracked.) Each image sensor 106 has a field of view (FOV) that covers some or all of the area occupied by support surface 116, and the field of view of the optional depth sensor at least matches that of the image sensor. Each separate sensor has its own perspective of the area on support surface 116 and of objects disposed thereon.

[0030] The controller 104 may be configured to control the light sources 110 to provide light guides to objects located on the support surface 116 or in specific regions of the support surface depending on the object or area of ​​interest. Examples of light sources 110 include, but are not limited to, lasers, projectors, LEDs, light bulbs, flashlights, and lights. The light sources 110 may be disposed on the support surface 116 or may be disposed remotely from and directed towards the support surface 116.

[0031] For example, a display 118 may be included in the object identification system 100 to provide a visual layout of objects on the support surface, visual navigation to objects or areas on the surface, and a user interface for use by persons entering and exiting the object holding area. The display 118 may be located at a threshold of the holding area or within the holding area, as appropriate. The display 118 may be part of an electronic device (e.g., computer, smartphone, mobile device) configured with input / output devices, such as a physical or virtual keyboard, keypad, barcode scanner, microphone, camera, and may be used to register the identity of persons entering the object holding area and / or scan object labels.

[0032] The controller 104 may also communicate with one or more servers 120 (i.e., server systems) via a network connection. These server(s) 120 may implement third-party services such as "cloud services" or may be implemented locally or on-site at an enterprise. As used herein, "cloud" refers to software and services that run on a remote network, such as the Internet, instead of on a sensor module or a local computer. Clouds may be public, private, or a combination thereof. An example of a cloud service suitable for the principles described herein is the Azure™ cloud service offered by Microsoft® of Redmond, Washington. The server(s) 120 may run virtual machines that provide the cloud services requested by the sensor module 102.

[0033] During operation of the object identification system 100, a person arrives at the object holding area to perform any one or more of at least four object handling actions, including placing an object, removing an object, moving an object to another spot within the holding area, or alerting personnel to an object worthy of inspection. In general, the object identification system registers the identity of a person who arrives at the holding area (i.e., interacts with the object identification system) and associates each registered person with one or more objects that the person is handling. Using image processing techniques, the object identification system continuously monitors and acquires real-time image data of the holding area. From the real-time image data, the object identification system detects when each such object is placed on the support surface 116, moved to another area of ​​the support surface, or removed from the support surface. Techniques for detecting and tracking objects disposed on a support surface within a holding area can be found in U.S. Patent Application Publication No. 15 / 091,180, entitled "Package Tracking Systems and Methods," filed April 5, 2016, which is incorporated herein by reference in its entirety. Additionally, the object identification system may identify perishable items and send staff notices of their expiration dates. Or, the object identification system may recognize damaged goods on the shelves and notify staff accordingly. In response to the notice, staff can then inspect the items and remove them if their expiration dates have passed or ascertain the extent of the damaged packaging.

[0034] The object identification system further recognizes each object on the support surface or with the handling action. Object recognition helps to identify the type of object detected and tracked (e.g., a package from a particular carrier, a pickle jar, a microscope). Such object recognition may involve human interaction to initially identify a given object or to verify if the recognition of a given object is correct or to fine-tune the recognition. The object identification system uses machine learning techniques to improve its object recognition capabilities. Recognition of a given object may facilitate tracking of the object while it is within the holding area and help to confirm the presence or movement of the object.

[0035] Occasionally, the sensor module 102 captures an image when object recognition falls below a threshold, i.e., the object identification system is unable to recognize an object in an image. Despite being unable to recognize the object (at least initially), the object identification system can still track the object based on the visual characteristics of the object, i.e., its initial placement and any subsequent locations within the holding area. The unidentifiable images are retained for later retraining of the DNN 112, so that the DNN can recognize the previously unrecognizable object if it is present in a subsequent processed image. Human interaction with the object identification system via voice recognition, gesture recognition, or keyboard input can clearly identify the object in the unidentifiable image and give the image an appropriate label. An example of gesture recognition is a person holding up three fingers to identify the object as type number 3, where the object identification system has stored an association of the three finger gesture with a particular object (e.g., three fingers corresponds to a microscope). After previously unidentifiable objects in images become recognizable, with the help of human input, the images and associated appropriate labels are stored in an image database 122. The object identification system 100 uses these stored images and labels to retrain the deep neural network 112. By retraining the deep neural network with previously unidentifiable images, now identifiable with human-provided information, the neural network 112 becomes increasingly "smarter." Over time, the probability that the neural network will recognize the object in subsequently captured images approaches 100 percent.

[0036] The image database 122 may be maintained in local storage 124 accessed via a central computer 126 proximate the sensor modules 102. In this embodiment, the central computer 126 provides access to the image database 122 for all deployed sensor modules 102. In another embodiment, shown in dotted lines in Figure 1, the image database 122 is stored in remote storage 128, e.g., a "cloud" that communicates with each sensor module 102 via server(s) 120. In addition to these initially unidentifiable images, the image database 122 also maintains an initial training set of images.

[0037] 2 illustrates one embodiment of the controller 104 of FIG. 1. The controller 104 includes one or more processors 200, examples of which include, but are not limited to, image processors, central processing units, and graphics processing units, each of which may be standard or custom designed. The one or more processors 200 are in communication with a memory 202. In the case of multiple processors 200, such processors may be located at different sites (e.g., one processor may be located locally (i.e., at the sensor module) and another processor may be located remotely (e.g., in the "cloud"). Similarly, the memory 202 may be located locally, remotely, or a combination thereof.

[0038] The one or more processors 200 communicate with a video interface 204, an optional light source interface 206, an optional audio interface 208, a network interface 210, and an interface 212 to I / O components (e.g., a display 118). Through the video interface 204, the controller 104 communicates with each image sensor 106 and depth sensor 108 (if present) in the sensor module 102, through the light source interface 206, the controller 104 controls the activation of the light sources 110 and, depending on the type of light source, the direction in which the emitted light beams are directed, and through the audio interface 208, the controller 104 communicates with an audio device that captures or plays sound.

[0039] In addition to conventional software such as an operating system and input / output routines, the memory 202 stores program code for configuring the one or more processors 200 to implement the deep neural network (DNN) 112 and to perform personnel enrollment 214, object detection in images 216, object tracking in a holding area 218, object recognition in images 220, neural network training 222, image pre-processing 224, change tracking in images 226, and optionally light guiding 228. The one or more processors 200 and the memory 202 may be implemented together or individually on a single or multiple integrated circuit (IC) devices. Furthermore, the program code stored in the memory 202 may reside at different sites. For example, the program code for implementing the DNN 112 may reside at a remote location (e.g., on the cloud) and the program code for user recognition may reside and execute locally (i.e., on the sensor module).

[0040] In brief overview, program code for personnel registration 214 records the identity and actions of individuals using the object identification system 100 and associates such individuals with objects that they affect, program code for object detection 216 detects the presence of an object in an image using image processing techniques, program code for object tracking 218 tracks the location of detected objects within a holding area, program code for object recognition 220 recognizes (i.e., identifies or classifies) objects in an image using the DNN 112, program code for training neural network 222 trains the DNN 112 to be able to recognize particular types of objects, program code for image pre-processing 224 applies image editing techniques to captured images to improve object detection and recognition activities within such images, program code for change tracking 226 detects changes in the images and assists in labeling the images, and optionally program code for light guide 228 uses light source 110 to guide a person to an object and / or a location within the object holding area. As described in more detail below, various elements or functions of the controller 104 may reside remotely, that is, in some embodiments, some elements or functions of the controller 104 are not part of the sensor module 102 (FIG. 1) but reside remotely (e.g., in the “cloud”).

[0041] 3 illustrates a functional block diagram of an embodiment of the object identification system 100 including an artificial intelligence (AI) module 300 in communication with a machine learning module 302. The AI ​​module 300 includes an image acquisition module 304, an image pre-processing module 306, an object tracking module 308, a human input acquisition module 310, and a tracking quality measurement module (QMM) 312. The machine learning module 302 includes a local storage 124 (which maintains the image database 122 described in connection with FIG. 1 ) and a deep neural network (DNN) trainer 314.

[0042] The image acquisition module 304 of the AI ​​module 300 is configured to acquire images from the image sensor 106 and optional depth sensor 108. The captured images are passed to an image pre-processing module 306, which forwards the images to an object tracking module 308. The image pre-processing module 306 sends each image to the computer vision module 114 (line 316), which sends a copy of the image to the DNN 112 (line 318) (alternatively, the computer vision module 114 receives a copy of the image).

[0043] Generally, the object tracking module 308 is configured to detect objects in an image, track such objects, and perform object recognition using the DNN 112 of Figure 1. The results generated by the object tracking module 308 are passed (line 320) to the tracking QMM 312, which uses a threshold (or other criteria) to prove whether an object has been recognized in the image. The results include two types of information: 1) the area in the image where the DNN determines that an object (or an area for multiple objects, depending on the image) has been found, and 2) a list of probabilities of each type of object being considered to be in the area(s).

[0044] Based on the information received from the object tracking module 308, the QMM 312 determines whether the DNN 112 was successful in identifying the object(s) in the image. If successful, the QMM 312 sends a success signal (line 322). The controller 104 may receive this success signal and respond accordingly to the signal in response to an end user application that seeks to determine the identity of an object, such as a package tracking application.

[0045] If the object is not identifiable in the image, the QMM 312 notifies the computer vision module 114 (line 324). The computer vision module 114 optionally sends the image to the DNN 112 (line 326), which is derived from the original image and focuses on the area in the original image where the change was detected. The DNN 112 may attempt to identify the object in this focused image (line 326). That is, the DNN 112 performs a second pass. If the DNN is not successful during the second pass, the QMM 312 sends a request to the human input acquisition module 310 (line 327) for labeling information about the unidentifiable object in the original image. Regardless of whether the DNN 112 succeeds or fails to recognize the object in the focused image, the computer vision module 114 sends the original image in which the object was not initially recognized to the local storage 124 (line 328). The stored images are linked / associated (box 330) with human-provided labels (line 332) from the human input acquisition module 310 or with labels (line 334) generated by the DNN 112 (line 320) and sent to the QMM 312 and then forwarded by the QMM 312. The DNN trainer 314 uses these images and their associated ID information (i.e., labels) in the local storage 124 to retrain the DNN 112 (line 336).

[0046] Each sensor module 102 (FIG. 1) can be configured to locally provide the functionality of the AI ​​module 300, and the central computer 126 (FIG. 1) is configured to provide the functionality of the machine learning module 302. Each sensor module 102 (FIG. 1) communicates with the central computer 126, which provides training of the DNN based on images in the image database 122. Thus, different sensor modules develop comparable image recognition capabilities.

[0047] 4 illustrates a functional block diagram of another embodiment of the object identification system 100 having an artificial intelligence (AI) module 400 in communication with a machine learning module 402. In general, the object identification system of FIG. 4 differs from that of FIG. 3 in that the machine learning module 402 resides remotely, i.e., on a network in the cloud or on the Internet. Furthermore, in this embodiment, the remote machine learning module 402 includes the DNN 112. Thus, the computation of the DNN occurs on a remote server (e.g., server 120 of FIG. 1), for example, implemented by a third-party cloud service.

[0048] Specifically, the AI ​​module 400 includes an image acquisition module 304 (FIG. 3), an image pre-processing module 306 (FIG. 3), an object tracking module 404, a human input acquisition module 310 (FIG. 3), and a tracking QMM 312. The object tracking module 404 includes a computer vision module 114 and a cloud interface module 406. The remote machine learning module 402 maintains an image database 122 (FIG. 1) in a cloud (i.e., remote) storage 128 and includes a DNN 112 and a DNN trainer 314 that is used to train and retrain (line 336) the DNN 112.

[0049] The computer vision module 114, the image acquisition module 304, the image pre-processing module 306, the object tracking module 404, the human input acquisition module 310, the tracking QMM 312, the cloud (i.e., remote) storage 128, the DNN trainer 314, and the DNN 112 operate like their corresponding modules in Fig. 3. The difference is that the AI ​​module 400 uses its cloud interface module 406 to communicate images received from the image acquisition module 304 over the network to the remote machine learning module 402 for the DNN 112 to use in its object detection, tracking, and recognition algorithms. For the image acquisition module 304 and the computer vision module 114, the cloud interface module 406 is effectively a "virtual DNN" that receives inputs from these two modules 304, 114 as if they were DNNs and forwards such inputs to the remotely residing DNN 112 (line 408). Thus, the processor(s) on the remote server 120 perform the operations of the DNN. Additionally, the machine learning module 402 is configured to return results (line 320) generated by the DNN 112 to the cloud interface module 406 of the AI ​​module 400. The cloud interface module 406 forwards these results (line 410) to the QMM 312, which determines from these results whether the DNN 112 recognized an object in the image.

[0050] If the QMM 312 determines that the DNN 112 was successful in identifying the object(s) in the image, the QMM 312 signals success (line 322). If the object is not identifiable in the image, the QMM 312 notifies the computer vision module 114 (line 324). The computer vision module 114 optionally sends the image to the cloud interface 414 for transmission to the remote DNN 112 (line 326). This image is derived from the original image and is focused on an area in the original image where the computer vision module 114 detected a change. The DNN 112 may attempt to identify the object in this focused image. If the DNN attempts during the second pass but is unsuccessful, the QMM 312 sends a request (not shown) to the human input acquisition module 310 for labeling information about the unidentifiable object in the original image.

[0051] Whether the DNN 112 succeeds or fails to recognize the object in this focused image during the DNN's second attempt, the computer vision module 114 transfers the original image (or an edited version of the original image) to the cloud storage 128 (line 328). The stored images are linked or associated (box 330) with the human-provided labels acquired by the human input acquisition module 310 (if the DNN fails) (line 332) or the labels generated by the DNN 112 in a successful second pass and transferred by the QMM 312 (if the DNN succeeds) (line 320). The DNN trainer 314 uses these images in the remote storage 128 and their associated identity information (i.e., labeling information) to retrain the DNN 112.

[0052] Each sensor module 102 (FIG. 1) can be configured to provide the functionality of the AI ​​module 400 locally, and the server system 120 (FIG. 1) is configured to provide the functionality of the machine learning module 402. In this embodiment, all sensor modules 102 in communication with the remote machine learning module 402 share the DNN 112 and image database 122 and contribute images for future retraining of the DNN 112. Generally, by having access to the same image database 122 and DNN 112, all sensor modules develop comparable image recognition capabilities.

[0053] FIG. 5 illustrates a functional block diagram of another embodiment of the object identification system 100 having an artificial intelligence (AI) module 500 in communication with a machine learning module 502. Each sensor module 102 (FIG. 1) can be configured to provide the functionality of the AI ​​module 500 locally, and the server system 120 (FIG. 1) is configured to provide the functionality of the machine learning module 502. In general, the object identification system of FIG. 5 differs from that of FIG. 4 in that certain functions of the AI ​​module 400 of FIG. 4 are performed remotely. Specifically, the image pre-processing module 306, the object tracking module 308, and the QMM 312 are part of the remote machine learning module 502, which also includes the DNN 112, the remote storage 128, and the DNN trainer 314. The AI ​​module 500 includes a cloud interface 504, an image acquisition module 304, and a human input acquisition module 310. The cloud interface 504 communicates with the image acquisition module 304 to forward input images received therefrom to the remote image pre-processing module 306 (line 450). The cloud interface 504 also receives a successful result indicator along with results from the remote QMM 312 (line 322) if the DNN 112 is able to recognize one or more objects in the input images, and the cloud interface 504 outputs the results for use by applications of the object identification system 100.

[0054] If the object is not identifiable in the image, the QMM 312 sends a signal to the computer vision module 114 (line 324). In response to the "DNN Failed" signal, the computer vision module 114 may send an image derived from the original image (or an edited version of the original image) focused on the area in the original image where the computer vision module 114 detected a change to the DNN 112 (line 326) to effectively perform a second pass at authentication in an attempt to identify the object in this focused image. The DNN 112 sends the results of this second attempt to the QMM 312 (line 320).

[0055] Regardless of whether the DNN 112 succeeds or fails to recognize the object in the focused image during the second attempt, the remote computer vision module 114 transfers to the cloud storage 128 the original image (or an edited version of the original image) in which the DNN 112 was initially unable to recognize the object (line 328).

[0056] If the object is not identifiable in this focused image, the QMM 312 sends a signal (line 327) to the AI ​​module 500, telling it to request human input. When the human input acquisition module 310 receives the human input, the cloud interface 504 sends the human input labels (line 332) to the cloud storage 128. Before being stored, the human input labels (line 332) are combined with or associated with the image coming from the remote computer vision module 114 (box 330).

[0057] If the object is identifiable in the focused image, the QMM 312 sends the labels generated by the DNN 112 (line 334) that are combined with or associated with the image sent by the computer vision module 114 to the cloud storage 128 (box 330). As before, the DNN trainer 314 uses these images and their associated labels in the image database 122 maintained in the remote storage 128 to retrain the DNN 112 (line 336).

[0058] Figure 6 illustrates an embodiment of an object tracking module 600 including an image pre-processing module (or image pre-processor) 306 and a computer vision module 114 for use in connection with the object identification system embodiments described in Figures 3 and 5. The object tracking module 600 includes a DNN 112. In an embodiment where the DNN 112 is remote to the object tracking module, such as in Figure 4, the object tracking module 600 has a cloud interface module 406 (Figure 4) that operates as a virtual DNN and communicates with the actual DNN.

[0059] In one embodiment, the DNN 112 has a deep learning architecture, e.g., a deep convolutional neural network having an input layer 602, an output layer 604, and multiple hidden layers (not shown). The hidden layers may include one or more convolutional layers, one or more fully connected layers, and one or more max pooling layers. Each convolutional and fully connected layer receives inputs from its previous layer and applies transformations to these inputs based on the current parameter values ​​of that layer. Exemplary architectures for implementing deep learning neural networks include, but are not limited to, the Darknet Open source Deep Neural Net framework available at the website pjreddie.com, and the Caffe framework available at the website caffe.berkeleyvision.org.

[0060] The DNN 112 is involved in two processes: object detection / recognition and training. For the purpose of object detection and recognition, an image 606 is provided as an input 608 to the DNN 112 from an image acquisition module. The image 606 includes a color image (e.g., RGB) and, optionally, a depth image. The color and depth images captured in real time at a given moment are linked as a pair. Such images may be passed through an image pre-processor 306, which generates image data 608 based on the processed images. The image pre-processor 306 may or may not modify the images 606 before they are passed to the DNN. In one embodiment, the image pre-processor 306 is configured to apply one or more image editing techniques determined to enhance the ability of the DNN to detect objects in the images by making such images robust (i.e., invariant) to changes in illumination. For RGB, one pre-processing algorithm uses a series of steps that counteract the effects of illumination variations, local shadows, and highlights. Algorithmic steps include gamma correction, differencing Gaussian filters, masking, and contrast adjustment. Depth data can be noisy and may have missing data depending on the circumstances under which the depth data is captured. Ambient light and highly reflective surfaces are major contributors to noise and missing data. This pre-filtering ensures that these artifacts are corrected and the data is well preserved. Pre-processing steps include ambient light filtering, edge-preserving smoothing, Gaussian blur, and time-varying blur. Once both the depth image and the RGB image are passed to the image pre-processor 306, the image pre-processor performs a blending transformation process that blends the RGB data with the depth data to generate image data 608. Examples of blending transformation processes include, but are not limited to, blending by concatenation or blending by interleaving, both of which are described in more detail below.

[0061] The image data 608 is passed to the input layer 602 of the DNN 112 (although not shown in FIG. 6, any label information obtained by human interaction is also passed to the input layer 602 for training, object recognition, or both). Both the image pre-processor 306 and the computer vision module 114 are configured to provide the image data 608 to the DNN 112. The image data 608 includes pixel values ​​for each of RGB (red, green, blue) obtained from the color image, and optionally for D (depth) taken from the depth image. FIG. 6 illustrates blending by concatenation, where each color and pixel value has its own channel, and the RGB image is blended with the depth image by concatenating the channels. In those embodiments in which D values ​​are submitted, less than the full color values ​​R, G, or B may be submitted. For example, R+D, G+D, B+D, R+G+D, R+B+D, G+B+D are examples where less than all three color (RGB) values ​​are submitted as input along with the D value, and each value submitted has its own channel.

[0062] Alternatively, interleaved blending can blend an RGB image with a depth image. In this blending technique, instead of concatenating the RGB and depth images to obtain channels, the channels of both images are blended in a way that preserves the original structure, i.e., the number of channels in the resulting image does not increase after blending from the number of channels in the original RGB image. One such example is as follows:

[0063] Consider an 8-bit three channel RGB image, i.e., the R channel has 8 bits, the G channel has 8 bits, and the B channel has 8 bits. Additionally, consider that the depth image is a single channel of 16-bit data, i.e., the D channel has 16 bits.

[0064] One method for combining data from multiple dimensions (i.e., channels) and packing the data into fewer dimensions (i.e., channels) is Morton Number Interleaving.

[0065] For example, a color pixel value [R,G,B] of [255,125,0] has an 8-bit binary representation of [11111111, 01111101, 00000000], with three 8-bit values ​​representing the R, G, and B channels, respectively.

[0066] For a 16-bit depth value, three 8-bit values ​​are derived. The first 8-bit value, referred to as D1, involves a conversion from a 16-bit value to an 8-bit value. This conversion is done by normalizing the decimal number equivalent to the 16-bit depth value and multiplying the normalized value by the maximum value of an 8-bit number (i.e., 255). For example, consider an original 16-bit depth value [D] that has a decimal value of 1465. Normalizing the decimal value 1465 involves dividing this decimal value by the maximum decimal value that can be represented in 16 bits, i.e., 65025. Thus, the multiplied and normalized decimal value of D1 = (1465 / 65025) x 255 = 6 (rounded up). The 8-bit binary representation of D1 is [00000110].

[0067] The next two bytes are obtained by partitioning the original 16-bit depth value [D] into two 8-bit bytes, called D2 and D3. For example, the aforementioned 16-bit depth value [D] of

[1465] has a binary representation of [0000010110111001]. The 8-bit D2 byte corresponds to the first byte of the 16-bit depth value [D], which is [00000101], and the 8-bit D3 byte corresponds to the second byte of the 16-bit depth value [D], which is [10111001]. Thus, [D2, D3] = [00000101, 10111001].

[0068] The three bytes [D1, D2, D3] derived from the original depth value [D] are [00000110, 00000101, 10111001]. As mentioned before, the 8-bit RGB values ​​of the three channels are [11111111, 01111101, 00000000].

[0069] From the three channels of depth value bytes [D1, D2, D3] and the three channels of RGB value bytes [R, G, B], a 16-bit three channel image is generated by appending the depth values ​​to the RGB values, such as [RD1, GD2, BD3], by interleaving the Morton order. As with the previous example, the Morton order interleaving generates three 16-bit channels: [1111111100000110, 0111110100000101, 0000000010111001]. This technique is performed for each pixel of the corresponding image 606 (i.e., the RGB image and its associated depth image). The result is a three channel image with both depth and color information. It should be understood that Morton order interleaving is merely one example of a technique for interleaving depth and color data for a given pixel, and that other interleaving techniques may be used without departing from the principles described herein.

[0070] Similar to blending with concatenation techniques, less than all color values ​​R, G, or B may be interleaved with depth values. For example, R+D, G+D, B+D, R+G+D, R+B+D, G+B+D are examples where less than all three color (RGB) values ​​are submitted as input along with the D value. In these cases, there is a separate channel for each color and depth interleave. When using less than three RGB channels, any of the D1, D2, D3 depth channels may be useful for interleaving. For example, combinations such as R+D, G+D, and B+D each require only one channel, while combinations such as R+G+D, R+B+D, G+B+D each have two channels. When using only one RGB channel, D1 is the preferred choice because the D1 depth channel contains the entire depth information. When using two color channels, two depth channels, e.g., D2 and D3, are used for interleaving (D2 and D3 together have the entire depth information). To illustrate, again using color pixel values ​​[R, G, B] of [255, 125, 0] and original depth value of

[1465] , the combination of R+G+D produces the following 16-bit two channel [RD2, GD3] input data [1111111100000110, 0111110100000101], with D2 and D3 being the selected depth channels. In general, the ability to achieve object detection benefits from having more information available, rather than less. Thus, blending by concatenation, which preserves all available color and potential depth data, may produce better detection performance than blending by interleaving, which reduces the number of channels and allows less than all color and depth to be used. A case in which blending by interleaving can be advantageous over blending by concatenation is when it comes to training speed.

[0071] The output layer 604 generates outputs 320 that are passed to the QMM, which may be via the cloud interface 406 (FIG. 4). The outputs 320 may include labels for objects detected in the image-based analysis, an indication of where the objects are located in the image, and a value representing the probability of correct recognition (i.e., confidence).

[0072] The DNN 112 also communicates with the DNN trainer to receive updates to parameter values ​​used in retraining.

[0073] In one embodiment, the DNN 112 is composed of two deep neural networks (not shown) operating in parallel. One neural network receives R, G, and B pixel values, while the other neural network receives R, G, B, and D values. Each neural network attempts to recognize one or more objects in the image provided based on the submitted image data 608. Each generates an output. The two outputs can be compared and / or combined for the purpose of confirming and / or extending each other's decision. Consider, for example, that an RGB neural network generates a result that one package has been detected in a particular area of ​​an image, and an RGBD neural network generates a result that two packages have been detected in the same particular area. Comparing the probabilities of the two neural networks (and logic circuits), the difference is resolved and the result is declared to be complete as either one or two packages.

[0074] The computer vision module 114 communicates with the QMM to receive a "DNN Fail" signal if the DNN 112 fails to successfully recognize an object in an image. Upon receiving such a signal, the computer vision module 114 outputs an image corresponding to the original image in which the DNN failed to identify the object (line 328). This image may become associated with human-supplied labeling information 332 (e.g., in response to a prompt from the AI ​​module if the DNN failed to identify the object). This labeling information and image combination 610 is passed to storage, where it becomes part of the image database 122. Alternatively, the combination 610 includes the image and labeling information coming from the QMM 312 (generated by the DNN 112) if the DNN successfully identified the object during the second pass (line 334).

[0075] FIG. 7 illustrates one embodiment of the computer vision module 114. In general, the image acquisition module may capture an image having a higher resolution than the DNN 112 requires as input. For example, the image acquisition module may acquire an image 606-1 of 3840×2160 pixels, but the DNN 112 requires an image of 416×416 pixels (e.g., based on required computation speed). Thus, the image pre-processing module 306 downsamples the original image 606-1 to generate a resized image 606-2 that matches the input resolution of the DNN 112. The DNN 112 attempts to detect an object(s) in the resized image 606-2 and sends the result to the QMM 312. The QMM determines according to the result whether the DNN 112 successfully detected one or more objects in the image.

[0076] Further, the image pre-processing module 306 sends the original image 606-1 and the resized image 606-2 to the computer vision module 114. The computer vision module 114 includes a change tracking module 700 in communication with the change locating module 702. In one embodiment, the computer vision module 114 performs a multi-pass verification process if the DNN 112 fails to detect an object in the image 606-2. In the event of a failed object detection, the QMM sends a signal to the change tracking module 700, which in response executes the change tracking program code 226 (FIG. 2) to identify an area 704 in the original image 606-1 where a change appears (with respect to the previously captured image).

[0077] The change location module 702 uses this information to generate an image 606-3 that is focused on the region 704 in the original image that contains the detected change. The focused image 606-3 has a resolution that matches the input resolution of the DNN 112. To achieve this resolution, the change location module 702 may need to reduce or increase the size of the region of change 704. The focused image 606-3 is passed to the DNN 112, which attempts to detect objects in this image. The computer vision module 114 sends the resized image 606-2 to storage (local or remote) and marks the boundaries of the focused region 704 as these boundaries are transformed into the resized image 606-2. The boundary information includes the row, column, height, and width of the relevant region in the resized image 606-2.

[0078] In storage, the resized image 606-2 is associated with a label name provided by human input (if the DNN fails to recognize the object in the focused image 606-3) or with a label generated by the DNN 112 (if the DNN successfully recognizes the object in the focused image 606-3). The resized image 606-2, the marked boundary, and the label information are used together in subsequent retraining of the DNN 112.

[0079] 8 illustrates one embodiment of a process 800 for object tracking according to principles described herein. A person enters an area designated to hold objects (step 802). Such an area could be, for example, a room for storing packages, or shelves that hold boxes of cereal or jars of mayonnaise.

[0080] The object identification system 100 registers the identity of the person (step 804). The registration can be done automatically, i.e., without the conscious involvement of the person. For example, the sensor module 102 can communicate wirelessly with a device carried by the person, such as, for example, a key fob or a smartphone. Alternatively, the controller 104 can perform facial recognition. As another example of a technique for obtaining the identity of a person, the person can discreetly identify themselves, such as by providing a name tag for scanning, entering a PIN code or password, submitting biometric information (e.g., a fingerprint or retina scan), speaking to enable voice recognition, etc. In another embodiment, the object identification system 100 identifies the person using skeletal tracking (i.e., the skeletal structure of the individual) and registers the skeletal structure. In addition to registering the person, the object identification system 100 can record the person's arrival time at the holding area.

[0081] In step 806, the object identification system 100 associates the person with one or more objects in the holding area. The association can be made directly from user input or indirectly based on actions performed by the user and observed by the system. As an example of a direct association, the system can explicitly request that the person provide information about the purpose of the visit, such as placing or removing an object, and the identity of each object involved in that purpose. The person can provide this information by any number of input techniques, for example by scanning a label on a package to be placed. Alternatively, the person can identify what the object is by typing the name of the object or by speaking to the system using voice recognition and speech-to-text techniques. After receiving information about each affected object, the system associates the object with the identity of the registered person.

[0082] As an example of indirect association, the object identification system 100 can detect actions performed by a person within a holding area. For example, through image processing, the system can detect that an object has been placed on or removed from a shelf, and then associate the newly placed or removed object with the registered person's identity.

[0083] In step 808, the object identification system 100 attempts to recognize what the object is. Recognition may result from information the user directly supplies to the system, for example when the user inputs that "the item is a microscope," or from a previous determination, for example when the system has detected that an object of known identity has been removed, or from object recognition, for example when the system has run its object recognition algorithms on an image of a newly detected object. In one embodiment, the system automatically requests human interaction, i.e., asking a human to identify the object that has been placed, moved, or removed. Such a request may occur before, during, or after the system attempts its own object recognition.

[0084] The decision to request human interaction may be based on the value of the threshold derived by the controller 104 in its attempt at recognizing the object from the captured image. For example, in step 810, if the threshold value is above a first (e.g., upper) threshold, the system considers the object recognized with high confidence and may dispense with human interaction, if the threshold value is less than the first threshold but greater than a second (e.g., lower) threshold, the system considers the object recognized but with medium confidence, and if the threshold value is below the second threshold, the system concludes that no object was recognized in the image. The system may request a person to confirm or correct the system's identification (step 812) if the determined threshold value is below the upper threshold but above the lower threshold, and may request a person to provide an identification (step 814) if the determined threshold value is below the lower threshold. Fewer or more than two thresholds may be used without departing from the principles described herein. Additionally, the system may request confirmation even if the threshold value exceeds an upper threshold, or may request the identity of the object if object recognition is ambiguous, inaccurate, or unsuccessful.

[0085] The actions of the person in the holding area may change the layout of the objects on the support surface. New objects may be placed, objects may be removed, objects may be moved to another location, or any combination of these. The new arrangement of objects creates irregularities of different perspectives and changed angles in association with the image and depth sensors of one or more sensor modules. Machine learning not only learns how objects look through both color and depth, but can now learn various perspectives for each object as it is placed in different locations in the area. This machine learning compensates for the dynamic perspective of the objects seen by the image sensor and learns how an identified object could be the same object if placed in a different area in the viewing area and at a different angle and depth in the shelf. Thus, the images captured by the image sensor now provide an opportunity to improve object recognition through machine learning techniques. The system retrains the neural network with these newly captured images where the neural network was not able to identify the object (at least initially) and required labeling information about the object provided by the user or the neural network during multi-pass authentication. The system can also record the departure time of the person when they leave the holding area, and then associate the person's arrival and departure times with the object.

[0086] Consider the following illustration as an example of the operation of one embodiment of the object identification system 100: Alice enters a room with several shelves. She carries a microscope and a smartphone. The smartphone has Bluetooth enabled. The controller 104 connects to the smartphone and communicates with it to verify the identity of the person as Alice. Furthermore, the controller verifies the time that Alice entered the room, for example, as Thursday, April 16, 2019 at 1:42 PM. Alice places the microscope on one of the shelves. Through image processing of the image captured by the image sensor, the controller detects the location of the object and the microscope. Furthermore, the controller can use machine learning to recognize the object as a microscope. The controller can ask Alice whether the controller correctly recognized the object to confirm its decision. If the controller fails to recognize the placed object, the controller can ask Alice to identify the object, which she can input electronically or verbally, depending on the configuration of the object identification system. Alternatively, the system may be configured to ask Alice for the identity of the object, regardless of its own recognition of the object. The system may then train its neural network, either locally or remotely on a server, immediately or at a later time, with the images captured by the microscope and the information provided by Alice, if any. Then, when Alice departs from the room, the controller records the time of departure as 1:48 PM on Thursday, April 16, 2019.

[0087] Bob enters the room and submits his identity to the controller using a PIN code. The controller registers Bob and the time he entered, for example, as Thursday, April 16, 2019 at 2:54 PM. The controller identifies Bob and recognizes his regular use of the microscope from his past patterns of habits. The controller asks, audibly or by a message displayed on the display screen, if Bob is looking for the microscope. If Bob responds in the affirmative, the controller illuminates the light source and directs the light beam to the location on the shelf where the microscope resides. Bob removes the microscope from the shelf and departs the room with it. The system records Bob's departure time as Thursday, April 16, 2019 at 2:56 PM, and records that Bob took the microscope. By linking the arrival of the microscope by Alice, the removal of the microscope by Bob, the times of such actions, and the presence of the microscope during them, all confirmed by video recording, the system has thus proven the chain of custody of the microscope. This chain of custody principle can extend to other areas of endeavor, such as the process for handling evidence. In this context, chain of custody means recording the sequence of custody (possession) and location of physical objects over time as they enter, move within, and leave a holding area. The object identification system understands who brought a particular piece of evidence into an evidence room, who took the evidence from that room, and the exact location of the evidence within the room in between, even if they have moved to a different section within the field of view of the image sensor.

[0088] 9 illustrates one embodiment of a process 900 for recognizing objects in an image. In step 902, the image acquisition module 304 receives a captured image. Optionally, the image pre-processing module 305 edits the image (step 904), including reducing the resolution of the image, before sending the image to the DNN 112 (step 906). The DNN attempts to detect and identify objects in the received image (step 908).

[0089] If the QMM 312 determines that the DNN successfully identified one or more objects in the image (step 910), the object identification system 100 uses information about each identified object, e.g., for object tracking purposes (step 912). The particular use of the object information depends on the application in which the object identification system is being used.

[0090] Alternatively, if the QMM determines that the DNN has failed in its attempt to identify an object in the image (step 910), the AI ​​module asks a human to identify the object (step 914). After the human provides the requested information, the optionally preprocessed image (produced in step 904) is stored in the image database 122 along with the human-provided labeling information for later use in retraining the DNN (step 916).

[0091] In one embodiment, shown by the dotted lines in FIG. 9, the AI ​​module performs the multi-pass authentication process described in connection with FIG. 7 (step 918). If the DNN is unable to identify the object in the second pass, in step 920, the AI ​​module asks a human to identify the object (step 914), and the optionally pre-processed image is stored in the image database 122 with the human-provided labeling information (step 916). Alternatively, if the DNN successfully identifies the object in the image in the second attempt, in step 920, the optionally pre-processed image is stored with the DNN-provided labeling information for later use in retraining the DNN (step 916). Thus, if the DNN is able to identify the object in the second pass, the image is saved with the identifying label generated by the DNN, and if the DNN is unable to identify the object in the second attempt, the image is stored with the human-supplied labeling information.

[0092] 10 illustrates one embodiment of a process 1000 for retraining a DNN with images in which the DNN does not initially identify objects. Periodically, the object identification system 100 determines that the DNN should be retrained with images stored in the image database. Such retraining may be based on a schedule (e.g., nightly, weekly, etc.).

[0093] For purposes of retraining the DNN 112 (FIG. 1), the DNN trainer 314 (e.g., FIG. 3) searches for images (step 1002) in the image database 122. Such images are stored in local storage 124 or remote storage 128. The image database 122 includes an initial training set of images as well as each image in which the DNN 112 was initially unable to identify an object.

[0094] Based on the images in the image database, the DNN trainer 314 runs the program code for neural network training 222 (FIG. 2), which generates a weighting file for a new neural network (step 1004). The new weighting file includes weighting (i.e., parameter) values ​​for each of the "neurons" of the DNN 112. To update the DNN, i.e., to retrain the DNN, the DNN trainer sends this new weighting file to the DNN, which is then applied by the DNN to its neuron structure for object detection and recognition in received images.

[0095] Typically, the DNN trainer maintains a copy of the current weights file for the DNN. Retraining a DNN can be done either fully or partially. When retraining fully, the entire DNN is trained from scratch, i.e., the current weights file is erased and replaced with a newly generated weights file. It is as if the DNN was trained from scratch again and trained for the first time. This retraining uses the initial training set of images and each additional image added to the image database that was not initially identified.

[0096] In the case of partial retraining, the retraining may be focused on a specific layer of the DNN. For example, considering that the DNN has 10 hidden layers, retraining may be performed only on the 7th, 8th, and 9th hidden layers, the working principle of which is to avoid performing full DNN training, which may take time, when focused retraining may be sufficient. In this embodiment, only those parameter values ​​in the current weighting file associated with the neurons of the 7th, 8th, and 9th hidden layers are changed. The new weighting file generated by the DNN trainer and sent to the DNN is a mixture of new parameter values ​​for the neurons of the 7th, 8th, and 9th hidden layers and old parameter values ​​for the rest of the DNN.

[0097] As an example of machine learning in action, consider that Alice places a microscope on a shelf. If upon Alice's enrollment, the object identification system 100 does not recognize the object, the system may ask Alice to identify the placed object, and she may respond that the object is a microscope. The object identification system 100 further captures one or more images of the object on the shelf and associates each captured image with the information provided by Alice (i.e., the object is a microscope). The DNN trainer uses each captured image and the information provided by Alice to train the neural network 112. This training may be an initial training of the system to identify the microscope, or may be cumulative to the system's current capabilities. In either case, after training, the object identification system is better suited to identify the microscope.

[0098] Although described with respect to detecting, tracking, and recognizing objects, the machine learning techniques described herein extend to detecting, tracking, and recognizing faces, skeletal structures, body positions, and movements of people in captured images. In a similar manner to how images of objects are used to train deep neural networks to improve object recognition, images of faces can be used to train such networks to improve face recognition for the purposes of user enrollment, and images of skeletal features such as hands, arms, and legs can be used to train such networks to improve for the purposes of identifying and tracking individual people and objects.

[0099] As will be appreciated by those skilled in the art, aspects of the systems described herein may be embodied as systems, methods, and computer program products. Thus, aspects of the systems described herein may be embodied entirely in hardware, entirely in software (including, but not limited to, firmware, program code, resident software, microcode), or a combination of hardware and software. All such embodiments may be generally referred to herein as circuits, modules, or systems. Furthermore, aspects of the systems described herein may be in the form of a computer program product embodied in one or more computer readable medium(s) having computer readable program code embodied on the computer program product.

[0100] Any combination of one or more computer readable media may be utilized. The computer readable medium may be a computer readable signal medium or a computer readable storage medium. The computer readable medium may be a non-transitory computer readable storage medium, examples of which include, but are not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination thereof.

[0101] As used herein, a computer-readable storage medium may be any tangible medium that may contain or store a program for use by or in connection with an instruction execution system, apparatus, device, computer, computing system, computer system, or any programmable machine or device that inputs, processes, and outputs instructions, commands, or data. A non-exhaustive list of specific examples of computer readable storage media includes an electrical connection having one or more wires, a portable computer diskette, a floppy disk, a hard disk, random access memory (RAM), read-only memory (ROM), a USB flash drive, non-volatile RAM (NVRAM or NOVRAM), erasable programmable read-only memory (EPROM or flash memory), a flash memory card, electrically erasable programmable read-only memory (EEPROM), optical fiber, a portable compact disc read-only memory (CD-ROM), a DVD-ROM, an optical storage device, a magnetic storage device, or any suitable combination thereof.

[0102] A computer-readable signal medium may include a propagated data signal having computer-readable program code embodied therein, for example, in baseband or as part of a carrier wave. Such a propagated signal may take any of a variety of forms, including, but not limited to, electrical, magnetic, optical, or any suitable combination thereof. A computer-readable signal medium is not a computer-readable storage medium, but may be any computer-readable medium that can communicate, propagate, or carry a program for use by or in connection with an instruction execution system, apparatus, or device. As used herein, a computer-readable storage medium is neither a computer-readable propagated signal medium nor a propagated signal.

[0103] The program code may be embodied as computer readable instructions stored on or in a computer readable storage medium, for example as source code, object code, interpreted code, executable code, or a combination thereof. Any standard or proprietary programming or interpreted language may be used to generate the computer executable instructions. Examples of such languages ​​include Python, C, C++, Pascal, JAVA, BASIC, Smalltalk, Visual Basic, and Visual C++.

[0104] Transmission of the program code embodied on the computer readable medium may occur using any suitable medium, including, but not limited to, wireless, wired, optical fiber cable, radio frequency (RF), or any suitable combination thereof.

[0105] The program code may execute entirely on the user's device, partially on the user's device, as a stand-alone software package, partially on the user's device and partially on a remote computer, or entirely on a remote computer or server. Any such remote computer may be connected to the user's device through any type of network, including a local area network (LAN) or a wide area network (WAN), or the connection may be to an external computer (e.g., through the Internet using an ISP (Internet Service Provider)).

[0106] Furthermore, the methods described herein may be implemented in special purpose computers, programmed microprocessors or microcontrollers and peripheral integrated circuit element(s), hardwired electronic or logic circuits such as ASICs or other integrated circuits, digital signal processors, discrete element circuits, programmable logic devices such as PLDs, PLAs, FPGAs, PALs, etc. In general, any device capable of implementing a state machine, which is capable of implementing the methods proposed herein, may be used to implement the principles described herein.

[0107] Furthermore, the disclosed methods can be readily implemented in software using object or object-oriented software development environments, providing portable source code that can be used on a variety of computer or workstation platforms. Alternatively, the disclosed systems can be partially or fully implemented in hardware using standard logic circuits or VLSI designs. The decision to use software or hardware to implement a system in accordance with the principles described herein depends on the system, the specific functionality, and the speed and / or efficiency requirements of the particular software or hardware system, or microprocessor or microcomputer system being utilized. However, the methods illustrated herein can be readily implemented in hardware and / or software using any known or later developed system or structure, device, and / or software by one of ordinary skill in the art from the functional descriptions provided herein and with a general basic knowledge of computer and image processing technology.

[0108] Moreover, the disclosed methods may be readily implemented in software running on a programmed general purpose computer, special purpose computer, microprocessor, etc. In these examples, the systems and methods of the principles described herein may be implemented as programs embedded in a personal computer, such as JAVA or CGI scripts, as resources resident on a server or graphics workstation, as plug-ins, etc. The systems may also be implemented by physically incorporating the systems and methods into a software and / or hardware system.

[0109] While the above principles have been described in conjunction with several embodiments, it is apparent that many alternatives, modifications, and variations will be or will be apparent to those skilled in the applicable art. Reference to "one embodiment" or "an embodiment" or "another embodiment" means that a feature, characteristic, structure, or characteristic described in connection with an embodiment is included in at least one embodiment described herein. References to specific embodiments within this specification do not necessarily all refer to the same embodiment. Features illustrated or described in connection with one exemplary embodiment may be combined with features of other embodiments. It is therefore intended to embrace all such alternatives, modifications, equivalents, and variations that are within the spirit and scope of the principles described herein.

[0110] Disclose the following: (1) 1. An object identification system, comprising: an image sensor configured to capture an image of an object disposed within an area designated to hold the object; a deep neural network trained to detect and recognize objects in images that resemble those objects contained within the designated areas; a controller in communication with the image sensor to receive images captured by the image sensor and in communication with the deep neural network, the controller submitting an image captured by the image sensor to the deep neural network; Detecting a location of the detected object while the detected object is within the area, regardless of whether the deep neural network was able to recognize the detected object in the submitted image or not; if the deep neural network is unable to recognize the object detected in the submitted image, retraining the deep neural network using the submitted image, information about the object, and a location of the object; 1. An object identification system comprising: one or more processors configured to: (2) 2. The object identification system of claim 1, wherein the controller is further configured to, in response to the deep neural network being unable to recognize the detected object in the submitted image, obtain labeling information for the detected object, associate the labeling information with a version of the image submitted to the deep neural network, and store the version of the image and the associated labeling information in an image database used to retrain the deep neural network. (3) The object identification system of claim 2, further comprising a human input acquisition module configured to acquire the labeling information from a user in response to a request from the controller when the deep neural network is unable to recognize the detected object in the submitted image. (4) the controller is further configured to: find areas in the image where changes occur where the deep neural network is unable to recognize the detected object; generate a version of the image focused on the changes in the areas; and submit the version of the image to the deep neural network to determine whether the deep neural network is able to recognize the detected object in the second version of the image; the controller is further configured to: obtain labeling information for the detected object, regardless of whether the deep neural network recognizes the detected object in the submitted version of the image; associate the obtained labeling information with the version of the image submitted to the deep neural network; and store the version of the image and the associated labeling information in an image database used to retrain the deep neural network; and The object identification system of (1), wherein the controller is further configured to obtain the labeling information from the deep neural network if the deep neural network recognizes the detected object in the submitted version of the image. (5) 13. The object identification system of claim 1, wherein the deep neural network is a first deep neural network and further includes a second deep neural network configured to operate in parallel with the first deep neural network, each of the first and second deep neural networks generating an output based on image data obtained from the image, and the image data submitted to the first deep neural network is different from the image data submitted to the second deep neural network. (6) An object identification system in accordance with (1), further comprising a depth sensor having a field of view substantially matching a field of view of the image sensor, wherein the depth sensor captures depth pixel values ​​of an image within the field of view of the depth sensor, and during training or object recognition, when the image is submitted to the deep neural network, the depth pixel value and less than three pixel values ​​taken from the group consisting of R (red), G (green), and B (blue) are submitted to the deep neural network as image data. (7) 2. The object identification system of claim 1, wherein the deep neural network resides on a remote server system, and the controller further comprises a network interface for communicating with the deep neural network on the server system. (8) 1. A method for identifying and detecting an object, the method comprising: registering the identity of any person who visits an area designated for holding an object; capturing an image of an area designated to hold the object; submitting a version of the image to a deep neural network trained to detect objects in images that resemble those objects contained within the designated area; detecting an object in the version of the image; associating the registered identity of the person with the detected object; Detecting a location of the detected object while the detected object is within an area designated to hold the object, regardless of whether the deep neural network was able to recognize the detected object in the submitted image. if the deep neural network is unable to recognize the detected object, retraining the deep neural network using the version of the image, information about the object, and the location of the object; , a method comprising: (9) The method further comprising: obtaining labeling information for the object detected in the version of the image in response to the deep neural network being unable to recognize the object detected in the version of the image; and associating the labeling information with the version of the image; storing the version of the captured image and associated labeling information in an image database used to retrain the deep neural network; 9. The method of claim 8, wherein the step of obtaining labeling information for the object detected in the version of the image in response to the deep neural network being unable to recognize the detected object in the version of the image includes obtaining the labeling information from user-supplied input. (10) the version of the image is a first version of the image, and the method comprises: locating areas in the first version of the image where changes occur when the deep neural network is unable to recognize the object detected in the first version of the image; generating a second version of the image focused on the areas where the changes were found; and submitting the second version of the image to the deep neural network to determine whether the deep neural network is able to recognize the detected object in the second version of the image; obtaining labeling information for the object detected in the first version of the image, regardless of whether the deep neural network recognizes the object detected in the second version of the image; Associating the labeling information with the first version of the image; storing the first version of the captured image and associated labeling information in an image database used to retrain the deep neural network; 9. The method of claim 8, wherein the step of obtaining labeling information for the object detected in the first version of the image obtains the labeling information from a deep neural network if the deep neural network recognizes the detected object in the second version of the image. (11) 9. The method of claim 8, wherein the deep neural network is a first deep neural network, and further comprising the step of submitting image data obtained from the version of the image to the first deep neural network and a second deep neural network in parallel, wherein the image data submitted to the first deep neural network is different from the image data submitted to the second deep neural network. (12) 9. The method of claim 8, wherein the step of submitting a version of the image to the deep neural network includes submitting a depth pixel value and less than three pixel values ​​taken from the group consisting of R (red), G (green), and B (blue) to the deep neural network as image data. (13) The object identification system of (1), wherein the one or more processors are configured to register an identity of a person who visits an area designated to hold the object, and associate the registered identity of the person with an object detected in the image submitted to the deep neural network.

Claims

[Claim 1] 1. An object identification system, comprising: an image sensor configured to capture an image of an object disposed within an area designated to hold the object; a deep neural network trained to detect and recognize objects in images that resemble those objects contained within the designated areas; a controller in communication with the image sensor to receive images captured by the image sensor and in communication with the deep neural network; The controller registering the identity of persons who visit an area designated for holding said objects; submitting images captured by the image sensor to the deep neural network; Associating the identity of the enrolled person with objects detected in images submitted to the deep neural network; and retraining the deep neural network using the submitted image if the deep neural network is unable to recognize an object detected in the submitted image; tracking the location of detected objects within the specified area, regardless of whether the deep neural network is able to recognize the detected objects within the submitted images; and locating a location of the detected object in an image submitted to the deep neural network while the object is within the designated area, regardless of whether the deep neural network was able to recognize the object in the submitted image, and retraining the deep neural network using the submitted images if the deep neural network is unable to recognize the object within the submitted area; one or more processors configured to Further, the controller obtaining labeling information for an object detected in the submitted image when the deep neural network is unable to recognize the object; Associating the obtained labeling information with versions of images submitted to the deep neural network; storing versions of the images and associated labeling information in an image database used to retrain the deep neural network; 1. An object identification system configured to:

Images