Object Recognition

Abstract

The present invention provides an object recognition system for recognizing objects in an environment, the system comprising tags, each tag associated with an individual object and including a tag memory for storing an object model, which is a computational model indicating a relationship between the individual objects and a shape coding layer; a sensing system including a sensing device for sensing the environment, a sensing system memory for storing a sensing device model, which is a computational model indicating a relationship between sensor data captured using the sensing device and the shape coding layer, and a sensing system processor for receiving the object model from a transceiver, retrieving the sensing device model from the sensing system memory, obtaining sensor data from the sensing device indicative of the environment, and analyzing the sensor data using the sensing device model and the object model, thereby recognizing objects in the environment.

Description

[Technical field]

[0001] The present invention relates to systems and methods for recognizing objects, and in one particular example, for recognizing objects using machine learning models. [Background technology]

[0002] Reference herein to any prior publication (or information derived therefrom) or any known matter is not, and should not be construed as, an acknowledgement or admission, or any form of suggestion that the prior publication (or information derived therefrom) or known matter forms part of the common general knowledge in the field of endeavor to which this specification pertains.

[0003] The process of using machine learning to recognize objects is known. Conventional techniques fall into one of two broad categories: generating specific models for recognizing individual objects, and generating generalized models for recognizing multiple different objects. Models for recognizing specific individual objects tend to be robust and perform well, but suffer from a lack of generality, whereas generalized models tend to be capable of recognizing multiple objects, but perform poorly. Additionally, regardless of the type of model used, these tend to be specific to the sensing modality used to train the model, meaning that it is necessary to have multiple different models to recognize the same object using different sensing modalities. Summary of the Invention [Means for solving the problem]

[0004] In one broad form, one aspect of the invention seeks to provide an object recognition system for recognizing one or more objects in an environment, the system comprising: one or more tags, each tag associated with an individual object in use, each tag including a tag memory configured to store an object model, the object model being a computational model that indicates a relationship between the individual objects and a shape coding layer, a tag memory, a tag transceiver, and a tag processing unit configured to cause the tag transceiver to transmit the object model; a sensing system configured to sense an environment, a sensing system memory configured to store the sensing device model, the sensing device model being a computational model that indicates a relationship between sensor data captured using the sensing device and the shape coding layer, a sensing system transceiver, and one or more sensing system processing units configured to receive the one or more object models from the transceiver, retrieve the sensing device model from the sensing system memory, obtain sensor data from the sensing device indicative of the environment, and analyze the sensor data using the sensing device model and the one or more object models, thereby recognizing one or more objects in the environment.

[0005] In one embodiment, the environment includes a plurality of objects, each associated with an individual tag, and one or more processing devices are configured to recognize different ones of the plurality of objects.

[0006] In one embodiment, each object model is either specific to a particular object and specific to a particular object type.

[0007] In one embodiment, the one or more processing devices are configured to use the sensor data and the sensing device model to generate shape-encoded parameters, and to use the shape-encoded parameters and the object model to recognize the object.

[0008] In one embodiment, the tag processing device is configured to cause the tag transceiver to transmit the object model at least one of periodically and in response to a model request message received by the tag transceiver.

[0009] In one embodiment, the sensing device model is either a model specific to a particular sensing device and a model specific to a particular sensing device type.

[0010] In one embodiment, the sensing device includes at least one of an imaging device, a lidar, a radar, and an acoustic mapping device.

[0011] In one embodiment, the object model is at least one of a generative adversarial neural network, a neural network, and a recurrent neural network.

[0012] In one embodiment, the sensing device model is at least one of a generative adversarial neural network, a neural network, and a recurrent neural network.

[0013] In one broad form, one aspect of the invention seeks to provide an object recognition method for recognizing one or more objects in an environment, the system comprising: one or more tags, each tag associated with an individual object in use, each tag including a tag memory configured to store an object model, the object model being a computational model that indicates a relationship between the individual objects and a shape coding layer; a tag memory, a tag transceiver, and a tag processing unit configured to cause the tag transceiver to transmit the object model; a sensing system configured to sense an environment, a sensing system memory configured to store the sensing device model, the sensing device model being a computational model that indicates a relationship between sensor data captured using the sensing device and the shape coding layer; a sensing system transceiver, and one or more processing units configured to receive the one or more object models from the transceiver, retrieve the sensing device model from the sensing system memory, obtain sensor data from the sensing device indicative of the environment, and analyze the sensor data using the sensing device model and the one or more object models, thereby recognizing one or more objects in the environment. 1. A system for generating models for use in recognizing individual objects in an environment, the system comprising one or more processing devices configured to obtain sensor data indicative of individual objects in the environment from a sensing device, use the sensor data to generate object models indicative of relationships between the individual objects and a shape coding layer, and generate a sensing device model indicative of the relationship between the sensor data captured using the sensing device and the shape coding layer.

[0014] In one embodiment, one or more processing devices are configured to obtain sensor data indicative of individual objects in an environment from a plurality of different sensing devices and to use the sensor data to at least one of generate an object model and generate a plurality of sensing device models.

[0015] In one embodiment, one or more processing devices are configured to obtain sensor data indicative of a plurality of different objects in an environment and to use the sensor data to at least one of generate a plurality of object models including individual object models for each of the different objects, and generate a sensing device model.

[0016] In one embodiment, each object model is either specific to a particular object and specific to a particular object type.

[0017] In one embodiment, the sensing device model is either a model specific to a particular sensing device and a model specific to a particular sensing device type.

[0018] In one embodiment, the object model is at least one of a generative adversarial neural network, a neural network, and a recurrent neural network.

[0019] In one embodiment, the sensing device model is at least one of a generative adversarial neural network, a neural network, and a recurrent neural network.

[0020] In one broad form, one aspect of the invention seeks to provide a system for generating models for use in recognizing individual objects in an environment, the system comprising one or more processing devices configured to obtain sensor data indicative of individual objects in the environment from a sensing device, use the sensor data to generate object models indicative of relationships between the individual objects and a shape coding layer, and generate a sensing device model indicative of the relationship between the sensor data captured using the sensing device and the shape coding layer.

[0021] In one broad form, one aspect of the invention seeks to provide a method for generating models for use in recognizing individual objects in an environment, the method comprising, in one or more processing devices, acquiring sensor data from a sensing device indicative of individual objects in the environment, generating an object model using the sensor data indicative of a relationship between the individual objects and a shape coding layer, and generating a sensing device model indicative of the relationship between the sensor data captured using the sensing device and the shape coding layer.

[0022] In one broad form, one aspect of the invention seeks to provide a computer program product for generating models for use in recognizing individual objects in an environment, the computer program product comprising computer-executable code that, when executed by one or more suitably programmed processing devices, causes the one or more processing devices to acquire sensor data indicative of individual objects in an environment from a sensing device, and use the sensor data to generate object models indicative of relationships between the individual objects and a shape coding layer, and to generate a sensing device model indicative of the relationship between the sensor data captured using the sensing device and the shape coding layer.

[0023] It will be understood that the broad aspects of the invention and its individual features can be used in combination and / or independently, and reference to separate broad aspects is not intended to be limiting. Further, it will be understood that method features can be performed using a system or apparatus, and system or apparatus features can be implemented using a method.

[0024] Various examples and embodiments of the present invention will now be described with reference to the accompanying drawings. [Brief description of the drawings]

[0025] [Figure 1] FIG. 1 is a schematic diagram of an example of an object recognition system. [Diagram 2]FIG. 2 is a schematic diagram of an example of a model structure of the object recognition system of FIG. [Diagram 3] 1 is a flow chart of an example of an object recognition process. [Figure 4] FIG. 1 is a schematic diagram of an example of a model training process. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0026] An example of an object recognition system will now be described with reference to FIGS.

[0027] In this example, the system includes one or more tags 110, each of which is associated with a particular object 101 in use. The tags 110 are typically electronic tags 110 capable of communicating using a short-range wireless communication protocol such as Bluetooth, Bluetooth Low Energy (BLE), Wi-Fi, etc. The tags 110 are associated with the object 101 by attaching or integrating the tag 110 to the object 101 in some manner, typically depending on the physical form factor of the tag 110 and the object. Such association is made such that the tag 110 is generally provided within the same environment as the object, although this is not required and other configurations may be used, such as simply placing the tag within the environment containing the object.

[0028] A tag may have any form, but typically includes components such as a tag memory 112 configured to store an object model, a tag transceiver 113 to enable wireless communication, e.g., to transmit or receive messages, and a tag processing device 111 configured to perform any required operations. The components may be of any suitable form and may include a short-range wireless transceiver, such as a Bluetooth transceiver, which may optionally be formed from a custom integrated circuit, such as a Bluetooth system-on-chip (SOC), coupled to or including an integrated antenna and any other components. A tag processing device may be any electronic processing device, such as a microprocessor, a microchip processor, a logic gate configuration, firmware optionally associated with implementing logic, such as a Field Programmable Gate Array (FPGA), or any other electronic device, system, or configuration. For ease of explanation, the remainder of the description refers to a processing device, but it will be understood that multiple processing devices may be used with processing distributed among the processing devices as needed, and that references to the singular encompass the plural configuration, and vice versa.

[0029] One or more sensing systems 120 are provided that communicate with the tags 110 using a short-range wireless communication protocol either directly or through an intermediate network or device (not shown), such as a wireless communication network. The sensing system 120 typically includes sensing devices 124 configured to sense the environment. The nature of the sensing devices and the manner in which they operate will depend on the intended application and may include imaging devices, such as cameras, or depth or range sensing configurations, such as lidar, radar, acoustic depth mapping devices, etc. Although reference is made to a single sensing device, it will also be appreciated that multiple sensing devices may be provided, for example as part of a multi-modal sensing configuration.

[0030] The sensing system 120 also includes a sensing system memory 122, typically configured to store a sensing device model, a sensing system transceiver 123, configured to communicate with tags, e.g., using a short-range wireless protocol, and one or more sensing system processors 121. The components may be of any suitable form and may include a short-range wireless transceiver, such as a Bluetooth transceiver, which may optionally be formed from a custom integrated circuit, such as a Bluetooth system-on-chip (SOC), coupled to or including an integrated antenna and any other components. The sensing system processor may be any electronic processing device, such as a microprocessor, a microchip processor, a logic gate configuration, firmware optionally associated with implementing logic, such as a field programmable gate array (FPGA), or any other electronic device, system, or configuration. For ease of explanation, the remainder of the description refers to a processing device, but it will be understood that multiple processing devices may be used, with processing distributed among the processing devices as necessary, and that reference to the singular encompasses the plural configuration, and vice versa.

[0031] From the above, it will be appreciated that the sensing system 120 can be of any suitable form and in certain embodiments may include one or more of a processing system, a computer system, a smartphone, a tablet, a mobile computing device, optionally with sensing devices coupled to or integrated with the sensing devices. However, it will be appreciated that this is not required and that the sensing system may be distributed and may include physically separate sensing devices and a processing system that includes the sensing devices in the environment, for example, and a separate computing device that communicates with the sensing devices via a wired or wireless communication network, etc. In certain embodiments, the sensing system is integrated into a robot or other autonomous or semi-autonomous configuration and used to assist the robot in recognizing objects in the environment, allowing the robot to interact with the objects as needed.

[0032] In use, tags 110.1, 110.2 associated with different objects store object models 231.1, 231.2, which are computational models that indicate the relationship between the respective objects and the shape coding layer 233. Similarly, different sensing systems 120.1, 120.2 store sensing device models 232.1, 232.2, which are computational models that indicate the relationship between sensor data captured using the respective sensing devices and the shape coding layer 233. The nature of the computational models will vary depending on the preferred implementation, but typically the models will be some form of neural network, as will be explained in more detail below.

[0033] The shape coding layer 233 typically defines a number of different shape parameters that can be used in identifying different objects. It will be appreciated that these parameters are derived during the creation of the computational model and are typically built up over time as more object models are generated for different objects. The parameters can be of any suitable form and are typically derived during the machine learning process used to generate the object model and the sensing device model. The parameters may thus include real-world physical parameters such as the dimensions of the object, but also other parameters derived by machine learning techniques that are useful in distinguishing between objects. In practice, the sensing device model converts the sensor data captured by the sensing device into values ​​of the different shape parameters, for example by generating one or more feature vectors based on the sensor data. The sensing system processor can then use the object model being used to map specific shape parameter values ​​to different objects, for example by applying the feature vectors to the different object models to identify any matches.

[0034] In use, object models are stored on the tags and transmitted to the sensing system enabling them to be used by the sensing system in recognising objects; an example of this process will now be described with respect to FIG.

[0035] In this example, in step 300, the tag transmits the stored object model. This transmission can be done periodically, for example by having the tag transmit the model periodically, for example every few minutes, using a broadcast or other similar message, so that the model can be received by any sensing system in the vicinity of the tag. Alternatively, the transmission can be performed in response to a request from a sensing system, in which case the model may be transmitted only to the requesting sensing system, or it can be broadcast in a similar manner. In either case, as mentioned above, the object models are typically neural networks, meaning that they are typically on the order of a few megabytes in size, making wireless transmission of the models feasible.

[0036] At step 310, the sensing system 120, and in particular the processing unit 121, receives the object model and at step 320 retrieves the sensing device model from the internal memory 122. At step 330, the processing unit 121 acquires sensor data from the sensing devices 124. The sensor data is typically acquired by scanning the environment using the sensing devices, thus typically capturing details of one or more objects as well as the surrounding environment.

[0037] It will be appreciated that these steps may be performed in any order, so that, for example, retrieval of the model may be performed before transmission and reception of the object model, etc.

[0038] Once the sensor data, object model, and sensing device model have been obtained, the sensing system processor identifies the object using the object model and sensing device model and the sensor data at step 340. Thus, in one example, this process involves applying the sensor data to the sensing device model to generate values ​​for shape-encoding parameters, which are then applied to the object model to determine whether a corresponding object has been captured within the sensor data, although it will be appreciated that other suitable techniques may be used depending on the nature of the model and the preferred implementation.

[0039] In any event, it will be appreciated that in practice this enables a sensing system, which may be incorporated in, for example, a robot, to receive object models from various kinds of objects in an environment, which can then be used to capture sensor data in the environment and recognize objects in the environment from the sensor data.

[0040] This approach offers several important advantages over traditional object recognition approaches. First, it uses object-specific models, so that, for example, an object model of a cup is used to identify a cup in the environment. As mentioned above, such object-specific models can be very robust, resulting in very high levels of object recognition accuracy.

[0041] Secondly, by the object models being transmitted by the tags periodically or on demand, this means that the sensing system does not need to access object models of objects that are not present in the environment. This eliminates the need for the sensing system to store or retrieve models of objects that are not present in the environment, reducing storage and / or data transmission requirements.

[0042] Furthermore, this means that the sensing system knows which object models are relevant for any given environment, and it only has to apply the sensor data to the relevant models. This avoids applying the sensor data to irrelevant models, reducing computational requirements and thus allowing for faster identification of present objects. This may also help to increase the robustness of the recognition process, for example avoiding inconsistent results that may occur when objects are very similar, since the processing unit of the sensing device will constantly use only object models that are relevant to the objects in the environment.

[0043] Third, the presence of an additional sensing device model means that the object model can be trained using a different modality than that used for sensing. For example, an object model for identifying a cup can be trained using only optical images captured using an imaging device. Previously, any such model was only useful to enable the cup to be recognized using optical sensing. However, in the above approach, the common shape coding layer means that the sensor data can be mapped to the shape coding layer and from there to the object, so that even if the cup model was created using only optical images as training data, a depth-based sensor such as LiDAR can recognize the cup.

[0044] The above approach can therefore be used to enable a sensing system, such as a robot, which enters an environment, such as a room, to recognize and therefore identify a series of different objects within the environment. This can be done more quickly and accurately than can be achieved using existing techniques, and more importantly, can be sensor independent, as long as a sensing device model is created.

[0045] Some further features are now described.

[0046] As mentioned above, in one example the environment includes multiple objects, each object associated with an individual tag. In this example, the processing unit 121 is configured to recognize different ones of the multiple objects. Thus, the sensing device can sense and recognize the multiple objects using sensor data that captures all of the objects.

[0047] Each object model may be specific to a particular object and / or specific to a particular type of object. For example, an object model may be derived that allows all cups to be identified as cups, while models may be derived to distinguish between different cups, for example to identify cups of a particular type, or even to identify cups that belong to a particular individual. It will be appreciated that which techniques are used will depend on the preferred implementation, and that techniques may be used in combination, for example to allow some individual objects to be identified, while other objects may be distinguished based solely on type.

[0048] Similarly, sensing device models may be specific to a particular sensing device and / or type of sensing device, which may depend on the variations that occur between different sensing devices of the same type. For example, lidars may be manufactured with a high degree of consistency, so a model for one lidar may be applicable to other lidars, whereas there may be a high degree of variation between different imaging devices, meaning that the model may have to be device specific.

[0049] In one example, a processing device is configured to generate shape-encoded parameters using the sensor data and a sensing device model, and to recognize objects using the shape-encoded parameters and an object model, although it will be appreciated that other approaches may be used. For example, the output of the sensing device model may be applied as an input to an object model, regardless of the form of the output. Alternatively, a composite model may be generated and used to directly recognize objects in the sensor data.

[0050] As previously mentioned, the tag processing device can be configured to cause the tag transceiver to transmit the object model periodically and / or in response to a model request message received by the tag transceiver. It will be appreciated that this latter approach reduces transmission requirements and therefore typically extends battery life, although the approach used will depend on the preferred implementation.

[0051] Sensing devices can include imaging devices, lidar, radar, and acoustic mapping devices.

[0052] The object model and / or the sensing device model may be any one or more of a neural network, a generative adversarial neural network (GAN), and / or a recurrent neural network, etc.

[0053] Typically the network is developed using machine learning techniques, an example of which is now described with respect to FIG.

[0054] In step 400, sensor data is captured for one or more objects. The sensor data is typically tagged with information that identifies individual objects and is optionally conditioned or augmented, e.g., to remove portions of the sensor data that do not contribute to object recognition. These processes can be performed using manual processes, etc.

[0055] In step 410, training is performed using the data, for example by using the data to train a GAN, which typically involves training the GAN's discriminator and generator using the sensor data and object identification, which is then used in step 420 to generate an object-specific model, and in step 430 to generate a sensing device-specific model.

[0056] In this regard, it will be appreciated that sensor data indicative of individual objects in an environment can be obtained from multiple different sensing devices and used to generate an object model as well as multiple sensing device models. Similarly, a single sensing device can be used to sense multiple objects and used to generate a single sensing device model and multiple object models. Training models in this manner is known in the art and therefore will not be described in further detail.

[0057] Throughout this specification and the appended claims, unless the context requires otherwise, the term "comprise" and variants such as "comprises" or "comprising" will be understood to imply the inclusion of a stated integer or group of integers or steps but not the exclusion of any other integer or group of integers.

[0058] It will be understood that numerous variations and modifications will become apparent to those skilled in the art, and all such variations and modifications become apparent to those skilled in the art and should be considered to be within the broad spirit and scope of the invention as described above.

Claims

1. 1. An object recognition system for recognizing one or more objects in an environment, comprising: a) one or more tags, each tag associated with a particular object in use, each tag comprising: i) a tag memory configured to store an object model, the object model being a computational model that indicates the relationship between the individual objects and a shape coding layer; ii) a tag transceiver; and iii) a tag processing device configured to cause said tag transceiver to transmit said object model; one or more tags, including b) a sensing system comprising: i) a sensing device configured to sense the environment; ii) a sensing system memory configured to store a sensing device model, the sensing device model being a computational model that describes a relationship between sensor data captured using the sensing device and the shape coding layer; iii) a sensing system transceiver; iv) one or more sensing system processors, (1) receiving one or more object models from the transceiver; (2) retrieving the sensing device model from the sensing system memory; (3) acquiring sensor data indicative of the environment from the sensing device; (4) one or more sensing system processors configured to analyze the sensor data using the sensing device model and the one or more object models, thereby recognizing the one or more objects in the environment; a sensing system including: Equipped with Object recognition system.

2. the environment includes a plurality of objects; Each object is associated with an individual tag, the one or more processing devices are configured to recognize distinct ones of the plurality of objects; The system of claim 1 .

3. Each object model is a) a model specific to a particular object, and b) a model specific to a particular object type; Either 3. The system according to claim 1 or 2.

4. the one or more processing devices: a) using the sensor data and the sensing device model to generate shape-encoded parameters; b) using the shape-encoded parameters and the object model to recognize an object; It is configured as follows:

3. The system according to claim 1 or 2.

5. The tag processing device a) periodically, and b) in response to a model request message received by said tag transceiver; configured to cause the tag transceiver to transmit the object model in at least one of 3. The system according to claim 1 or 2.

6. the sensing device model: a) a model specific to a particular sensing device, and b) a model specific to a particular sensing device type; Either 3. The system according to claim 1 or 2.

7. The sensing device a) an imaging device; b) Rider; c) radar, and d) acoustic mapping equipment; at least one of:

3. The system according to claim 1 or 2.

8. The object model: a) Generative Adversarial Neural Networks, b) neural networks, and c) recurrent neural networks; At least one of 3. The system according to claim 1 or 2.

9. the sensing device model: a) Generative Adversarial Neural Networks, b) neural networks, and c) recurrent neural networks; At least one of 3. The system according to claim 1 or 2.

10. 1. An object recognition method for recognizing one or more objects in an environment, said method comprising: a) one or more tags, each tag associated with a particular object in use, each tag comprising: i) a tag memory configured to store an object model, the object model being a computational model that indicates the relationship between the individual objects and a shape coding layer; ii) a tag transceiver; and iii) a tag processing device configured to cause said tag transceiver to transmit said object model; one or more tags, including b) a sensing system comprising: i) a sensing device configured to sense the environment; ii) a sensing system memory configured to store a sensing device model, the sensing device model being a computational model that describes a relationship between sensor data captured using the sensing device and the shape coding layer; iii) a sensing system transceiver; iv) one or more processing units, (1) receiving one or more object models from the transceiver; (2) retrieving the sensing device model from the sensing system memory; (3) acquiring sensor data indicative of the environment from the sensing device; (4) one or more processing devices configured to analyze the sensor data using the sensing device model and the one or more object models, thereby recognizing the one or more objects in the environment; a sensing system including: Equipped with Object recognition methods.

Images