Object recognition system and method for object recognition

The object recognition system addresses detection failures in conventional systems by using feature determination for equipment and containers, ensuring accurate and reliable equipment tracking and placement in assembly lines despite environmental changes.

GB2626313BActive Publication Date: 2025-05-14AUTOCRAFT SOLUTIONS GRP LTD
View PDF 1 Cites 0 Cited by

Patent Information

Application Number
GB2023000513
Authority / Receiving Office
GB · GB
Patent Type
Patents
Current Assignee / Owner
Filing Date
2023-01-13
Publication Date
2025-05-14
Estimated Expiration
2043-01-13

AI Technical Summary

Technical Problem

Conventional camera-based recognition systems in industrial automation fail to accurately detect tools or parts due to wear and tear, changes in lighting conditions, and environmental variations, leading to inefficiencies and inconsistencies in assembly lines.

Method used

An object recognition system that determines a set of features for both equipment and their containers, including area, color, and shape, to ensure accurate detection regardless of visual state changes, using image-capture devices and a controller to track equipment placement and sequence.

Benefits of technology

Ensures fail-safe and efficient detection of equipment in assembly lines, reducing disruptions and improving throughput by maintaining accuracy despite variations in lighting and equipment condition.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure 00000001_0000
    Figure 00000001_0000
  • Figure 00000002_0000
    Figure 00000002_0000
Patent Text Reader

Abstract

Method of recognising and tracking objects (e.g. tools, machine parts), comprising: from a first plurality of images taken at a first time, determining a first set of physical characteristics (appeara
Need to check novelty before this filing date? Find Prior Art

Description

TECHNICAL FIELD The present disclosure relates generally to the field of industrial automation and more specifically, to an object recognition system and a method for object recognition, for example, for use in an assembly line. BACKGROUND With advances in manufacturing, more and more assembly lines are being automated. In an assembly line, a typical goal is to achieve the highest quality of product at the lowest cost with the shortest lead time. Generally, in industrial automation, image-processing technology using cameras is used in numerous processes, such as product manufacturing, assembly, and visual inspection. Conventional camera-based recognition systems rely on recognising simple shapes of tools, for example, recognising common geometric shapes like round, triangle, square, etc. However, it is observed that as a result of continuous use, there is wear and tear in such tools, which affects the shape of the tools, thereby causing the conventional recognition systems to fail in detecting such tools or misidentifying the tool. Thus, such conventional recognition systems need to be re-trained on the new deformed shape of the tool to enable its re-detection, which is not desirable. In certain scenarios, it is also observed that uneven lighting conditions or any change in lighting in the production environment also affects the recognition of the tools and parts, causing inefficiency on a given assembly line. For example, an operator looking for parts may cause an unwanted delay in each product assembly. In another example, producing a defective sub-assembly or product due to incorrect use of parts or tools causes inconsistency in operations as the more time a defect remains undetected, the more cost and correction time is added, leading to inefficiencies. Therefore, in light of the foregoing discussion, there exists a need to overcome the aforementioned drawbacks associated with conventional camera-based recognition systems. SUMMARY The present disclosure provides an object recognition system and a method for object recognition. The present disclosure provides a solution to the existing problem of detection failure or wrong detection of tools or parts used in an assembly line because of prolonged use of such tools or parts and the sensitivity problem of conventional recognition systems to environmental or visual conditions, such as a change in lighting conditions. An aim of the present disclosure is to provide a solution that overcomes at least partially the problems encountered in the prior art and provide an improved object recognition system and an improved method for object recognition for accurate and fail-safe detection of any equipment (e.g., kitting parts and tools) in a predetermined area (e.g., in an assembly line environment) irrespective of any variation in a visual state (e.g., a change in lighting, shading or other visual factors) in the predetermined area. One or more objects of the present disclosure are achieved by the solutions provided in the enclosed independent claims. Advantageous implementations of the present disclosure are further defined in the dependent claims. In one aspect, the present disclosure provides an object recognition system that includes a controller configured to obtain a plurality of images of a predefined area from one or more image-capture devices at a first time instant when the predefined area is in a first visual state. Hie predefined area includes one or more equipment containers and a plurality of equipment. Further, the controller is configured to determine a first set of features of the one or more equipment containers and a second set of features of the plurality of equipment in the predefined area from the obtained plurality of images. The first set of features and die second set of features correspond to physical characteristics of the one or more equipment containers and the plurality of equipment, respectively. Furthermore, the controller detects at a second time instant whether the plurality of equipment is present in their respective positions in the one or more equipment containers based on the determined first set of features and the second set of features. Moreover, the predefined area is in the first visual state or a second visual state different from the first visual state at the second time instant. The object recognition system determines features for not only the plurality of equipment (i.e., the kiting parts and tools) in the predefined area (e.g., an area in an assembly line) but also for the one or more equipment containers (e.g., kitting boxes) in the predefined area. Such holistic feature sets determination for both the plurality of equipment and the one or more equipment containers makes the object recognition system accurate and fail-safe for detection of any equipment and insensitive to any variation in a visual state (e.g., lighting condition, shading condition, etc.)in tire predetermined area. Thus, even if there is any change in the shape of the equipment due to wear and tear or if the images captured for detection have some variation due to uneven lighting conditions, still the object recognition system does not fail to recognise each piece of equipment and their placement in their respective positions in one or more equipment containers. In an implementation form, the determining of the first set of features comprises calculating an area of a base portion of one or more storage sections of the one or more equipment containers from the obtained plurality of images The equipment containers (e.g., kitting boxes) are designed such that only specific equipment (e.g., a specific kitting part or tool) fits into a specific shaped storage section (e.g., a specific shaped compartment). The object recognition system is able to calculate the area of the base portion that is visible and understand whether the correct part or tool is present with improved accuracy. The use of the calculated area of the base portion for equipment detection makes the object recognition system insensitive to a visual state variation (e.g., any change in environmental or visual factors, such as lighting condition, shading condition, background lights, or a change in a shape, or orientation of the equipment) in the predefined area. In a further implementation form, the determining of the first set of features further comprises identifying a colour of the base portion of the one or more storage sections of the one or more equipment containers from the obtained plurality of images. Some equipment containers may include multiple storage sections (e.g., multiple compartments), while some equipment containers may include one storage section (e.g., one compartment) to hold or store equipment. Beneficially, each storage section in the one or more equipment containers may be of a different colour. The colour acts as one of the 3 parameters used to recognise the correct storage place for a particular equipment. In other words, the storage section bases (e.g., the compartment bases) are coloured in such a way as to maximise system recognition. In a further implementation form, the determining of the first set of features further comprises determining a shape of the base portion of the one or more storage sections of the one or more equipment containers from the obtained plurality of images. The shape of the base portion of the one or more storage sections may be complementary' to a corresponding shape of a particular equipment. In an implementation, each storage section (e.g., a compartment) in the one or more equipment containers may be of different shape to improve equipment recognition so that the right tools are placed at the right position (i.e., at their designated area) in an equipment container. In a further implementation form, the determining of the second set of features comprises determining a colour and a shape of each of the plurality of equipment. The determination of the second set of features, such as the shape and colour of each of the equipment and correlating such features with the first set of features of one or more equipment containers makes the object recognition system accurate and fail-safe for detection of any equipment and insensitive to any variation in a visual state (e.g., lighting condition, shading condition, etc.) m the predetermined area. In a further implementation form, the controller is further configured to track the plurality of equipment in the predefined area based on the determined first set of features and the second set of features. Beneficially, the first set of features and tine second set of features are synergistic to each other to improve tracking of the plurality of equipment, removing inconsistency from the assembly process, enabling a better throughput, less disruption and traceability for all equipment (e.g., kitting parts or tools). For instance, the combination of the first set of features and the second set of features used for tracking enables overcoming changes in visual states; for example, any differences in the colour and shape of individual tools and components do not affect recognition of such tools and components. Furthermore, the combination of the first set of features and the second set of features in tracking provides a capability to the object recognition system to recognise when an equipment (e.g., a kitting part or tool) is present in its respective position in an equipment container and when it is used. In a further implementation form, the object recognition system further comprises a plurality of actuators, where the tracking of the plurality of equipment further includes causing one or more actuators of the plurality of actuators to pick-up one or more equipment from one or more respective positions in a sequential order in a product assembly line. The controller may keep track of the equipment picked up by the one or more actuators of the plurality of actuators and further may track the sequence of the equipment required to perform a particular task, for example, in an assembly line. The tracking of the sequence of the equipment used to perform a particular task removes inconsistency from the assembly process, enabling a better throughput, less disruption and traceability for all equipment. In a further implementation form, a change from the first visual state to the second visual state corresponds to a change in one or more of: a lighting condition, a shading condition, a current shape of a given equipment, or an orientation of each of the plurality of equipment in the one or more equipment containers, and wherein each of the one or more imagecapture devices is a colour camera device. The object recognition system can handle not only any changes in lighting conditions but also the shading condition and any changes in the shape or orientation of equipment, thereby improving the system's reliability and accuracy under different variations in visual states of the predefined area. In another aspect, the present disclosure provides a method for object recognition. The method includes obtaining, by a controller, a plurality of images of a predefined area from one or more image-capture devices at a first time instant when the predefined area is in a first visual state. The predefined area includes one or more equipment containers and a plurality of equipment. The method further includes determining, by the controller, a first set of features of the one or more equipment containers and a second set of features of the plurality of equipment in the predefined area from the obtained plurality of images. Further, the first set of features and the second set of features correspond to physical characteristics of the one or more equipment containers and the plurality of equipment, respectively. The method further includes detecting, by the controller, at a second time instant whether tire plurality of equipment are present in their respective positions in the one or more equipment containers based on the determined first set of features and the second set of features. Further, the predefined area is in the first visual state or a second visual state different from the first visual state at the second time instant. The method for object recognition achieves all the advantages and effects of die object recognition system. It is to be appreciated that all the aforementioned implementation forms can be combined. It has to be noted that all devices, elements, circuitry, units, and means described in the present application could be implemented in the software or hardware elements or any kind of combination thereof. All steps which are performed by die various entities described in the present application, as well as the functionalities described to be performed by the various entities are intended to mean that the respective entity is adapted to or configured to perform the respective steps and functionalities. Even if, in the following description of specific embodiments, a specific functionality or step to be performed by external entities is not reflected in the description of a specific detailed element of that entity that performs that specific step or functionality, it should be clear for a skilled person that these methods and functionalities can be implemented in respective software or hardware elements, or any kind of combination thereof. It will be appreciated that features of the present disclosure are susceptible to being combined in various combinations without departing from the scope of the present disclosure as defined by the appended claims. Additional aspects, advantages, features and objects of the present disclosure would be made apparent from the drawings and the detailed description of the illustrative implementations construed in conjunction with the appended claims that follow. BRIEF DESCRIPTION OF THE DRAWINGS The summary above, as well as the following detailed description of illustrative embodiments, is better understood when read in conjunction with the appended drawings. For the purpose of illustrating the present disclosure, exemplary constructions of the disclosure are shown in the drawings. However, the present disclosure is not limited to specific methods and instrumentalities disclosed herein. Moreover, those in the art will understand that the drawings are not to scale. Wherever possible, like elements have been indicated by identical numbers. Embodiments of the present disclosure will now be described, by way of example only, with reference to the following diagrams wherein: FIG. 1 is a block diagram illustrating an object recognition system, in accordance with an embodiment of the present disclosure; and FIG. 2 is a flowchart of a method for object recognition, in accordance with an embodiment of the present disclosure. In the accompanying drawings, an underlined number is employed to represent an item over which the underlined number is positioned or an item to which the underlined number is adjacent. A non-underlined number relates to an item identified by a line linking the non-underlined number to the item. When a number is non-underlined and accompanied by an associated arrow, the non-underlined number is used to identify a general item at which the arrow is pointing. DETAILED DESCRIPTION OF EMBODIMENTS The foilowing detailed description illustrates embodiments of the present disclosure and ways in which they can be implemented. Although some modes of carrying out the present disclosure have been disclosed, those skilled in the art would recognise that other embodiments for carrying out or practicing the present disclosure are also possible. FIG. 1 is a diagram illustrating an object recognition system, in accordance with an embodiment of the present disclosure. With reference to FIG. 1, there is shown a block diagram of an object recognition system 100. The object recognition system 100 comprises a server 102 and one or more image-capture devices 106. The server 102 includes a controller 104. The server 102 is communicatively coupled to the one or more image capture devices 106 via a communication network 108. There is further shown one or more equipment containers 110 and a plurality of equipment 114 (such as a first equipment 114A, a second equipment 114B, up to a Nth equipment 114N), and a plurality of actuators 116 present in a predefined area 118. In an example, the predefined area 118 may be a work area of a product assembly line. Each of the one or more equipment containers 110 may include one or more storage sections 112 (e.g., compartments). The object recognition system 100 is configured to detect the presence of the plurality of equipment 114 in their respective positions in the one or more equipment containers 110. The object recognition system 100 may be used to accurately detect and track equipment (i.e., parts or tools) used in a product assembly process. Thus, the object recognition system 100 finds application in the automotive industry, original equipment manufacturers (OEMs), product assemblers (e.g., battery' assembly line) and remanufacturers, where there is a need to assemble product parts using variable equipment (e.g., different kitting parts and tools) that also need to be traced. The controller 104 may include suitable logic, circuitry, interfaces, or code that is configured to obtain a plurality of images from the one or more image-capture devices 106 for processing. In an exemplary implementation, the controller 104 may be implemented in the server 102. Examples of the controller 104 may include but are not limited to, a processor, a digital signal processor (DSP), a microcontroller, a state machine, a data processing unit, a graphics processing unit (GPU), and other processors or control circuitry'. The one or more image-capture devices 106 may include suitable logic, circuitry', interfaces, or code that is configured to capture the plurality of images of a predefined area 118, for example, a work area of a product assembly line. In an implementation, there may be a plurality of image-capture devices (e.g., colour camera devices) distributed and strategically positioned in the predefined area, for example, the work area of the product assembly line. The plurality of image-capture devices may be configured to capture a plurality of field-of-views (FOVs) comprising one or more equipment containers 110 and one or more equipment of the plurality of equipment 114. In an implementation, each of the one or more image-capture devices 106 is a colour camera device. Other examples of the one or more image-capture devices 106 include but are not limited to a colour image sensor, a colour camcorder, and an artificial intelligence-based colour camera. Ilie communication network 108 may be a wired or a wireless network. The one or more equipment containers 110 may also be referred to as a kitting container, a kitting box, a tool kit, a toolbox, an equipment box, or an equipment kit. In an example, some of the equipment containers may include multiple storage sections, for example, compartments, to accommodate multiple equipment while some equipment containers may include one storage section (e.g., one compartment) to accommodate one equipment only. In an implementation, the one or more equipment containers 110 may include a plurality of specific shaped compartments to accommodate equipment of corresponding shapes and sizes. Each of the one or more equipment containers 110 (e.g., the kitting boxes) are designed in such a way that a specific equipment (i.e., specific kitting parts or tools) fit into specific shaped storage sections (e.g., specific shaped compartments) such that only those equipment can fit. In other words, a given equipment having a specific shape and size is kept (when not in use) on a corresponding base portion of a specific storage section (i.e., a specific compartment) that has a complementary size and shape (e.g., like a mould) to accommodate a given equipment with the right fit. In operation, tire controller 104 is configured to obtain a plurality of images of the predefined area 118 from the one or more image-capture devices 106 at a first time instant when the predefined area 118 is in a first visual state. The predefined area 118 comprises one or more equipment containers 110 and a plurality of equipment 114. In an example, the one or more image-capture devices 106 (e.g., one or more colour camera devices) are installed in a work area of a product assembly line to monitor the predefined area 118. The one or more image-capture devices 106 may be positioned in the predefined area 118 to allow capture of FOVs that include one or more equipment containers 110 and one or more equipment of the plurality of equipment 114. The plurality of images are captured, for example, at a first time instant or a first time period, in a first visual state. For example, at a first time instant, the plurality of images may be captured with a first level of brightness, a first type of shading, where each equipment has an initial shape and an initial orientation when kept at corresponding container from the one or more equipment containers 110. The plurality of images may be captured periodically or continuously and transmitted to the server 102. In an implementation, the first visual state refers to a lighting condition (e.g., a brightness level), a shading condition, or any other visual factors captured at the first time instant in the predefined area. For example, the first visual state may also indicate a current shape of a given equipment at the first time instant, or an orientation of each of the plurality of equipment 114 in the one or more equipment containers 110 at the first time instant. In other words, the first visual state refers to what each image-capture device sees in its FOV to capture the plurality of equipment 114 and the one or more equipment containers 110 and their surroundings in the predefined area. Thus, any change in the first visual state may also influence how the plurality of equipment 114 and the one or more equipment containers 110 visually appears (e.g., may affect image quality) in the captured plurality of images of the predefined area. The controller 104 is further configured to determine a first set of features of the one or more equipment containers 110 and a second set of features of the plurality of equipment 114 in the predefined area 118 from the obtained plurality of images. The first set of features corresponds to physical characteristics of the one or more equipment containers 110. Similarly, the second set of features corresponds to the physical characteristics of the plurality of equipment 114. Examples of the physical characteristics include, but are not limited to a size, shape, colour, or other physical appearance factors (such as smoothness, roughness, and reflective or non-reflective surface). The first set of features refers to image features of the one or more equipment containers 110. The first set of features are extracted from the obtained plurality of images to facilitate distinguishing and identification of each of the one or more equipment containers 110 in the predefined area 118, such as the work area of the product assembly line. Similarly, the second set of features refers to image features of each of the plurality of equipment 114. The second set of features are also extracted along with the first set of features from the obtained plurality of images. The second set of features are extracted to facilitate distinguishing of different equipment from each other and for identification of each of the plurality of equipment 114 in the predefined area 118. It is to be understood that there may usually be many objects in each image of the obtained plurality of images. However, the controller 104 is configured to selectively extract the features for each of the one or more equipment containers 110 and each of the plurality of equipment 114 and exclude other objects for feature extraction. In an implementation, the determining of the first set of features includes calculating an area of a base portion of the one or more storage sections 112 (e.g., compartments) of the one or more equipment containers 110 from the obtained plurality of images. The controller 104 is able to calculate the area of the base portion that is visible and determine whether the correct equipment (i.e., a correct kitting part or tool) is present in a given storage section (e.g., a given compartment) of a given equipment container. In an example, the controller 104 may calculate the area of the base portion by evaluating a number of pixels that defines the visible base portion of a storage section (i.e., one compartment) of an equipment container with respect to a total number of pixels of a given image of the predefined area 118. In another example, the controller 104 calculates the area of the base portion by evaluating the shape or area vectors of the base portion of each storage section in the one or more equipment containers 110 visible in the given image. Further, each base portion of the one or more storage sections 112 (e.g., compartments) of the one or more equipment containers 110 is complementary to the size and shape of a particular equipment of the plurality of equipment 114. The area of the base portion of each storage section (e.g., each compartment) is calculated to find if the correct equipment is placed at the correct position at a given storage section of an equipment container. In an implementation, the determining of the first set of features further includes identifying a colour of the base portion of the one or more storage sections 112 (e.g., compartments) of the one or more equipment containers 110 from the obtained plurality of images. The base portions of the storage sections 112 (i.e., the compartment bases) are coloured in such a way as to maximise recognition by the object recognition system 100. In an implementation, the controller 104 identifies the colour of the base portion by using a colour detection algorithm. Hie colour detection algorithm facilitates the controller 104 to identify pixels in the plurality of images that matches a specified colour or a colour range. In an implementation, the colour of one base portion of the one or more storage sections 112 (e.g., compartments) of the one or more equipment containers 110 may be complementary' to or the same as the colour of a particular equipment from the plurality of equipment 114. For example, the colour of the first equipment 114A may be red, and the 11 colour of the base portion of a first storage section of a first equipment container may also be red. This predefined colour similarity maximises the recognition ability of the controller 104 to determine the correct placement of a given equipment in a given storage section of a given equipment container. In an exemplary implementation, each base portion of the one or more storage sections 112 in one equipment container may have a unique colour. In an implementation, the determining of the first set of features further comprises determining a shape of the base portion of the one or more storage sections 112 (e.g., compartments) of the one or more equipment containers 110 from the obtained plurality of images. Hie controller 104 determines the shape of the base portion by using one or more shape detection algorithms. In an example, firstly, edge detection may be performed to detect edges of the base portion displayed in the plurality of images. Further, image segmentation may be performed in which each base portion is labelled, and then the shape of each base portion is determined by the controller 104. The shape of a given base portion of one or more storage sections 112 (e.g., compartments) of the one or more equipment containers 110 is complementary- and approximately congruent to the shape of a given equipment of the plurality of equipment 114. In an implementation, the determining of the second set of features includes determining the colour and the shape of each of the plurality of equipment 114. All the equipment do not have the same shape and colour. Such distinctive characteristics of the plurality of equipment 114 are exploited so that their correct placement at correct positions in the one or more equipment containers 110 may be determined from the obtained plurality of images. In an example, the colour and the shape of each of the plurality of equipment 114 may be complementary to the colour and the shape of each storage section (e.g., each compartment) of the one or more storage sections 112. For instance, the first equipment 114A may have a red colour and may be circular in shape. In such a case, the base portion of a given compartment may also be designed in red colour and circular in shape, which facilitates the controller 104 to detect the correct placement of the first equipment 114A. The determination of the second set of features, such as the shape and colour of each of the equipment and correlating such features with the first set of features of one or more equipment containers makes the object recognition system accurate and fail-safe for detection of any equipment in the predetermined area of storage sections 112 (e.g., compartments). The controller 104 is further configured to detect at a second time instant whether the plurality of equipment 114 are present in their respective positions in the one or more equipment containers 110 based on the determined first set of features and the second set of features. At the time of detection, the predefined area 118 is in the first visual state or a second visual state different from the first visual state at the second time instant. In other words, the detection is insensitive to any variation in a visual state (e.g., lighting condition, shading condition, etc) in tire predefined area 118. The determined first set of features of each of the one or more equipment containers 110 is correlated with the determined second set of features of each of the plurality of equipment 114 (i.e., the kiting parts and tools) used, for example, in the product assembly line. Based on the correlation, the controller 104 is able to detect whether each of tire plurality of equipment 114 is present in their respective positions (e.g., in their designated storage sections) in the one or more equipment containers 110 or is in use by an operator. For example, the controller 104 analyses plurality of images and correlates the determined first set of features with the second set of features to detect whether the first equipment 114A is present in a first position (e.g., a first storage section) in the one or more equipment containers 110 or not. Similarly, the controller 104 analyses plurality of images and correlates the determined first set of features with the second set of features to detect whether the second equipment 114B is present in a second position (e.g., a second storage section) in the one or more equipment containers 110 or not. Unlike conventional systems that require attaching different Radio Frequency Identification (RFID) chips and antenna devices to different equipment in an assembly line, the disclosed object recognition system 100 does not require such additional devices to be attached to the equipment or any kitting boxes. Moreover, in certain scenarios, there may be an environmental variation (i.e., a variation in the first visual state) in the predefined area 118 at a second time instant or second time period different from the first time instant or the first time period when the plurality of images may be initially obtained. In an example, a change from the first visual state to the second visual state corresponds to a change m a lighting condition, a shading condition, a current shape of a given equipment, 13 or an orientation of each of the plurality of equipment 114 in the one or more equipment containers 110. Thus, a number of images obtained after a first time instant (i.e., at a second time instant or a second time period) may have a different level of brightness or a different type of shading. Moreover, the initial shape of some equipment of the plurality of equipment 114 may change due to wear and tear during use. Despite such environmental variation, i.e., a change from the first environment condition to a second environment condition different from the first environment condition, the detection of whether tire plurality of equipment 114 are present in their respective positions in the one or more equipment containers 110 is not affected. In other words, the detection is insensitive to any environmental variation in the predefined area 118. Such holistic feature sets determination for both the plurality of equipment 114 and the one or more equipment containers 110 makes the object recognition system 100 accurate and fail-safe for detection of any equipment. The feature sets determination for both the plurality' of equipment 114 and tire one or more equipment containers 110 further makes the object recognition system 100 insensitive to any variation in a visual state (e.g.. lighting condition, shading condition, etc) in the predetermined area, such as the work area of the product assembly line. In an implementation, the controller 104 is further configured to track the plurality of equipment 114 in the predefined area 118 based on the determined first set of features and the second set of features. The tracking is performed by analysing a sequence of image frames obtained periodically or continuously from the one or more image-capture devices 106. The sequence of image frames may be a part of one or more video feeds or images obtained from the one or more image-capture devices 106. The controller 104 correlates the determined first set of features with the second set of features in the sequence of image frames of the predefined area 118 in order to track the plurality of equipment 114. In a case where the controller 104 detects the absence of one or more equipment from the plurality of equipment 114, then the controller 104 keeps track of the one or more equipment to identify whether they are in use or misplaced at some other locations in the predefined area 118. Based on the tracking of the plurality of equipment 114, an operator is able to trace the equipment in the predefined area 118. Beneficially, the combination of the first set of features and the second set of features act synergistically to improve tracking of tire plurality of equipment, removing inconsistency from the product assembly process. This enables better throughput, less disruption and better traceability for all equipment (e.g., 14 kitting parts or tools). For instance, the combination of the first set of features and the second set of features used for tracking enables overcoming changes in visual state(s); for example, any differences in the colour and shape of individual tools and components do not affect recognition of such tools and components. Furthermore, the combination of the first set of features and the second set of features in tracking provides a capability to the object recognition system 100 to recognise when an equipment (e.g., a kitting part or tool) is present in its respective position in an equipment container and when it is used. In an implementation, the controller 104 is further configured to generate a custom message to indicate any misplaced equipment or to assist an operator to trace a given equipment in the predefined area 118. The custom message may be communicated to one or more pre-registered electronic devices, such as a user device of an operator or a computing device located in the predefined area 118. The custom message may be an audio message, a video clip, an audio-visual instruction, or an augmented reality-based projection. In an implementation, the object recognition system 100 further includes a plurality of actuators 116, for example, electronically controllable arms. In such an implementation, the tracking of the plurality of equipment 114 further includes causing one or more actuators of the plurality of actuators to pick up one or more equipment from one or more respective positions in a sequential order in a product assembly line. If there is a requirement for one or more equipment to perform a task in the product assembly line, then the controller 104 commands at least one of the plurality of actuators 116 to pick up a specific equipment followed by another equipment in a defined sequence as per the need for the completion of the task. For example, to complete a task in the product assembly line, there may be three steps in which a different type of equipment may be required in each step. The second equipment 114B may be required to perform a first step, the Nth equipment 114N may be required in a second step, and the first equipment 114A may be required in the third step. In such a scenario, the controller 104 commands the one or more actuators of the plurality of actuators to firstly pick-up the second equipment 114B to complete the first step one, and after completion of the first step, the controller 104 commands the actuator to put back the second equipment 114B in its designated storage section in a specific equipment container. Thereafter, the controller 104 commands another actuator to pick up the Nth equipment 114N to complete the second step and put back the Nth equipment 114N m its designated position in a specific storage section (i.e., a specific compartment) after completion of the second step. Furthermore, the controller 104 commands the actuator to pick up the first equipment 114A to complete the third step and put back the first equipment 114A after completion of the third step. The controller keeps track of the equipment picked up by the one or more actuators of the plurality of actuators and also tracks the sequence of the equipment required to perform a particular task, for example, in accordance with a selected product assembly line. The sequence may change when the selected product assembly line changes. The tracking of the sequence of the equipment used to perform a particular task removes inconsistency from the assembly process, enabling a better throughput, less disruption and improved traceability for the plurality of equipment 114. FIG. 2 is a flow chart of a method for object recognition in accordance with an embodiment of the present disclosure. FIG. 2 is described in conjunction with elements of FIG. 1. With reference to FIG. 2 there is shown a flowchart of a method 200 that includes steps 202 to 206. The method 200 may be implemented by the controller 104. At step 202, tire method 200 includes obtaining a plurality-- of images of a predefined area 118 from the one or more image-capture devices 106 at a first time instant when the predefined area 118 is in a first visual state. The predefined area 118 includes one or more equipment containers 110 and a plurality of equipment 114. The controller 104 is configured to obtain the plurality of images of the predefined area 118 from the one or more image-capture devices 106. Hie one or more image-capture devices 106 may be positioned in the predefined area 118 to allow capture of FOVs that includes one or more equipment containers 110 and one or more equipment of the plurality of equipment 114. At step 204, the method 200 includes determining a first set of features of the one or more equipment containers 110 and a second set of features of the plurality of equipment 114 in the predefined area 118 from the obtained plurality of images. The first set of features and the second set of features correspond to physical characteristics of the one or more 16 equipment containers and the plurality of equipment 114, respectively. The controller 104 is further configured to determine the first set of features of the one or more equipment containers 110 and the second set of features of a plurality of equipment 114 in the predefined area 118. The controller 104 is configured to selectively extract the features for each of the one or more equipment containers 110 and each of the plurality of equipment 114 and exclude other objects. In an implementation, the determining of the first set of features includes calculating an area of a base portion of one or more storage sections 112 (e.g., compartments) of the one or more equipment containers 110 from the obtained plurality of images. Each base portion of the one or more storage sections 112 (e.g., compartments) of the one or more equipment containers 110 is complementary to the size and shape of a given equipment of the plurality of equipment 114. In an implementation, the determining of the first set of features further includes identifying colour of the base portion of the one or more storage sections 112 (e.g., compartments) of the one or more equipment containers 110 from the obtained plurality of images. The base portions of the storage sections 112 (i.e., the compartment bases) are coloured in such a way as to maximise recognition by the object recognition system 100. In an exemplary implementation, each base portion of the one or more storage sections 112 in one equipment container may have a unique colour. In an implementation, the determining of the first set of features further includes determining the shape of the base portion of the one or more storage sections 112 (e.g., compartments) of the one or more equipment containers 110 from the obtained plurality of images. The shape of a given base portion of one or more storage sections 112 (e.g., compartments) of the one or more equipment containers 110 is complementary and approximately congruent to the shape of a given equipment of the plurality of equipment 114 In an implementation, the determining of the second set of features includes determining a colour and a shape of each of the plurality of equipment 114. All the equipment do not have the same shape and colour. Such distinctive characteristics of the plurality7 of equipment 114 is exploited so that their correct placement at correct positions in the one or more equipment containers 110 may be determined from the obtained plurality of images. At step 206, the method 200 further includes detecting whether the plurality of equipment 114 are present in their respective positions in the one or more equipment containers 110 based on the determined first set of features and the second set of features. The predefined area 118 is in the first visual state or a second visual state different from the first visual state at the second time instant. In other words, the detection is insensitive to an variation in the visual state in the predefined area 118. The determined first set of features of each of the one or more equipment containers 110 is correlated with the determined second set of features of each of the plurality of equipment 114 (i.e., the kiting parts and tools) used, for example, in the product assembly line. Based on the correlation, the controller 104 is able to detect whether each of the plurality of equipment 114 is present in their respective positions (e.g., in their designated storage sections) in the one or more equipment containers 110 or is in use by an operator. In an implementation, the method 200 further comprises tracking the plurality of equipment 114 in the predefined area 118 based on the determined first set of features and the second set of features. The tracking is performed by analysing a sequence of image frames obtained periodically or continuously from the one or more image-capture devices 106. The controller 104 correlates the determined first set of features with the second set of features in the sequence of image frames of the predefined area 118 in order to track the plurality of equipment 114. In an implementation, the tracking of the plurality of equipment 114 further includes causing one or more actuators to pick-up one or more equipment from one or more respective positions in a sequential order in a product assembly line. The method 200 determines features for not only the plurality of equipment 114 (i.e., the kiting parts and tools) in the predefined area 118 (e.g., an area in an assembly line) but also for the one or more equipment containers 110 (e.g., kitting boxes) in the predefined area 118. Such holistic feature sets determination for both the plurality of equipment 114 and the one or more equipment containers 110 makes the method 200 accurate and fail-safe for detection of any equipment while being insensitive to any visual state (e.g., any 18 environmental variation) in the predetermined area 118. Modifications to embodiments of the present disclosure described in the foregoing are possible without departing from the scope of the present disclosure as defined by the accompanying claims. Expressions such as "including", "comprising", "incorporating", "have", "is" used to describe and claim the present disclosure are intended to be construed in a non-exclusive manner, namely allowing for items, components or elements not explicitly described also to be present. Reference to the singular is also to be construed to relate to the plural. The word "exemplary" is used herein to mean "serving as an example, instance or illustration". Any embodiment described as “exemplary” is not necessarily to be construed as preferred or advantageous over other embodiments and / or to exclude the incorporation of features from other embodiments. The word "optionally" is used herein to mean "is provided in some embodiments and not provided in other embodiments". It is appreciated that certain features of the present disclosure, which are, for clarity, described in tire context of separate embodiments, may also be provided in combination in a single embodiment. Conversely, various features of the present disclosure, which are, for brevity, described in the context of a single embodiment, may also be provided separately or in any suitable combination or as suitable in any other described embodiment of the disclosure. APPLICANT'S AMENDED CLAIM SET

Claims

1. An object recognition system (100), comprising: a controller (104) configured to:obtain a plurality of images of a predefined area (118) from one or more image-capture devices (106) at a first time instant when the predefined area (118) is in a first visual state, wherein the predefined area (118) comprises one or more equipment containers (110) and a plurality of equipment (114);determine a first set of features of the one or more equipment containers (110) and a second set of features of the plurality of equipment (114) in the predefined area (118) from the obtained plurality of images, wherein the first set of features and the second set of features correspond to physical characteristics of the one or more equipment containers and the plurality of equipment (114), respectively;detect at a second time instant whether the plurality of equipment (114) are present in their respective positions in the one or more equipment containers (110) based on the determined first set of features and the second set of features, wherein the predefined area (118) is in the first visual state or a second visual state different from the first visual state at the second time instant; andtrack the plurality of equipment (114) in the predefined area (118) based on the determined first set of features and the second set of features,wherein the system further comprises a plurality of actuators (116), and wherein the tracking of the plurality of equipment (114) further comprises causing one or more actuators of the plurality of actuators(116) to pick up one or more equipment from one or more respective positions in a sequential order in a product assembly line.

2. The system (100) according to claim 1, wherein the determining of the first set of features comprises calculating an area of a base portion of one or more storage sections of the one or more equipment containers (110) from the obtained plurality of images.

3. The system (100) according to claim 1 or 2, wherein the determining of the first set of features further comprises identifying a colour of the base portion of the one or more storage sections of the one or more equipment containers (IIO) from the obtained plurality of images.

4. The system (100) according to any one of the claims 1 to 3, wherein the determining of the first set of features further comprises determining a shape of the base portion of the one or more storage sections of the one or more equipment containers (IIO) from the obtained plurality of images.

5. The system (100) according to claim 1, wherein the determining of the second set of features comprises determining a colour and a shape of each of the plurality of equipment (114).

6. The system (100) according to any of the preceding claims, wherein a change from the first visual state to the second visual state corresponds to a change in one or more of: a lighting condition, a shading condition, a current shape of a given equipment, or an orientation of each of the plurality of equipment (114) in the one or more equipment containers (IIO), and wherein each of the one or more image- capture devices (106) is a colour camera device.

7. A method (200) for object recognition, comprising:obtaining, by a controller (104), a plurality of images of a predefined area(118) from one or more image-capture devices (106) at a first time instant when the predefined area ( 118) is in a first visual state, wherein the predefined area ( 118) comprises one or more equipment containers (110) and a plurality of equipment (114);determining, by the controller (104), a first set of features of the one or more equipment containers (110) and a second set of features of the plurality of equipment ( 114) in the predefined area ( 118) from the obtained plurality of images, wherein the first set of features and the second set of features correspond to physical characteristics of the one or more equipment containers and the plurality of equipment (114), respectively;detecting, by the controller (104), at a second time instant whether the plurality of equipment (114) are present in their respective positions in the one or more equipment containers (110) based on the determined first set of features and the second set of features, wherein the predefined area (118) is in the first visual state or a second visual state different from the first visual state at the second time instant; andtracking the plurality of equipment (114) in the predefined area (118) based on the determined first set of features and the second set of features,wherein the system further comprises a plurality of actuators (116), and wherein the tracking of the plurality of equipment (114) further comprises causing one or more actuators of the plurality of actuators (116) to pick up one or more equipment from one or morerespective positions in a sequential order in a product assembly line.

8. The method (200) according to claim 9, wherein the determining of the first set of features comprises calculating an area of a base portion of one or more storage sections of the one or more equipment containers (110) from the obtained plurality of images.

9. The method (200) according to claim 7 or 8, wherein the determining of the first set of features further comprises identifying a colour of the base portion of the one or more storage sections of the one or more equipment containers (110) from the obtained plurality of images.

10. The method (200) according to any one of the claims 7 to 9, wherein the determining of the first set of features further comprises determining a shape of the base portion of the one or more storage sections of the one or more equipment containers (110) from the obtained plurality of images.

11. The method (200) according to claim 9, wherein the determining of the second set of features comprises determining a colour and a shape of each of the plurality of equipment (114).

Citation Information

Patent Citations

  • Inventory control system

    WO2008029159A1