Manufacturing part recognition using computer vision and machine learning

Abstract

This disclosure provides "Manufactured Part Recognition Using Computer Vision and Machine Learning". Systems, methods, and apparatuses for object detection, including manufactured part detection, are disclosed. A method for recognizing a manufactured part includes receiving an image containing the manufactured part from an augmented reality device, and determining the perimeter of a boundary encapsulating the manufactured part. The method includes receiving a predicted label and a confidence value for the manufactured part from a neural network. The method includes generating a message containing the image, the predicted label, and the confidence value, and providing the message to the augmented reality device.

Description

Technical Field

[0001] This disclosure generally relates to systems, methods, and apparatus for identifying and selecting objects, such as manufactured parts, and providing information associated with those objects. Background Technology

[0002] Augmented reality-assisted devices are used to improve the efficiency and accuracy of manufacturing and selecting manufactured parts. Some known "visual selection" procedures in this field provide information about the location of manufactured parts, or they may provide a diagram of the process for assembling manufactured parts. Such procedures include numerous limitations, including, for example, guiding the user to select the wrong part from the wrong location, failing to provide detailed information about assembling parts, requiring parts to be labeled with barcodes or other unique identifiers, or failing to provide accurate information to the user in real time. These limitations may lead to users selecting the wrong parts or assembling parts incorrectly, which can result in significant losses of productivity in manufacturing facilities.

[0003] Many errors occur when selecting parts from storage bins and shelves. Common errors include selecting the wrong parts, especially when parts are very similar in size or shape, and omitting one or more parts. Such errors reduce efficiency and productivity, cause downtime in manufacturing, and can result in significant costs for the manufacturer. Some parts are very small or lack barcodes or other unique identifiers; these parts may only be identifiable by size, color, and other identifiers. Such parts can be difficult to distinguish by the human eye and are frequently misidentified during manufacturing.

[0004] Systems, methods, and apparatus for identifying objects such as manufactured parts improve manufacturing productivity, increase manufacturing efficiency and accuracy, and reduce downtime. In some manufacturing industries that use hundreds or thousands of unique parts to complete complex projects, leveraging computer vision and machine learning to identify manufactured parts and ensure project completion according to agreements is beneficial. Summary of the Invention

[0005] According to one embodiment of this disclosure, a method for identifying manufactured parts is disclosed. In one embodiment, the method is performed by a computer server communicating with an augmented reality device and a neural network. The method includes receiving an image containing the manufactured part from the augmented reality device. The image can be extracted from a video stream received in real time from the augmented reality device. The method includes determining a boundary perimeter encapsulating the manufactured part and providing a sub-image created by the boundary perimeter to the neural network. The method includes receiving a predicted label and a confidence value for the manufactured part from the neural network. The method includes generating a message containing the image, the predicted label, and the confidence value, wherein the predicted label and the confidence value are overlaid on the original image received from the augmented reality device. The method includes providing the message to the augmented reality device. In one embodiment, the image including the predicted label and the confidence value is displayed to a user wearing the augmented reality device.

[0006] According to one embodiment, a system for identifying manufactured parts is disclosed. The system includes an augmented reality device comprising a camera. The augmented reality device may include a wearable device, such as headphones or glasses that can be worn by a user. The system includes a server communicating with the augmented reality device, comprising a computer-readable storage medium storing instructions that, when executed by one or more processors, cause the one or more processors to perform one or more method steps. The system includes a neural network communicating with the server, wherein the neural network is configured to determine a predicted label and confidence value for a manufactured part in an image captured by the augmented reality device. In one embodiment, the neural network comprises a CNN and is configured to learn and update new information related to the manufactured part, an option table, and the storage location of the manufactured part in real time.

[0007] According to one embodiment, the neural network comprises a CNN based on those CNNs used for object detection, which is trained using a labeled training dataset of images of manufactured parts to be used as classifiers. The convolutional layers apply convolution operations to the input, such as images of manufactured parts, and pass the results to the next layer. The neural network is trained to recognize and identify various manufactured parts with a threshold accuracy level. The neural network is further trained to determine a confidence value for each object recognition performed by the neural network. Attached Figure Description

[0008] The following figures illustrate non-limiting and non-exhaustive embodiments of the present disclosure, wherein, unless otherwise stated, the same reference numerals refer to the same parts in the various figures. The advantages of the present disclosure will be better understood with reference to the following description and figures, in which:

[0009] Figure 1 This is a schematic diagram illustrating a system for identifying manufactured parts according to one implementation method;

[0010] Figure 2 A wearable augmented reality device for detecting manufactured parts according to one implementation method is shown;

[0011] Figure 3 An exemplary image captured by an augmented reality device according to one implementation is shown;

[0012] Figure 4 An exemplary image captured by an augmented reality device and modified by a server, according to one implementation, is shown.

[0013] Figure 5 An exemplary image captured by an augmented reality device and modified by a server, according to one implementation, is shown.

[0014] Figure 6 An exemplary image prepared by a server for processing by a neural network, according to one implementation, is shown.

[0015] Figure 7 An exemplary image prepared by a server for processing by a neural network, according to one implementation, is shown.

[0016] Figure 8 It is a schematic flowchart of an exemplary process flow for object detection and recognition according to one implementation method;

[0017] Figure 9 This is a schematic flowchart illustrating a method for identifying manufactured parts according to one implementation.

[0018] Figure 10 This is a schematic flowchart illustrating a method for identifying manufactured parts according to one implementation.

[0019] Figure 11 This is a schematic flowchart illustrating a method for identifying manufactured parts according to one implementation; and

[0020] Figure 12 This is a schematic block diagram illustrating an exemplary computing system according to one implementation. Detailed Implementation

[0021] Augmented reality (AR) assistive devices improve efficiency and accuracy in manufacturing. These devices are used in a variety of fields, including automotive manufacturing, where thousands or millions of manufacturing parts are located on-site, and a single project may require hundreds of unique parts. In many fields, including automotive manufacturing, individual projects require parts of similar size and shape that cannot be mismatched during assembly. Such parts can be difficult to distinguish with the human eye. AR devices, including wearable AR headsets or glasses, help users select and identify the correct manufacturing parts and assemble complex projects requiring many unique components. It should be understood that currently available AR systems can be used in multiple scenarios, including any manufacturing and assembly industry.

[0022] In this disclosure, the applicant proposes and presents systems, methods, and apparatuses for object recognition using computer vision and machine learning. Such systems, methods, and apparatuses can be integrated into augmented reality devices that can be used by a user responsible for selecting and assembling manufacturing parts. Such systems, methods, and apparatuses can be integrated with convolutional neural networks (CNNs) based on such CNNs for object detection and trained with labeled training datasets.

[0023] The applicant proposes systems, methods, and apparatus for improving the efficiency and accuracy of selecting parts or objects. In one embodiment, the system provides a user with indication of the location of a manufactured part, including, for example, aisle number, shelf number, and / or location, and box number and / or location. When the user approaches the correct location, the system identifies the manufactured part and indicates to the user whether the correct part has been selected. In one embodiment, the system includes the ability to record an image of the manufactured part when it is stored and associated with a part number, barcode, QR code, etc. As parts are stored in real time, the system is taught and learns about each of a plurality of parts. The system automatically receives and updates an options table containing a list of manufactured parts required to complete a manufacturing project. The system includes the ability to read coded labels (including barcodes, QR codes, etc.).

[0024] Before disclosing and describing methods, systems, and apparatus for detecting objects such as manufactured parts, it should be understood that this disclosure is not limited to the configurations, process steps, and materials disclosed herein, as such configurations, process steps, and materials may vary. It should also be understood that the terminology used herein is for descriptive purposes only and is not intended to be limiting, as the scope of this disclosure will be limited only by the appended claims and their equivalents.

[0025] In describing and claiming protection for this invention, the following terms will be used in accordance with the definitions set forth below.

[0026] It should be noted that, unless the context clearly indicates otherwise, the singular forms “a” and “the” used in this specification and drawings include the plural reference forms.

[0027] As used herein, the terms “comprising,” “including,” “containing,” “characterized by,” and their grammatical equivalents are inclusive or open-ended terms that do not exclude additional, unlisted elements or method steps.

[0028] Now refer to the attached diagram, Figure 1 An exemplary system 100 is shown that can be used to select and identify parts (including manufactured parts). System 100 includes an augmented reality device 102 that communicates with a network 110. System 100 includes a server 104 that communicates with the network 110 and a neural network 106. The network 110 may also communicate with a database 108. It should be noted that, without departing from the scope of this disclosure, network 110 includes any suitable network known in the art, including cloud computing networks and / or the Internet, and / or portions of closed or private networks.

[0029] Augmented reality device 102 includes any suitable augmented reality device 102 known in the art, including, for example, augmented reality glasses, augmented reality headphones, etc. The augmented reality techniques utilized by augmented reality device 102 may include, for example, projection-based augmented reality and overlay-based augmented reality. Augmented reality device 102 includes one or more cameras capable of capturing images or video streams. It should be understood that the one or more cameras may include depth-sensing cameras and environmental understanding cameras to aid in the identification and recognition of manufactured parts. The one or more cameras may be used to view one or more manufactured parts to provide images or video streams of the manufactured parts to server 104. It should be understood that augmented reality device 102 may include one or more additional sensors that can aid in the identification and recognition of manufactured parts. Such sensors include, for example, LiDAR sensors, radar sensors, accelerometers, Global Positioning System (GPS) sensors, etc.

[0030] Server 104 communicates with augmented reality device 102 and can send and receive information from augmented reality device 102. Server 104 also communicates with neural network 106, such as CNN, and can send and receive information from neural network 106. Server 104 is responsible for identifying one or more process steps in manufacturing parts.

[0031] In one embodiment, a user is responsible for retrieving manufactured parts from their storage location. The user is equipped with an augmented reality device 102, such as an augmented reality wearable headset. The augmented reality device 102 can connect to a server 104 and request an option table. The server 104 can retrieve the appropriate option table from a database 108 or from memory stored on the server 104. The server 104 provides the appropriate option table to the augmented reality device 102. The option table contains necessary information to guide the user in retrieving one or more necessary manufactured parts. This information may include part name, part number, part location, etc. The augmented reality device 102 provides the information to the user and guides the user to the correct location of the first part. The user can then approach the correct location and view the surrounding environment through the augmented reality device 102 to view one or more parts stored at that location.

[0032] In one embodiment, a user can activate the object recognition component 112 of the augmented reality device 102. The object recognition component 112 can be activated by any means known in the art, including, for example, via voice commands, gestures, or activating a button on the augmented reality device 102. The object recognition component 112 captures video streams or images via the camera on the augmented reality device 102 and provides the video streams or images to a server 104. The server 104 receives the video streams or images. In the case where the server 104 receives a video stream, the server 104 can extract individual images from the video stream, wherein the individual images contain screenshots of one or more manufactured parts.

[0033] In one embodiment, server 104 receives and / or extracts an image containing one or more manufactured parts. Server 104 applies gradient thresholds to the color and size of the image to effectively remove the background and retain only the outlines of possible objects of interest (e.g., one or more manufactured parts) within the image. Server 104 determines the contours of the manufactured parts to determine a mask representing the location of the manufactured parts in the original image. Server 104 uses the contours to determine a boundary perimeter that encapsulates the entire contour of the manufactured parts. The boundary perimeter is applied to the original image, and the boundary perimeter creates a sub-image on the original image. Server 104 resizes the sub-image to fit the machine learning model utilized by neural network 106. Server 104 provides the sub-image to neural network 106.

[0034] Neural network 106 receives a sub-image from server 104 in one channel, making the image grayscale. In one embodiment, neural network 106 receives an image from only one channel instead of three channels (e.g., a color image) to reduce the number of nodes in neural network 106. Reducing the number of nodes in neural network 106 significantly increases processing time but does not significantly reduce accuracy. Neural network 106 determines a prediction label containing a prediction of the identification of a manufactured part in the sub-image. The prediction label indicates, for example, the common name of the part, a specialized name of an industry-known part, a part number, a part descriptor, the storage location of the part, etc. Neural network 106 determines a confidence value containing the statistical likelihood that the prediction label is correct. In one embodiment, the confidence value represents the percentage likelihood that the prediction label is correct. The determination of the confidence value is based on one or more parameters, including, for example, the quality of the image received by neural network 106, the number of similar parts that may mismatch the part in question, the past performance of neural network 106 in correctly identifying the prediction label, etc. Neural network 106 provides the prediction label and confidence value to server 104. Server 104 overlays predicted labels and confidence values ​​onto the original image and provides the overlaid image to augmented reality device 102. In one embodiment, server 104 overlays a boundary perimeter around a manufactured part, indicates a predicted label near the boundary perimeter, and indicates a confidence value near the boundary perimeter. In one embodiment where a single image contains more than one manufactured part, server 104 may provide a boundary perimeter, predicted label, and confidence value of different colors for each manufactured part.

[0035] In one embodiment, neural network 106 is a convolutional neural network (CNN) known in the field. A CNN contains convolutional layers as the core building blocks of neural network 106. The parameters of the convolutional layers include a set of learnable filters or kernels that have small receptive fields but stretch across the entire depth of the input volume. During forward propagation, each filter is convolved across the width and height of the input volume, the dot product between the filter entries and the input is computed, and a two-dimensional activation map of the filters is produced. As a result, neural network 106 learns which filters to activate when it detects a specific type of feature (such as a specific feature on a manufactured part) at a certain spatial location in the input. In neural network 106, the activation maps of all filters stacked along the depth dimension form the entire output volume of the convolutional layers. Each entry in the output volume can therefore also be interpreted as the output of a neuron that focuses on a small region in the input and shares parameters with neurons in the same activation map. Neural network 106, as a CNN, can successfully perform image recognition, including identifying manufactured parts from images captured by augmented reality device 102 with a very low error rate.

[0036] Further, as an example of neural network 106, a single camera image (or other single set of sensor data) can be fed into a common layer of neural network 106, which serves as the foundational part of neural network 106. The common layer performs feature extraction on the image and provides one or more output values ​​reflecting the feature extraction. Because the common layer is trained for each task, a single feature extraction extracts the features needed for all tasks. The feature-extracted values ​​are output to sub-task parts, including, for example, a first task layer, a second task layer, and a third task layer. Each of the first, second, and third task layers processes the feature-extracted values ​​from the common layer to determine the output for its respective task.

[0037] Those skilled in the art will understand that a single neural network 106 consists of multiple nodes and edges connecting the nodes. The weights or values ​​of the edges or nodes are used to compute the output of the edges connected to subsequent nodes. Therefore, a single neural network 106 can be composed of multiple neural networks to perform one or more tasks. Figure 1 The neural network 106 includes several common layers as its foundational or common components. These common layers can be understood as subnetworks that form part of the neural network 106. The first task layer, second task layer, third task layer, and so on, then utilize the computations and processing performed in the common layers. Therefore, the neural network 106 includes a branching topology, where the results of the common layers are then used independently by each of the multiple subnetworks within the branches of the neural network 106. Because the common layers are trained sequentially while performing multiple tasks to avoid forgetting previously trained tasks, they can perform tasks that serve each branch of the neural network well. Furthermore, the common layers result in reduced computation because the tasks of the common layers are performed once for all tasks represented by the branches, rather than once for each individual task. An example of a task that the common layers perform is feature extraction. However, any task that might share an initial processing task can share a common layer.

[0038] Figure 2 An augmented reality device 102 for viewing a manufactured part 202 according to one implementation is shown. The augmented reality device 102 has a field of view marked by a first periphery 204 and a second periphery 206. The manufactured part 202 within the field of view can be detected by the augmented reality device 102. In one embodiment, the augmented reality device 102 captures a video stream while a user is wearing the augmented reality device 102, and the field of view can reflect or approximate the user's actual field of view when not wearing the augmented reality device 102.

[0039] Figures 3 to 7 Various embodiments of images and sub-images captured and determined by the systems, methods and apparatuses disclosed herein are shown. Figure 3An exemplary raw image 300 captured by augmented reality device 102 is shown. The raw image 300 includes a first component 302 and a second component 304. The raw image 300 can be captured by augmented reality device 102 as part of a video stream (containing multiple image frames) and extracted by server 104 as a separate raw image 300. Server 104 can receive the raw image 300 from augmented reality device 102.

[0040] Figure 4 The altered original image 400, determined by server 104, is shown. Server 104 applies gradient thresholds to the color and size of the original image 300 to effectively remove the background of the original image 300 and leave only the outlines of possible objects of interest. In one embodiment, the altered original image 400 includes a black background and one or more white outline shapes representing objects in the original image 300. In one embodiment, the altered original image 400 includes a first outline 402 representing the outline shape of a first part 302 and a second outline 404 representing the outline shape of a second part 304. The first outline 402 is used to determine the perimeter or outline of the entire body encapsulating the first part 302. Similarly, the second outline 404 is used to determine the perimeter or outline of the entire body encapsulating the second part 304.

[0041] Figure 5 A boundary perimeter image 500 including an original image 300 is shown, wherein boundary perimeters 502, 504 include one or more objects applied to the original image 300. The boundary perimeter image 500 includes a first boundary perimeter 502 enclosing a first part 302 and a second boundary perimeter 504 enclosing a second part 304. In one embodiment, server 104 applies the first boundary perimeters 502, 504 to parts 302, 304 after determining the outlines 402, 404. The boundary perimeters 502, 504 are used to tightly enclose objects formed by, for example, objects formed by ... Figure 4 The outline shown defines the entire body of the part.

[0042] Figure 6 A first sub-image 600, created and defined by the size and shape of a first boundary perimeter 502, is shown, which includes a first part 302. Server 104 determines the first sub-image 600 and provides it to neural network 106. The first sub-image 600 includes the entire original image 300 located within the area of ​​the first boundary perimeter 502.

[0043] Figure 7A second sub-image 700 of the second part 304, created and defined by the size and shape of the second boundary perimeter 504, is shown. Server 104 determines the second sub-image 700 and provides it to neural network 106. The second sub-image 700 includes the entire original image 300 located within the area of ​​the second boundary perimeter 504.

[0044] Figure 8 A schematic block diagram of a process flow 800 for object detection and recognition is shown. The process flow 800 is performed in part by computer vision 820 and machine learning 830. Computer vision 820 receives an input image 802, such as a raw image 300 captured by augmented reality device 102. Computer vision 820 performs image reprocessing at 804 and feature extraction at 806. In one embodiment, the computer vision 820 processing is performed by server 104. Image reprocessing 804 and feature extraction 806 include, for example... Figures 3 to 7 This reprocessing and extraction is shown in the diagram.

[0045] Machine learning 830 is responsible for building the neural network model at 808, training the neural network model at 810, testing the neural network model and tuning the parameters in the neural network model at 812, and completing the neural network model at 814. Machine learning 830 may receive training data (including, for example, images of manufactured parts) at 816 to help train the neural network model 808. In one embodiment, machine learning 830 is performed by a neural network 106 such as a CNN.

[0046] Computer vision 820 provides images to the final model at 814 for classification at 818 via machine learning 830. In one embodiment, neural network 106 includes the final model 814 of the machine learning 830 process flow. After the final model 814 is formed via machine learning 830, the final model 814 can communicate with server 104 and thus with augmented reality device 102.

[0047] Figure 9 A schematic flowchart of a method 900 for identifying manufactured parts is shown. Method 900 begins, and server 104 receives an image containing the manufactured part from an augmented reality device at 902. Server 104 determines the boundary perimeter of the packaged manufactured part at 904. At 906, server 104 receives a predicted label and confidence value for the manufactured part from a neural network. At 908, server 104 generates a message containing the image, predicted label, and confidence value. Server 104 provides the message to the augmented reality device at 910.

[0048] Figure 10A schematic flowchart of a method 1000 for identifying manufactured parts is shown. Method 1000 begins, and server 104 receives a video stream containing the manufactured parts from an augmented reality device at 1002. At 1004, server 104 extracts an image containing the manufactured parts from the video stream. At 1006, server 104 applies a gradient threshold to the image for color and size to remove the background. At 1008, server 104 determines the outline shape of the manufactured parts. At 1010, server 104 determines the boundary perimeter of the outline shape of the manufactured parts. At 1012, server 104 crops the image to create a sub-image containing only the boundary perimeter of the manufactured parts and provides the sub-image to the neural network. At 1014, server 104 receives a predicted label and confidence value for the manufactured parts from the neural network. At 1016, server 1016 creates or generates a message containing an image overlaid with the predicted label and confidence value. At 1018, server 104 provides the message to the augmented reality device.

[0049] Figure 11 A schematic flowchart of method 1100 for identifying manufactured parts is shown. Method 1100 begins and neural network 106 receives a sub-image from server 104 in one channel at 1102, wherein the sub-image contains a grayscale representation of the manufactured part. Neural network 106 determines a predicted label for the manufactured part at 1104, wherein the predicted label contains the most likely identifier of the manufactured part. Neural network 106 determines a confidence value associated with the predicted label at 1106, wherein the confidence value contains the statistical likelihood that the predicted label is correct. Neural network 106 provides the predicted label and confidence value to the server at 1108.

[0050] Now for reference Figure 12 A block diagram of an exemplary computing device 1200 is shown. The computing device 1200 can be used to execute various programs, such as those discussed herein. In one embodiment, the computing device 1200 can be used as an augmented reality device 102, a server 104, a neural network 106, etc. The computing device 1200 can perform various monitoring functions as discussed herein, and can execute one or more applications, such as the applications or functions described herein. The computing device 1200 can be any of a variety of computing devices, such as a desktop computer, an internal computer, a vehicle control system, a laptop computer, a server computer, a handheld computer, a tablet computer, etc.

[0051] The computing device 1200 includes one or more processors 1202, one or more memory devices 1204, one or more interfaces 1206, one or more mass storage devices 1208, one or more input / output (I / O) devices 1210, and a display device 1230, all of which are coupled to a bus 1212. The one or more processors 1202 include one or more processors or controllers that execute instructions stored in one or more memory devices 1204 and / or one or more mass storage devices 1208. The one or more processors 1202 may also include various types of computer-readable media, such as cache memory.

[0052] One or more memory devices 1204 include various computer-readable media, such as volatile memory (e.g., random access memory (RAM) 1214) and / or non-volatile memory (e.g., read-only memory (ROM) 1216). One or more memory devices 1204 may also include rewritable ROM, such as flash memory.

[0053] One or more mass storage devices 1208 include various computer-readable media, such as magnetic tape, magnetic disk, optical disk, solid-state storage (e.g., flash memory), etc. Figure 12 As shown, a specific mass storage device is hard disk drive 1224. Various drives may also be included in one or more mass storage devices 1208 to enable reading from and / or writing to various computer-readable media. One or more mass storage devices 1208 include removable media 1226 and / or non-removable media.

[0054] One or more I / O devices 1210 include various means that allow data and / or other information to be input to or retrieved from the computing device 1200. One or more exemplary I / O devices 1210 include cursor control devices, keyboards, microphones, monitors or other display devices, speakers, printers, network interface cards, modems, etc.

[0055] Display device 1230 includes any type of device capable of displaying information to one or more users of computing device 1200. Examples of display device 1230 include monitors, display terminals, video projection devices, etc.

[0056] One or more interfaces 1206 include various interfaces that allow computing device 1200 to interact with other systems, devices, or computing environments. One or more exemplary interfaces 1206 may include any number of different network interfaces 1220, such as interfaces for local area networks (LANs), wide area networks (WANs), wireless networks, and the Internet. One or more other interfaces include user interfaces 1218 and peripheral device interfaces 1222. One or more interfaces 1206 may also include one or more user interface elements 1218. One or more interfaces 1206 may also include one or more peripheral interfaces, such as interfaces for printers, pointing devices (mice, trackpads, or any suitable user interfaces now known to or later discovered by a person skilled in the art), keyboards, etc.

[0057] Bus 1212 allows one or more processors 1202, one or more memory devices 1204, one or more interfaces 1206, one or more mass storage devices 1208, and one or more I / O devices 1210 to communicate with each other and with other devices or components coupled to bus 1212. Bus 1212 represents one or more types of bus architectures, such as system bus, PCI bus, IEEE bus, USB bus, etc.

[0058] For illustrative purposes, programs and other executable program components are shown herein as discrete blocks; however, it should be understood that such programs and components may reside in different storage units of computing device 1200 at different times and be executed by one or more processors 1202. Optionally, the systems and processes described herein may be implemented in hardware or a combination of hardware, software, and / or firmware. For example, one or more application-specific integrated circuits (ASICs) may be programmed to execute one or more of the systems and processes described herein.

[0059] Example

[0060] The following examples relate to further embodiments.

[0061] Example 1 is a method for identifying manufactured parts. The method includes receiving an image containing the manufactured part from an augmented reality device, and determining the perimeter of a boundary encapsulating the manufactured part. The method includes receiving a predicted label and a confidence value for the manufactured part from a neural network. The method includes generating a message containing the image, the predicted label, and the confidence value, and providing the message to the augmented reality device.

[0062] Example 2 is the method as described in Example 1, which further includes applying a gradient threshold of color and size to the image such that the background of the image is largely eliminated while the outline of the manufactured part is preserved;

[0063] Example 3 is the method as described in Example 2, which further includes determining the profile shape of the manufactured part based on the remaining profile of the manufactured part;

[0064] Example 4 is a method as described in any one of Examples 1 to 3, wherein the augmented reality device includes a camera and a display with augmented reality functionality, and wherein the augmented reality device is configured to be worn by a user.

[0065] Example 5 is a method as described in any one of Examples 1 to 4, wherein receiving the image includes receiving a video stream from the augmented reality device, and wherein the method further includes extracting the image from the video stream as a single frame.

[0066] Example 6 is a method as described in any one of Examples 1 to 5, which further includes extracting a sub-image containing image data within a range surrounding the boundary, and resizing the sub-image to create a resized sub-image to be processed by the neural network.

[0067] Example 7 is a method as described in Example 6, wherein the neural network receives the sub-image as a grayscale image from the server in one channel.

[0068] Example 8 is a method as described in any one of Examples 1 to 7, which further includes providing the augmented reality device with a manufacturing parts option table, wherein the manufacturing parts option table contains data for guiding a user operating the augmented reality device to retrieve one or more manufacturing parts.

[0069] Example 9 is a method as described in any one of Examples 1 to 8, wherein the neural network comprises a convolutional neural network trained for object detection.

[0070] Example 10 is a method as described in any one of Examples 1 to 9, wherein generating the message includes overlaying the predicted label and the confidence value onto the image such that the predicted label and the confidence value are located near the manufactured part or overlaid on top of the manufactured part.

[0071] Example 11 is a manufacturing part recognition system. The system includes an augmented reality device comprising a camera. The system includes a server communicating with the augmented reality device, comprising a computer-readable storage medium storing instructions that, when executed by one or more processors, cause the one or more processors to: receive an image containing a manufacturing part from the augmented reality device; determine the outline shape of the manufacturing part based on the image; and determine the boundary perimeter of an enclosure of the manufacturing part. The system includes a neural network communicating with the server, wherein the neural network is configured to: determine a predicted label for the manufacturing part; and determine a confidence value for the statistical likelihood that the predicted label is correct.

[0072] Example 12 is a system as described in Example 11, wherein the augmented reality device includes a camera and a display with augmented reality capabilities, and wherein the augmented reality device is configured to be worn by a user.

[0073] Example 13 is a system as described in any one of Examples 11 to 12, wherein the server, which includes a computer-readable storage medium, further enables the one or more processors to determine a sub-image, wherein the sub-image comprises the image cropped to the periphery of the boundary of the packaged manufacturing part.

[0074] Example 14 is a system as described in any one of Examples 11 to 13, wherein the server, which includes a computer-readable storage medium, further enables the one or more processors to resize the sub-image to a shape and / or resolution acceptable to the neural network.

[0075] Example 15 is a system as described in any one of Examples 11 to 14, wherein the neural network is further configured to receive the sub-image as a grayscale image from the server in one channel.

[0076] Example 16 is a system as described in any one of Examples 11 to 15, wherein the server, which includes a computer-readable storage medium, further causes the one or more processors to apply gradient thresholds of color and size to the image such that the background of the image is largely eliminated while the outline of the manufactured part is preserved.

[0077] Example 17 is a system as described in any one of Examples 11 to 16, wherein the neural network comprises a convolutional neural network trained for object detection.

[0078] Example 18 is a system as described in any one of Examples 11 to 17, wherein the augmented reality device is configured to provide a video stream to the server in real time for real-time detection of manufactured parts.

[0079] Example 19 is a system as described in any one of Examples 11 to 18, wherein the server, which includes a computer-readable storage medium, further enables the one or more processors to overlay the predicted labels and the confidence values ​​onto the image and to provide the image to the augmented reality device.

[0080] Example 20 is a system as described in any one of Examples 11 to 19, wherein the server, which includes a computer-readable storage medium, further enables the one or more processors to provide a manufacturing parts option table to the augmented reality device, wherein the manufacturing parts option table contains data for guiding a user operating the augmented reality device to retrieve and select one or more manufacturing parts.

[0081] Example 21 is a computer-readable storage medium storing instructions that, when executed by one or more processors, cause the one or more processors to: receive an image containing a manufactured part from an augmented reality device; apply color and size of gradient thresholds to the image such that the background of the image is effectively eliminated; determine the outline shape of the manufactured part; determine the boundary perimeter encapsulating the outline shape; receive a predicted label and confidence value of the manufactured part from a convolutional deep neural network; generate a message containing the image, the predicted label, and the confidence value; and provide the message to the augmented reality device.

[0082] In the foregoing disclosure, reference is made to the accompanying drawings, which form part of this disclosure and illustrate specific implementations of this disclosure by way of illustration. It should be understood that other implementations and structural changes can be utilized without departing from the scope of this disclosure. References to "one embodiment," "exemplary embodiment," etc., indicate that the described embodiment may include a particular feature, structure, or characteristic; however, each embodiment may not necessarily include said particular feature, structure, or characteristic. Furthermore, such phrases do not necessarily refer to the same embodiment. Moreover, when a particular feature, structure, or characteristic is described in connection with an embodiment, whether explicitly described or not, implementing such a feature, structure, or characteristic in conjunction with other embodiments is within the knowledge of those skilled in the art.

[0083] Implementations of the systems, apparatuses, and methods disclosed herein may include or utilize dedicated or general-purpose computers comprising computer hardware, such as, for example, one or more processors and system memories discussed herein. Implementations within the scope of this disclosure may also include physical and other computer-readable media for carrying or storing computer-executable instructions and / or data structures. Such computer-readable media may be any available medium accessible by a general-purpose or dedicated computer. A computer-readable medium storing computer-executable instructions is a computer storage medium (apparatus). A computer-readable medium carrying computer-executable instructions is a transmission medium. Thus, by way of example and not limitation, implementations of this disclosure may include at least two distinct types of computer-readable media: a computer storage medium (apparatus) and a transmission medium.

[0084] Computer storage media (devices) include RAM, ROM, EEPROM, CD-ROM, solid-state drives (“SSDs”) (e.g., RAM-based), flash memory, phase-change memory (“PCM”), other types of memory, other optical disc storage devices, disk storage devices or other magnetic storage devices, or any other medium that can be used to store desired program code in the form of computer-executable instructions or data structures and that can be accessed by a general-purpose or special-purpose computer.

[0085] The apparatuses, systems, and methods disclosed herein are implemented via computer networks. A “network” is defined as one or more data links capable of transmitting electronic data between computer systems and / or modules and / or other electronic devices. When information is transmitted or provided to a computer via a network or another communication connection (hardwired, wireless, or a combination of hardwired and wireless), the computer appropriately regards the connection as a transmission medium. The transmission medium may include networks and / or data links, which may be used to carry desired program code in the form of computer-executable instructions or data structures and may be accessible by a general-purpose or special-purpose computer. The above combinations should also be included within the scope of computer-readable media.

[0086] Computer-executable instructions comprise instructions and data that, when executed, cause a general-purpose computer, special-purpose computer, or special-purpose processing device to perform a function or group of functions, for example, when executed in a processor. Computer-executable instructions can be, for example, binary files, intermediate format instructions (such as assembly language), or even source code. While the subject matter has been described in language specific to structural features and / or methodological actions, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the features or actions described above. Rather, the features and actions are disclosed as exemplary forms for implementing the claims.

[0087] Those skilled in the art will appreciate that this disclosure can be practiced in networked computing environments with many types of computer system configurations, including built-in vehicle computers, personal computers, desktop computers, laptop computers, message processors, handheld devices, multiprocessor systems, microprocessor-based or programmable consumer electronics, network PCs, minicomputers, mainframe computers, mobile phones, PDAs, tablet computers, pagers, routers, switches, various storage devices, etc. This disclosure can also be practiced in distributed system environments, where both local and remote computer systems, linked via a network (via a hardwired data link, a wireless data link, or a combination of hardwired and wireless data links), perform tasks. In a distributed system environment, program modules can reside on both local and remote memory storage devices.

[0088] Furthermore, where appropriate, the functions described herein can be performed in one or more of the following: hardware, software, firmware, digital components, or analog components. For example, one or more application-specific integrated circuits (ASICs) can be programmed to perform one or more of the systems and processes described herein. Certain terms are used throughout the description and claims to refer to specific system components. The terms “module” and “component” are used in the names of certain components to reflect their implementation independence in software, hardware, circuits, sensors, etc. It will be understood by those skilled in the art that components may be referred to by different names. This document is not intended to distinguish between components with different names but the same function.

[0089] It should be noted that the sensor embodiments discussed above may include computer hardware, software, firmware, or any combination thereof to perform at least a portion of their functions. For example, a sensor may include computer code configured to execute in one or more processors, and may include hardware logic / circuit controlled by the computer code. These exemplary devices are provided herein for illustrative purposes and are not intended to be limiting. Embodiments of this disclosure may be implemented in one or more other types of devices as known to those skilled in the art.

[0090] At least some embodiments of this disclosure relate to computer program products that include such logic (e.g., in software form) stored on any computer-usable medium. When such software is executed in one or more data processing devices, it causes the devices to operate as described herein.

[0091] While various embodiments of the invention have been described above, it should be understood that these embodiments are presented by way of example only and not by way of limitation. Those skilled in the art will appreciate that various changes in form and detail can be made without departing from the spirit and scope of this disclosure. Therefore, the breadth and scope of this disclosure should not be limited to any of the exemplary embodiments described above, but should be defined only by the following claims and their equivalents. The above description has been presented for purposes of illustration and description. It is not exhaustive, nor does it limit this disclosure to the precise forms disclosed. Many modifications and variations are possible in light of the foregoing teachings. Furthermore, it should be noted that any combination of any or all of the foregoing alternative implementations may be expected to form additional hybrid implementations of this disclosure.

[0092] Furthermore, while specific implementations of this disclosure have been described and illustrated, this disclosure is not limited to the particular forms or arrangements of the portions so described and illustrated. The scope of this disclosure is defined by the appended claims, the claims filed herein, and any future claims filed in different applications and their equivalents.

[0093] According to the present invention, a method for identifying a manufactured part includes: receiving an image containing the manufactured part from an augmented reality device; determining the perimeter of a boundary encapsulating the manufactured part; receiving a predicted label and a confidence value of the manufactured part from a neural network; generating a message including the image, the predicted label, and the confidence value; and providing the message to the augmented reality device.

[0094] According to one embodiment, the invention is further characterized in that the color and size of the gradient threshold are applied to the image such that most of the background of the image is eliminated while the outline of the manufactured part is preserved.

[0095] According to one embodiment, the invention is further characterized in that the outline shape of the manufactured part is determined based on the remaining outline of the manufactured part.

[0096] According to one embodiment, the augmented reality device includes a camera and a display with augmented reality capabilities, and the augmented reality device is configured to be worn by a user.

[0097] According to one embodiment, receiving the image includes receiving a video stream from the augmented reality device, and the method further includes extracting the image from the video stream as a single frame.

[0098] According to one embodiment, the invention is further characterized by extracting a sub-image containing image data within a range surrounding the boundary, and adjusting the size of the sub-image to create an adjusted sub-image to be processed by the neural network.

[0099] According to one embodiment, the invention is further characterized in that a manufacturing parts option table is provided to the augmented reality device, wherein the manufacturing parts option table contains data for guiding a user operating the augmented reality device to retrieve one or more manufacturing parts.

[0100] According to one embodiment, the neural network includes a convolutional neural network trained for object detection.

[0101] According to one embodiment, generating the message includes overlaying the predicted label and the confidence value onto the image such that the predicted label and the confidence value are located near the manufactured part or overlaid on top of the manufactured part in the image.

[0102] According to the present invention, a manufacturing part identification system includes: an augmented reality device including a camera; a server communicating with the augmented reality device, including a computer-readable storage medium storing instructions, the instructions, when executed by one or more processors, causing the one or more processors to: receive an image containing a manufacturing part from the augmented reality device; determine the outline shape of the manufacturing part based on the image; and determine the boundary perimeter of the packaged manufacturing part; and a neural network communicating with the server, wherein the neural network is configured to: determine a predicted label for the manufacturing part; and determine a confidence value for the statistical likelihood that the predicted label is correct.

[0103] According to one embodiment, the augmented reality device includes a camera and a display with augmented reality capabilities, and the augmented reality device is configured to be worn by a user.

[0104] According to one embodiment, the server, which includes a computer-readable storage medium, further causes the one or more processors to determine a sub-image, wherein the sub-image includes the image cropped to the periphery of the boundary of the packaged manufacturing part.

[0105] According to one embodiment, the server, which includes a computer-readable storage medium, further enables the one or more processors to resize the sub-image to a shape and / or resolution acceptable to the neural network.

[0106] According to one embodiment, the neural network is also configured to receive the sub-image as a grayscale image from the server in one channel.

[0107] According to one embodiment, the server, which includes a computer-readable storage medium, further enables the one or more processors to apply gradient thresholds of color and size to the image such that the background of the image is largely eliminated while preserving the outline of the manufactured part.

[0108] According to one embodiment, the neural network includes a convolutional neural network trained for object detection.

[0109] According to one embodiment, the augmented reality device is configured to provide a video stream to the server in real time to detect manufactured parts in real time.

[0110] According to one embodiment, the server, which includes a computer-readable storage medium, further enables the one or more processors to overlay the predicted labels and the confidence values ​​onto the image and to provide the image to the augmented reality device.

[0111] According to one embodiment, the server, which includes a computer-readable storage medium, further enables the one or more processors to provide a manufacturing parts option table to the augmented reality device, wherein the manufacturing parts option table contains data for guiding a user operating the augmented reality device to retrieve and select one or more manufacturing parts.

[0112] According to the present invention, a computer-readable storage medium storing instructions, which, when executed by one or more processors, cause the one or more processors to: receive an image containing a manufactured part from an augmented reality device; apply a gradient threshold of color and size to the image such that the background of the image is largely eliminated while the outline of the manufactured part is preserved; determine the outline shape of the manufactured part; determine the boundary perimeter encapsulating the outline shape; receive a predicted label and a confidence value of the manufactured part from a convolutional deep neural network; generate a message including the image, the predicted label, and the confidence value; and provide the message to the augmented reality device.

Claims

1. A method for identifying manufactured parts, the method comprising: Receive raw images containing the manufactured parts from the augmented reality device; Determine the perimeter of the boundary of the manufactured part; The boundary perimeter is applied to the original image to create a sub-image on the original image and provide the sub-image to the neural network, the sub-image containing the original image data within the range of the boundary perimeter; wherein, the neural network receives the sub-image in only one channel; Receive the predicted labels and confidence values ​​of the manufactured parts from the neural network; Generate a message containing the original image, the predicted label, and the confidence value; and The message is provided to the augmented reality device.

2. The method of claim 1, further comprising: Applying gradient thresholds to the color and size of the original image results in the background of the original image being largely eliminated while preserving the outline of the manufactured part; And adjust the size of the sub-image to create a resized sub-image to be processed by the neural network; as well as A manufacturing parts option table is provided to the augmented reality device, wherein the manufacturing parts option table contains data for guiding a user operating the augmented reality device to retrieve one or more manufacturing parts.

3. The method of claim 1, wherein the augmented reality device comprises a camera and a display with augmented reality functionality, and wherein the augmented reality device is configured to be worn by a user.

4. The method of claim 1, wherein receiving the original image comprises receiving a video stream from the augmented reality device, and wherein the method further comprises extracting the original image from the video stream as a single frame.

5. The method of claim 1, wherein the neural network comprises a convolutional neural network trained for object detection.

6. The method of claim 1, wherein generating the message comprises overlaying the predicted label and the confidence value onto the original image such that the predicted label and the confidence value are located near or overlaid on the manufactured part in the original image.

7. A manufacturing part identification system, comprising: Augmented reality device, the augmented reality device including a camera; A server, which communicates with the augmented reality device, and includes a computer-readable storage medium containing instructions that, when executed by one or more processors, cause the one or more processors to: Receive raw images containing the manufactured parts from the augmented reality device; The outline shape of the manufactured part is determined based on the original image; and Determine the boundary perimeter of the manufactured part, apply the boundary perimeter to the original image, and create a sub-image on the original image, the sub-image containing the original image data within the range of the boundary perimeter; and A neural network, which communicates with the server, wherein the neural network is configured to: in, The neural network receives only a sub-image from one channel; Determine the predicted label of the manufactured part; and Determine the confidence value for the statistical likelihood that the predicted label is correct.

8. The system of claim 7, wherein the augmented reality device comprises a camera and a display with augmented reality functionality, and wherein the augmented reality device is configured to be worn by a user.

9. The system of claim 7, wherein the server comprising a computer-readable storage medium further causes the one or more processors to perform the following steps: Determine a sub-image, wherein the sub-image comprises the original image cropped to the periphery of the boundary of the packaged manufacturing part; The size of the sub-image is adjusted to a shape and / or resolution acceptable to the neural network; and Applying gradient thresholds to the color and size of the original image results in the background of the original image being largely eliminated while preserving the outline of the manufactured part.

10. The system of claim 9, wherein the neural network is further configured to receive the sub-image as a grayscale image from the server in one channel.

11. The system of claim 7, wherein the neural network comprises a convolutional neural network trained for object detection.

12. The system of claim 7, wherein the augmented reality device is configured to provide a video stream to the server in real time for real-time detection of manufactured parts.

13. The system of claim 7, wherein the server, which includes a computer-readable storage medium, further enables the one or more processors to overlay the predicted label and the confidence value onto the original image and to provide the original image to the augmented reality device.

14. The system of claim 7, wherein the server, which includes a computer-readable storage medium, further enables the one or more processors to provide a manufacturing parts option table to the augmented reality device, wherein the manufacturing parts option table contains data for guiding a user operating the augmented reality device to retrieve and select one or more manufacturing parts.

15. A computer-readable storage medium storing instructions, said instructions causing said one or more processors, when executed, to: Receive raw images containing the manufactured parts from the augmented reality device; Applying gradient thresholds to the color and size of the original image results in the background of the original image being largely eliminated while preserving the outline of the manufactured part; Determine the outline shape of the manufactured part; Determine the boundary perimeter of the encapsulated contour shape; The boundary perimeter is applied to the original image to create a sub-image on the original image and provides the sub-image to the convolutional deep neural network. The sub-image contains the original image data within the range of the boundary perimeter. The convolutional deep neural network receives the sub-image from only one channel. The predicted labels and confidence values ​​of the manufactured parts are received from the convolutional deep neural network; Generate a message containing the original image, the predicted label, and the confidence value; and The message is provided to the augmented reality device.

Images